Jul 19, 2013

Install Debian GNU/kFreeBSD on a FreeBSD Jail with ZFS

Once we have installed and updated a FreeBSD 9 system, the next step is to provide it with apps to make it a useful system.

In recent years, I have been working on Solaris 10 systems, where services had in production were encapsulated in Solaris Containers. Given my recent interest in FreeBSD seems natural to attempt to play a similar execution environment, which gave me such good results in the past. 

Moreover, it seems interesting the possibility to include the ability to run Linux applications and having a jail specifically intended for that purpose. 

In this case, instead of installing a Jail with the  FreeBSD native  userland, we will install Debian GNU/kFreeBSD that provides an alternative userland for FreeBSD 9.

Creating our Jail

1. - Install the tool Debootstrap, which installs Debian in a subdirectory of another system. 
root@morsa:/root # portsnap fetch update 
root@morsa:/root # portmaster sysutils/debootstrap
2. - Configure the file system where we will install the Jail. 
root@morsa:/root # zfs create -o mountpoint=/jailz fbsdzpool1/jailz
root@morsa:/root # zfs create fbsdzpool1/jailz/deb-master
root@morsa:/root # mkdir /jailz/etc
3 - Load the necessary modules for Linux compatibility.
root@morsa:/root # kldload fdescfs linprocfs linsysfs tmpfs
4.-Install  DebianSqueeze on the route of the Jail.
root@morsa:/root # debootstrap squeeze /jailz/deb-master http://cdn.debian.net/debian
I: Retrieving Release
W: Cannot check Release signature; keyring file not available /usr/share/keyring
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Found additional required dependencies: insserv libbsd0 libbz2-1.0 libcam0 li
bdb4.8 libedit2 libexpat1 libgeom0 libkiconv4 libkvm0 libsbuf0 libslang2 ufsutils
I: Found additional base dependencies: libipx2 libmemstat3 libnetgraph4 libsqlite3-0
I: Checking component main on http://cdn.debian.net/debian...
I: Retrieving libacl1 2.2.49-4


I: Configuring gnupg...
I: Configuring debian-archive-keyring...
I: Configuring apt...
I: Configuring libept1...
I: Configuring apt-utils...
I: Configuring aptitude...
I: Configuring tasksel-data...
I: Configuring tasksel...
I: Base system installed successfully.
5. - Define the mounting points for the Jail in  /jail/etc/fstab.dev-master:

// Begin the File /jailz/etc/fstab.deb-master
linsys   /jailz/deb-master/sys         linsysfs  rw          0 0
linproc  /jailz/deb-master/proc        linprocfs rw          0 0
tmpfs    /jailz/deb-master/lib/init/rw tmpfs     rw,mode=777 0 0 
// End of the File /jailz/etc/fstab.deb-master

6. - Enable the Jails.
root@morsa:/root # echo 'jail_enable="YES"' >> /etc/rc.conf
7. - Include in the configuration file /jailz/etc/jail.conf:

/ / Start fragment
deb-master {
  path = /jailz/deb-master;
  host.hostname = deb-master;
  ip4.addr =;
  interface = lo0;
  exec.start = "/etc/init.d/rc 3";
  exec.stop = "/etc/init.d/rc 0";
 / / End of excerpt

8. - boot the jail, check that it is enabled and login.
root@morsa:/root # jail -f /jailz/etc/jail.conf -c deb-master
deb-master: created
Using makefile-style concurrent boot in runlevel 3.
Starting enhanced syslogd: rsyslogdln: creating symbolic link `/dev/xconsole': Operation not permitted.
Starting periodic command scheduler: cron.
root@morsa:/root # jls
   JID  IP Address      Hostname                      Path
     1      deb-master                    /jailz/deb-master
root@morsa:/root # jexec 1 /bin/bash
root@deb-master:/# uname -a

GNU/kFreeBSD deb-master 9.1-RELEASE-p4 FreeBSD 9.1-RELEASE-p4 #0: Mon Jun 17 11:
42:37 UTC 2013     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENER
IC x86_64 amd64 Intel(R) Pentium(R) CPU B960 @ 2.20GHz GNU/kFreeBSD
 9. - Shutdown the Jail.
root@morsa:/root # jail -f /jailz/etc/jail.conf -r deb-master
Using makefile-style concurrent boot in runlevel 0.
Stopping device state change daemon: devd.
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds....done.
Stopping enhanced syslogd: rsyslogd.
/etc/init.d/umountnfs.sh: 106: cannot open /etc/mtab: No such file
Deconfiguring network interfaces...done.
Cleaning up ifupdown....
Deactivating swap...done.
mount: /: unknown special file or file system
Will now halt.
ifdown: shutdown usbus0: Operation not permitted
ifdown: shutdown ath0: Operation not permitted
ifdown: shutdown usbus1: Operation not permitted
ifdown: shutdown lo0: Operation not permitted
ifdown: shutdown wlan0: Operation not permitted
startpar: service(s) returned failure: umountnfs.sh ... failed!
deb-master: removed
10. - Make a ZFS snapshot of the Jail.
root@morsa:/root # zfs snapshot zfs snapshot fbsdzpool1/jailz/deb-master@squeeze
Now, we have a Base Jail, from which we can generate new Jails with ZFS clones,  then  we added a  new entry to /jailz/jail.conf and we created the file /jailz/etc/fstab.newjail.
# zfs clone fbsdzpool1/jailz/deb-master@sequeeze fbsdzpool1/jailz/newjail
On these cloned Jails, we can begin installing applications provided by debian repositories, but the elf format of the binaries  is not linux, is  freebsd.


Final Thoughts

In this test environment we have defined the Jail file system in the Root Pool. In a production environment it would be advisable to have a different pool, in another disk or set of disks, which are different from the system disks.

Another aspect that we can improve the network settings, where we can use virtual network devices, instead of IP alias.

Remains to implement resource management by rctl.
I have reserved the path /usr/jail as a route for the Jails that can be created from a BE generated with beadm. Such Jails are intended to used for testing system upgrade,  and then put into production.


  1. Thanks for putting together the guideline.

    I'll have to say that I'm getting quite stuck when it comes to devfs. I've never gotten it quite right. Since the architecture is kFreeBSD but Debian/Linux code expects a certain (/dev) API then I'd guess that the /dev under the kFreeBSD chroot would have to behave accordingly. And so a

    mount -v -t nullfs /dev $CHROOT/dev

    wouldn't do the trick. But what I'm getting is that ttys are not being created or accessed properly. For example, I set up sshd within the chroot, and then I connect to it from outside the jail. I get:

    PTY allocation request failed on channel 0
    stdin: is not a tty

    Instead of having a shell prompt, I just have this blank line. Now I can still type commands, but ssh(d) doesn't believe that there's a real tty so it's hard to do some real work. There is no $CHROOT/dev/ptmx; there are entries in $CHROOT/dev/pts/; and in the host /etc/defaults/devfs.rules, I've put various rules like:

    add path ptmx unhide
    add path pts unhide
    add path 'pts/*' unhide
    add path fd unhide
    add path 'fd/*' unhide
    add path stdin unhide
    add path stdout unhide
    add path stderr unhide
    add path 'ptyp*' unhide...

    The entry for /dev/pts/0 and $CHROOT/dev/pts/0 has the same inode as well as the same device number. Also, I restarted the system after modifying my devfs rules. $CHROOT/dev seems to be mounted as type devfs. Also, /etc/jail.conf has an entry like:

    kfreebsd {
    path = "/var/jail/kfreebsd";
    host.hostname = kfreebsd;
    ip4.addr =;
    interface = lo0;
    exec.start = "/etc/init.d/rc 3";
    exec.stop = "/etc/init.d/rc 0";

    Within the chroot, other device entries like /dev/null and /dev/random are behaving as they should. But somehow I'm not able to ssh in and get a real tty. The only way I'm able to get a normal shell prompt is if I bypass the jail system and chroot there, but then I don't really necessarily own the tty in that case either. Any ideas would be appreciated.

  2. hi Anonymous,

    I see the same behaviour after updating to 10.x. I was pretty careless during the upgrade and tried real hard to provoke a few errors, this is one one that still sticks out.

    So, to clarify:
    FreeBSD 9: SSH Login to Debian/kFreebsd jail worked
    FreeBSD 10: SSH Login hangs as you describe

    If I get progress around the pts issue, I'll update accordingly.


Comments are welcome, I encourage you to contribute by proposing topics of your interest to develop in this blog.