Install Debian GNU/Linux on a FreeBSD Jail with ZFS

Continuing with the process of configuring a  newly installed FreeBSD system to get a production environment with hosted services in Jails. 

We will continue working with Debian and Linux-based distributions that provide the right Userland to meet the goal, not fully achieved, of executing  Linux binaries on FreeBSD Jail.

Preliminary considerations

Binary compatibility for Linux that provides FreeBSD 9 is limited to 32-bit applications. FreeBSD provides binary interface, not a 'emulation', with support for Linux kernel up to version 2.6.16. So when selecting a version of Debian Linux should keep in mind these limitations.

 

Creating our Jail

The implementation of a Debian GNU/Linux Jail is similar to a Debian GNU/kFreeBSD Jail, however, has significant differences which must be highlighted.

1. - Install the tool Debootstrap, which installs Debian in a subdirectory of another system.

root@morsa:/root # portsnap fetch update 
root@morsa:/root # portmaster sysutils/debootstrap

2. - Configure the file system where we will install the Jail.

root@morsa:/root # zfs create -o mountpoint=/jailz fbsdzpool1/jailz
root@morsa:/root # zfs create fbsdzpool1/jailz/deb-master
root@morsa:/root # mkdir /jailz/etc

3 - Load the necessary modules for Linux compatibility.

 root@morsa:/root # kldload fdescfs linprocfs linsysfs tmpfs

4.- We enable Jails support and compatibility with Linux.

root@morsa:/root # echo 'jail_enable="YES"' >> /etc/rc.conf
root@morsa:/root # echo 'linux_enable="YES"' >> /etc/rc.conf

5.- Only if installing squeeze, change the compatibility level declared 2.6.18.

root@morsa:/root #  echo 'compat.linux.osrelease=2.6.18' >> /etc/sysctl.conf

6.- Include in the configuration file /jailz/etc/jail.conf:

// Start fragment

deb-master {
  path = /jailz/deb-master;
  allow.mount;
  mount.devfs;
  host.hostname = deb-master;
  mount.fstab="/jailz/etc/fstab.deb-master";
  ip4.addr = 127.0.0.10;
  interface = lo0;
  exec.start = "/etc/init.d/rc 3";
  exec.stop = "/etc/init.d/rc 0";
}

// End of excerpt

7.- Define the mounting points for the Jail in  /jail/etc/fstab.dev-master: 

// Begin the File /jailz/etc/fstab.deb-master

linsys   /jailz/deb-master/sys         linsysfs  rw          0 0
linproc  /jailz/deb-master/proc        linprocfs rw          0 0
tmpfs    /jailz/deb-master/lib/init/rw tmpfs     rw,mode=777 0 0

// End of the File /jailz/etc/fstab.deb-master


8.- With debootstrap install Debian GNU/Linux, lenny or squeeze versions, in the path of the Jail.

root@morsa:/root # debootstrap --foreign --arch=i386 lenny /jailz/deb-master http://archive.debian.org/debian/
I: Retrieving Release
W: Cannot check Release signature; keyring file not available /usr/share/keyring
s/debian-archive-keyring.gpg
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://archive.debian.org/debian...
I: Retrieving libacl1 2.2.47-2
I: Validating libacl1 2.2.47-2
I: Retrieving adduser 3.110
I: Validating adduser 3.110
I: Retrieving apt-utils 0.7.20.2+lenny2
I: Validating apt-utils 0.7.20.2+lenny2
I: Retrieving apt 0.7.20.2+lenny2
....
I: Extracting login...
I: Extracting passwd...
I: Extracting libslang2...
I: Extracting initscripts...
I: Extracting sysv-rc...
I: Extracting sysvinit-utils...
I: Extracting sysvinit...
I: Extracting tar...
I: Extracting tzdata...
I: Extracting bsdutils...
I: Extracting mount...
I: Extracting util-linux...
I: Extracting zlib1g...

9. - Inside the Jail, delete  the configuration files sysvinit_*.

root@morsa:/jailz/etc # ls /jailz/deb-master/var/cache/apt/archives/sysvinit_*
/jailz/deb-master/var/cache/apt/archives/sysvinit_2.86.ds1-61_i386.deb
root@morsa:/jailz/etc # rm /jailz/deb-master/var/cache/apt/archives/sysvinit_*

  10. - We mount the filesystems of the Jail, in this case, takes place after installation with debootstrap.

root@morsa:/root # mount -t linprocfs none /jailz/deb-master/proc
root@morsa:/root # mount -t devfs none /jailz/deb-master/dev
root@morsa:/root # mount -t linsysfs none /jailz/deb-master/sys
root@morsa:/root # mount -t tmpfs none /jailz/deb-master/lib/init/rw 

 11. - Run a shell with chroot within the path of the Jail.

root@morsa:/root # chroot /jailz/deb-master /bin/bash

12. - To complete the configuration of the Jail, from the shell started in the chroot environment, run :

 I have no name!@morsa:/# dpkg --force-depends -Ei /var/cache/apt/archives/*.deb

13. -  We left the previous shell and unmount the previously mounted file systems in step 10.

root@morsa:/root # umount /jailz/deb-master/proc
root@morsa:/root # umount /jailz/deb-master/dev
root@morsa:/root # umount /jailz/deb-master/sys
root@morsa:/root # umount /jailz/deb-master/lib/init/rw

 In the likely event that can not be unmounted /jailz/deb-master/dev because the filesystem is busy, run:

# fstat | grep deb-master
# kill -9 PID (For each process listed in the previous step)

Or, restart the system.

Never start the Jail without cleaning the processes and unmounting /jailz/deb-master/dev.

14. - You need to disable rsyslog inside the Jail,  because is not supported by the Linux compatibility module.

Therefore, before starting the Jail, for each directory in the path /jailz/deb-master/etc/rcX.d (where X takes values from 0 to 6) rename the service startup scripts.

# mv S10rsyslog _S10rsyslog
# mv K90rsyslog _K90rsyslog

15. - Start the Jail, check that is correctly started and login.

root@morsa:/root # jail -f /jailz/etc/jail.conf -c deb-master
deb-master: created
Starting periodic command scheduler: crond.

root@morsa:/root # jls
JID IP Address Hostname Path
1 127.0.0.10 deb-master /jailz/deb-master

root@morsa:/root # jexec 1 /bin/bash
deb-master:/# uname -a
Linux deb-master 2.6.16 FreeBSD 9.1-RELEASE-p4 #0: Mon Jun 17 11:42:37 UTC 2013 i686 GNU/Linux

16. - Edit the file /etc/apt/sources.list and correct their content.
// Start the excerpt.

http://archive.debian.org/debian/ deb lenny main contrib non-free

// End of excerpt

17. - Update the package list.

deb-master:/# apt-get update
Get:1 http://archive.debian.org lenny Release.gpg [1034B]
Get:2 http://archive.debian.org lenny Release [99.6kB]
Get:3 http://archive.debian.org lenny/main Packages [6872kB]
Get:4 http://archive.debian.org lenny/non-free Packages [124kB]
Get:5 http://archive.debian.org lenny/contrib Packages [94.3kB]
Fetched 7191kB in 11s (649kB/s)
Reading package lists... Done

18. - Shutdown the Jail, the error messages are due to processes within the Jail can not perform certain operations.

root@morsa:/root # jail -f /jailz/etc/jail.conf -r deb-master

umount2: Operation not permitted
umount: fbsdzpool1/jailz: must be superuser to umount
umount2: Operation not permitted
umount: fbsdzpool1/ROOT/91_30062013/usr: must be superuser to umount
umount2: Operation not permitted
umount: fbsdzpool1/ROOT/91_30062013/usr: must be superuser to umount
umount2: Operation not permitted
umount: fbsdzpool1/ROOT/91_30062013/var: must be superuser to umount
umount2: Operation not permitted
umount: fbsdzpool1/ROOT/91_30062013/var: must be superuser to umount
failed.
mount: fbsdzpool1/ROOT/91_30062013: unknown device
Will now halt.
ifdown: shutdown usbus0: Invalid argument
ifdown: shutdown ath0: Invalid argument
ifdown: shutdown usbus1: Invalid argument
ifdown: shutdown lo0: Invalid argument
ifdown: shutdown lo0: Invalid argument
ifdown: shutdown eth1: Invalid argument

deb-master: removed

 19. - Make a ZFS snapshot of the Jail.

root@morsa:/root # zfs snapshot zfs snapshot fbsdzpool1/jailz/deb-master@lenny 

Now, we have a Base Jail, from which we can generate new Jails with ZFS clones, then  we added a new entry to /jailz/jail.conf and we created the file /jailz/etc/fstab.newjail.

# zfs clone fbsdzpool1/jailz/deb-master@lenny fbsdzpool1/jailz/newjail

On these cloned Jails, we can begin to run 32-bit linux apps with their own Linux elf format.

 

Final Thoughts

All final thoughts about to install Debian GNU/kFreeBSD in a Jail are equally applicable to this post.

In this post I installed Userlands with versions of lenny and squeeze, but I recommend using the lenny release. The Linux  compatibility that FreeBSD provides is for kernel version 2.6.16 and minimal kernel version required by squeeze is 2.6.18;  although it works fairly well, those applications using features specific to the 2.6.18 not work at all.

As a final note, to enable internet access from the Jail, required to install packages, we enable the pf module, then we make NAT with the IP addresses of the Jails, to do this execute the following:

root@morsa:/root # kldload pf
root@morsa:/root # echo 'pf_enable="YES"' >> /etc/rc.conf
root@morsa:/root # echo "nat on wlan0 from 127.0.0.10 to any -> (wlan0)" >> /etc/pf.conf