In recent years, I have been working on Solaris 10 systems, where services had in production were encapsulated in Solaris Containers. Given my recent interest in FreeBSD seems natural to attempt to play a similar execution environment, which gave me such good results in the past.
Moreover, it seems interesting the possibility to include the ability to run Linux applications and having a jail specifically intended for that purpose.
In this case, instead of installing a Jail with the FreeBSD native userland, we will install Debian GNU/kFreeBSD that provides an alternative userland for FreeBSD 9.
Creating our Jail
1. - Install the tool Debootstrap, which installs Debian in a subdirectory of another system.
root@morsa:/root # portsnap fetch update2. - Configure the file system where we will install the Jail.
root@morsa:/root # portmaster sysutils/debootstrap
root@morsa:/root # zfs create -o mountpoint=/jailz fbsdzpool1/jailz3 - Load the necessary modules for Linux compatibility.
root@morsa:/root # zfs create fbsdzpool1/jailz/deb-master
root@morsa:/root # mkdir /jailz/etc
root@morsa:/root # kldload fdescfs linprocfs linsysfs tmpfs4.-Install DebianSqueeze on the route of the Jail.
root@morsa:/root # debootstrap squeeze /jailz/deb-master http://cdn.debian.net/debian
I: Retrieving Release5. - Define the mounting points for the Jail in /jail/etc/fstab.dev-master:
W: Cannot check Release signature; keyring file not available /usr/share/keyring
s/debian-archive-keyring.gpg
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Found additional required dependencies: insserv libbsd0 libbz2-1.0 libcam0 li
bdb4.8 libedit2 libexpat1 libgeom0 libkiconv4 libkvm0 libsbuf0 libslang2 ufsutils
I: Found additional base dependencies: libipx2 libmemstat3 libnetgraph4 libsqlite3-0
I: Checking component main on http://cdn.debian.net/debian...
I: Retrieving libacl1 2.2.49-4
.........
I: Configuring gnupg...
I: Configuring debian-archive-keyring...
I: Configuring apt...
I: Configuring libept1...
I: Configuring apt-utils...
I: Configuring aptitude...
I: Configuring tasksel-data...
I: Configuring tasksel...
I: Base system installed successfully.
// Begin the File /jailz/etc/fstab.deb-master
linsys /jailz/deb-master/sys linsysfs rw 0 0// End of the File /jailz/etc/fstab.deb-master
linproc /jailz/deb-master/proc linprocfs rw 0 0
tmpfs /jailz/deb-master/lib/init/rw tmpfs rw,mode=777 0 0
6. - Enable the Jails.
root@morsa:/root # echo 'jail_enable="YES"' >> /etc/rc.conf7. - Include in the configuration file /jailz/etc/jail.conf:
/ / Start fragment
deb-master {/ / End of excerpt
path = /jailz/deb-master;
allow.mount;
mount.devfs;
host.hostname = deb-master;
mount.fstab="/jailz/etc/fstab.deb-master";
ip4.addr = 127.0.0.10;
interface = lo0;
exec.start = "/etc/init.d/rc 3";
exec.stop = "/etc/init.d/rc 0";
}
8. - boot the jail, check that it is enabled and login.
root@morsa:/root # jail -f /jailz/etc/jail.conf -c deb-master
deb-master: created
Using makefile-style concurrent boot in runlevel 3.
Starting enhanced syslogd: rsyslogdln: creating symbolic link `/dev/xconsole': Operation not permitted.
Starting periodic command scheduler: cron.
root@morsa:/root # jls
JID IP Address Hostname Path
1 127.0.0.10 deb-master /jailz/deb-master
root@morsa:/root # jexec 1 /bin/bash9. - Shutdown the Jail.
root@deb-master:/# uname -a
GNU/kFreeBSD deb-master 9.1-RELEASE-p4 FreeBSD 9.1-RELEASE-p4 #0: Mon Jun 17 11:
42:37 UTC 2013 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENER
IC x86_64 amd64 Intel(R) Pentium(R) CPU B960 @ 2.20GHz GNU/kFreeBSD
root@morsa:/root # jail -f /jailz/etc/jail.conf -r deb-master
Using makefile-style concurrent boot in runlevel 0.
Stopping device state change daemon: devd.
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds....done.
Stopping enhanced syslogd: rsyslogd.
/etc/init.d/umountnfs.sh: 106: cannot open /etc/mtab: No such file
Deconfiguring network interfaces...done.
Cleaning up ifupdown....done.
Deactivating swap...done.10. - Make a ZFS snapshot of the Jail.
mount: /: unknown special file or file system
Will now halt.
ifdown: shutdown usbus0: Operation not permitted
ifdown: shutdown ath0: Operation not permitted
ifdown: shutdown usbus1: Operation not permitted
ifdown: shutdown lo0: Operation not permitted
ifdown: shutdown wlan0: Operation not permitted
startpar: service(s) returned failure: umountnfs.sh ... failed!
deb-master: removed
root@morsa:/root # zfs snapshot zfs snapshot fbsdzpool1/jailz/deb-master@squeeze
Now, we have a Base Jail, from which we can generate new Jails with ZFS clones, then we added a new entry to /jailz/jail.conf and we created the file /jailz/etc/fstab.newjail.
In this test environment we have defined the Jail file system in the Root Pool. In
a production environment it would be advisable to have a different
pool, in another disk or set of disks, which are different from the
system disks. # zfs clone fbsdzpool1/jailz/deb-master@sequeeze fbsdzpool1/jailz/newjailOn these cloned Jails, we can begin installing applications provided by debian repositories, but the elf format of the binaries is not linux, is freebsd.
Final Thoughts
Another aspect that we can improve the network settings, where we can use virtual network devices, instead of IP alias.
Remains to implement resource management by rctl.
I have reserved the path /usr/jail as a route for the Jails that can be created from a BE generated with beadm. Such Jails are intended to used for testing system upgrade, and then put into production.
Thanks for putting together the guideline.
ReplyDeleteI'll have to say that I'm getting quite stuck when it comes to devfs. I've never gotten it quite right. Since the architecture is kFreeBSD but Debian/Linux code expects a certain (/dev) API then I'd guess that the /dev under the kFreeBSD chroot would have to behave accordingly. And so a
mount -v -t nullfs /dev $CHROOT/dev
wouldn't do the trick. But what I'm getting is that ttys are not being created or accessed properly. For example, I set up sshd within the chroot, and then I connect to it from outside the jail. I get:
PTY allocation request failed on channel 0
stdin: is not a tty
Instead of having a shell prompt, I just have this blank line. Now I can still type commands, but ssh(d) doesn't believe that there's a real tty so it's hard to do some real work. There is no $CHROOT/dev/ptmx; there are entries in $CHROOT/dev/pts/; and in the host /etc/defaults/devfs.rules, I've put various rules like:
add path ptmx unhide
add path pts unhide
add path 'pts/*' unhide
add path fd unhide
add path 'fd/*' unhide
add path stdin unhide
add path stdout unhide
add path stderr unhide
add path 'ptyp*' unhide...
The entry for /dev/pts/0 and $CHROOT/dev/pts/0 has the same inode as well as the same device number. Also, I restarted the system after modifying my devfs rules. $CHROOT/dev seems to be mounted as type devfs. Also, /etc/jail.conf has an entry like:
kfreebsd {
path = "/var/jail/kfreebsd";
allow.mount;
mount.devfs;
host.hostname = kfreebsd;
mount.fstab="/etc/fstab.kfreebsd";
ip4.addr = 127.0.0.30;
interface = lo0;
exec.start = "/etc/init.d/rc 3";
exec.stop = "/etc/init.d/rc 0";
}
Within the chroot, other device entries like /dev/null and /dev/random are behaving as they should. But somehow I'm not able to ssh in and get a real tty. The only way I'm able to get a normal shell prompt is if I bypass the jail system and chroot there, but then I don't really necessarily own the tty in that case either. Any ideas would be appreciated.
hi Anonymous,
ReplyDeleteI see the same behaviour after updating to 10.x. I was pretty careless during the upgrade and tried real hard to provoke a few errors, this is one one that still sticks out.
So, to clarify:
FreeBSD 9: SSH Login to Debian/kFreebsd jail worked
FreeBSD 10: SSH Login hangs as you describe
If I get progress around the pts issue, I'll update accordingly.