Install FreeBSD 9 with root on ZFS optimized for 4K sectors and support for beadm

In the last 6 years I have worked with   Solaris 10 SPARC systems  - M3000 , M4000 , V1280 - where I used ZFS as filesystem.
 

For a while I looked for solutions on x64 architecture servers similar in performance and features of these systems.

Of course the obvious answer is to opt for environments under Solaris 11 for x64; technologically is a good solution -not perfect- but given the apparent corporate culture change that has occurred after the acquisition of Sun by Oracle, never hurts to evaluate alternatives

Btrfs in Linux seems an option to keep in mind, however it suffers , in my opinion of a certain technological immaturity that makes not advisable its use, at the moment, for deployment in production environments.
 
The solution presented in this post is  today, I think of the most interesting, because combines an open source OS, with an undeniable technological maturity and probably the best filesystem today; ZFS.

Scripts for creating the ZFS Root Pool

Installing FreeBSD 9 on ZFS requires a number of additional steps that are necessarily performed in command line. In order to make the process more comfortable and reduce the likelihood of making errors, I created a set of shell scripts to be executed during the installation.

Script 1: env_fbsd_rpool.sh

This script defines a set of environment variables that are used by the other scripts, to adapt the behavior of the scripts to the specific needs of each facility simply change the environment variables.

// Start the Script env_fbsd_rpool.sh

#!/bin/sh

now=$(date +"%d%m%Y")
BENAME="91_${now}"
RPOOLNAME="fbsdzpool1"
BEPATH=${RPOOLNAME}/ROOT/${BENAME}
SWAP_GB="4G"
GPT_DEV=freedsk1
ALT_ROOT=/mnt

export BENAME    # Initial BE name
export RPOOLNAME # Root Pool Name
export BEPATH    # Boot Enviroment Path
export SWAP_GB   # swap size (GB)
export GPT_DEV   # installation device
export ALT_ROOT  # installation mount point

echo ""
echo "Initial BE name          : ${BENAME}"
echo "Root Pool Name           : ${RPOOLNAME}"
echo "Boot Enviroment Path     : ${BEPATH}"
echo "swap size (GB)           : ${SWAP_GB}"
echo "installation device      : ${GPT_DEV}"
echo "installation mount point : ${ALT_ROOT}"
echo ""

// End of Script env_fbsd_rpool.sh
 

Script 2: mk_fbsd_rpool.sh

This script creates the Root Pool and FreeBSD filesystems.

// Start the Script mk_fbsd_rpool.sh

#!/bin/sh

. ./env_fbsd_rpool.sh

#
# We created the pseudo-device to initialize the ZFS pool with blocks of 4KB
#

echo ""
echo "Creating pseudo-device ${GPT_DEV}"
echo ""

gnop create -S 4096 /dev/gpt/${GPT_DEV}

#
# Creating the root pool
#
echo ""
echo "Creating Pool ${RPOOLNAME}"
echo ""

zpool create -f -m none -o cachefile=/var/tmp/zpool.cache ${RPOOLNAME} /dev/gpt/${GPT_DEV}.nop

#
# Deleting pseudo-device
#

zpool export ${RPOOLNAME}

echo ""
echo "Deleting pseudo-device ${GPT_DEV}"
echo ""

gnop destroy /dev/gpt/${GPT_DEV}.nop
zpool import -N -o cachefile=/var/tmp/zpool.cache ${RPOOLNAME}

#
# setting properties to the file systems hierarchy
#

echo ""
echo "Changing properties ${RPOOLNAME} "
echo ""

zfs set mountpoint=none ${RPOOLNAME}      # No mount point is assigned to the Root Pool
zfs set canmount=off ${RPOOLNAME}         # the Root Pool can't mount
zfs set checksum=fletcher4 ${RPOOLNAME}   # Improved algorithm error detection
zfs set atime=off ${RPOOLNAME}            # Disable the access time

#
# We create the Root of the file systems hierarchy
#

echo ""
echo "Creating ${RPOOLNAME}/ROOT"
echo ""

zfs create -o canmount=off ${RPOOLNAME}/ROOT

echo ""
echo "Creating ${BEPATH}"
echo ""

zfs create -o mountpoint=${ALT_ROOT} ${BEPATH}
zpool set bootfs=${BEPATH} ${RPOOLNAME}

#
# swap and /tmp
#

echo ""
echo "Creating ${RPOOLNAME}/swap"
echo ""

zfs create -V ${SWAP_GB} ${RPOOLNAME}/swap
zfs set org.freebsd:swap=on ${RPOOLNAME}/swap
zfs set checksum=off ${RPOOLNAME}/swap

echo ""
echo "Creating ${RPOOLNAME}/tmp"
echo ""

zfs create -o compression=on -o exec=on -o setuid=off -o mountpoint=${ALT_ROOT}/tmp ${RPOOLNAME}/tmp
chmod 1777 ${ALT_ROOT}/tmp

#
# /usr hierarchy
#

echo ""
echo "Creating ${BEPATH}/usr ${BEPATH}/usr/home"
echo ""

zfs create ${BEPATH}/usr
zfs create ${BEPATH}/usr/home

cd ${ALT_ROOT} ; ln -s ${ALT_ROOT}/usr/home home

zfs create -o compression=lzjb -o setuid=off             ${BEPATH}/usr/ports
zfs create -o compression=off  -o exec=off -o setuid=off ${BEPATH}/usr/ports/distfiles
zfs create -o compression=off  -o exec=off -o setuid=off ${BEPATH}/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off ${BEPATH}/usr/src

#
# /var hierarchy
#

echo ""
echo "Creating ${BEPATH}/var"
echo ""

zfs create ${BEPATH}/var
zfs create -o compression=lzjb -o exec=off -o setuid=off ${BEPATH}/var/crash
zfs create                     -o exec=off -o setuid=off ${BEPATH}/var/db
zfs create -o compression=lzjb -o exec=on  -o setuid=off ${BEPATH}/var/db/pkg
zfs create                     -o exec=off -o setuid=off ${BEPATH}/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off ${BEPATH}/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off ${BEPATH}/var/mail
zfs create                     -o exec=off -o setuid=off ${BEPATH}/var/run
zfs create -o compression=lzjb -o exec=on  -o setuid=off ${BEPATH}/var/tmp

chmod 1777 ${ALT_ROOT}/var/tmp

echo ""
echo "zpool status ${RPOOLNAME}"
echo ""

zpool status ${RPOOLNAME}

echo ""
echo "Mount Points"
echo ""

mount 

echo ""
echo "Ashift value should be 12 to indicate that it is aligned to 4K"
echo ""

zdb -C -U /var/tmp/zpool.cache | grep ashift

echo ""
echo "Type CTRL-D and begin the Installation"
echo ""

// End of Script mk_fbsd_rpool.sh

 

Script 3: post_install_fbsd_rpool.sh

These are the final steps of the installation process, you run right after installation is complete and before the first reboot.

// Start the Script post_install_fbsd_rpool.sh

#!/bin/sh

. ./env_fbsd_rpool.sh

zfs set readonly=on ${BEPATH}/var/empty
cp /var/tmp/zpool.cache ${ALT_ROOT}/boot/zfs/
ls -la ${ALT_ROOT}/boot/zfs/zpool.cache
touch ${ALT_ROOT}/etc/fstab

#
# The parameter vfs.root.mountfrom IS NOT INCLUDED BY INTENT
# because since revision r243502 the Root pool is automatically detected.
# You do not need to declare it in loader.conf
#

cat << EOF >> ${ALT_ROOT}/boot/loader.conf
zfs_load=YES
opensolaris_load="YES"
EOF

cat << EOF >> ${ALT_ROOT}/etc/rc.conf
zfs_enable=YES
EOF

echo ""
more ${ALT_ROOT}/boot/loader.conf
echo ""

echo ""
more ${ALT_ROOT}/etc/rc.conf
echo ""

zfs umount -a

echo ""
echo "Assigning definitive mounting points / /usr /tmp /var"
echo ""

zpool set cachefile='' ${RPOOLNAME}
zfs set mountpoint=legacy ${BEPATH}
zfs set mountpoint=/tmp ${RPOOLNAME}/tmp
zfs set mountpoint=/usr ${BEPATH}/usr
zfs set mountpoint=/var ${BEPATH}/var
reboot

// End of Script post_install_fbsd_rpool.sh

Pre-installation Tasks

Hardware compatibility

Before installing an OS we should verify that there no compatibility issues with the system hardware.

The most important tool, that we have to undertake this task, is the compatibility list all OSs available provides  and FreeBSD is no exception.

 

installation method

Unfortunately for us, it's impossible to install FreeBSD 9 using the GRUB2  method of boot from ISO images.

As installation methods, apart from the mount a network server installations, we have ISO image with full installation, an image ISO to pendrive and  a boot-only ISO to install from the FreeBSD internet servers.

In my case I will use a pendrive as installation media and preparing it from Linux, assuming you do not have another FreeBSD where support us.


1. - Downaload the memstick image for FreeBSD 9 amd64

2. - Proceed to completely overwrite the pendrive with zeros, assuming that your device to the flash drive is /dev/db.

# sudo  dd if=/dev/zero of=/dev/sdb bs=128M

3. - We transfer the image with the dd command.

# sudo dd if=FreeBSD-9.1-RELEASE-amd64-memstick.img  of=/dev/sdb  bs="10240" conv="sync"

4. - Optionally, mount the stick to check the contents before installing. 

# sudo mount -t ufs -o ufstype=44bsd,ro /dev/sdb /mnt

 

Preparing Partitions

This guide focuses on installing FreeBSD 9 with the root filesystem on ZFS without mirrors, in an environment with GRUB2 as the boot loader, partition table already defined and coexisting with other OSs.

If you want to install FreeBSD 9 as the exclusive OS, with its own bootloader, preparing the partition table varies substantially.

We started by reviewing the partition table, we choose the partition 8 as installation target,  change the partition type to FreeBSD ZFS -a504- and finally we added as a comment "freedsk1", that we use in scripts as the device name in /dev/gpt.

Step 0: Choosing the  Partition to Install

Do not forget to follow the cautions to install multiple OS; backup GPT table, make a screenshot with the list of the GPT partition table and backup important data.
 
Preparing the GRUB2 Boot Menu 
Normally this step would be part of the Post-Installation Tasks, as I use the Filesystem UUID to select the partition and it is generated after creating the file system during installation.

However, as we are using ZFS, we know in advance which is the name of the ZFS Pool; for GRUB2 this name is a label, so we can use it to set the grub.cfg file entry.

If a system is intended to install different OSs with root on ZFS is a good practice to use unique names for the Root Pools.

Otherwise, if we have several Root Pools with the same name to find the partition using GRUB2 command search; it return the first found, therefore, this method is useless and we would have to use the file system UUID for searching the partition.

 
The new entry in grub.cfg is the following:

/ / BEGIN OF EXCERPT

menuentry "FreeBSD 9 en freedsk1" {
        search --no-floppy --label --set=root fbsdzpool1
        kfreebsd /ROOT/91_30062013/@/boot/zfsloader
}

/ / END OF EXCERPT 

 

Transferring the Scripts

In order to use the ZFS configuration scripts for FreeBSD 9 is required to be present once booted the menstick, there are several possibilities, copying from another pendrive, configure the network and transfer them by ftp.

I will use the auxiliary partition that I have set up in anticipation of these needs.

Therefore, I mount the partition 19 and I copy into it the three scripts mentioned above.

Installation Steps

1. - Booting from pendrive

From the BIOS Boot menu we choose the pendrive to boot.

STEP 1a: We started the installation from Pendrive

From the Beastie menu, press Enter or wait for it to boot automatically.

Step 1b: Start FreeBSD 9

2. - Start Installation, basic configuration

The installer -bsdinstall- starts automatically from Bestie menu.

Step 2a: Initial Menu select Install

In the initial menu we choose the option Install and we perform the following steps.

1. Keyboard Configuration, I changed the default keyboard settings. choosing for Spanish "Spanish ISO-8859-15".
2. setting hostname.
3. Selecting System Components to Install , I choose  all.

For more detailed information see the HandBook.
 

3. - Partition Table Settings

At this point we are given to choose from to partition-Guided-guided, manual or go to the Shell.

STEP 3: In partitioning choose SHELL

We went to Shell.

From the Shell run the following commands :

# mount -t ext2fs -o ro /dev/ada0p19 /media
# cp/media/*.sh /var/tmp
# cd /var/tmp
# chmod 750 *.sh
#./mk_fbsd_rpool.sh >> mk_fbsd_rpool_log 2>&1

We check the log file.

# more mk_fbsd_rpool_log

// Begin of  log mk_fbsd_rpool_log

Initial BE name          : 91_30062013
Root Pool Name           : fbsdzpool1
Boot Enviroment Path     : fbsdzpool1/ROOT/91_30062013
swap size (GB)           : 4G
installation device      : freedsk1
installation mount point :/mnt
Creating pseudo-device freedsk1
Creating Pool fbsdzpool1
Deleting pseudo-device freedsk1
Changing properties Fbsdzpool1
Creating fbsdzpool1/ROOT
Creating fbsdzpool1/ROOT/91_30062013
Creating fbsdzpool1/swap
Creating fbsdzpool1/tmp
Creating fbsdzpool1/ROOT/91_30062013/usr fbsdzpool1/ROOT/91_30062013/usr/home
Creating fbsdzpool1/ROOT/91_30062013/var
zpool status fbsdzpool1
pool: fbsdzpool1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE cksum
fbsdzpool1 ONLINE 0 0 0
gpt/freedsk1 ONLINE 0 0 0
errors: No known data errors
Mount Points
/Dev/ ufs /FreeBSD_Install on / (ufs, local, noatime, read-only)
devfs on /dev (devfs, local, multilabel)
/Dev/md0 on /var (ufs, local)
/Dev/md1 on /tmp (ufs, local)
/Dev/ada0p19 on /media (ext2fs, local, read-only)
fbsdzpool1/ROOT/91_30062013 on / mnt (zfs, local, noatime, nfsv4acls)
fbsdzpool1/tmp on /mnt/tmp (zfs, local, noatime, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/usr on /mnt/usr (zfs, local, noatime, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/usr/home on /mnt /usr/home (zfs, local, noatime, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/usr/ports on  7mnt/ usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/usr/ports/distfiles on /mnt/usr/ports/distfiles (zfs, local, noatime, noexec, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/usr/ports/packages on /mnt/usr/ports/packages (zfs, local, noatime, noexec, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/usr/src on /mnt/usr/src (zfs, local, noatime, noexec, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/var on /mnt/var (zfs, local, noatime, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/var/crash on /mnt/var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/var/db on /mnt/var/db (zfs, local, noatime, noexec, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/var/db/pkg on /mnt/var/db/pkg (zfs, local, noatime, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/var/empty on /mnt/var/empty (zfs, local, noatime, noexec, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/var/log on /mnt/var/ og (zfs, local, noatime, noexec, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/var/mail on /mnt/var/mail (zfs, local, noatime, noexec, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/var/run on /mnt/var/run (zfs, local, noatime, noexec, nosuid, nfsv4acls)
fbsdzpool1/ROOT/91_30062013/var/tmp on /mnt/var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
Ashift value should be 12 to indicate that it is aligned to 4K
ashift 12
Type CTRL-D and begin the Installation

// End of  log mk_fbsd_rpool_log

STEP 3: Output in Console

4. - Base System Load

From the shell, we press CTRL-D and automatically starts charging the system.
 

Step 4: Load FreeBSD 9

5. - Additional Settings

Once loaded the system, bsdintall guides us through the following tasks.

1. Selecting the root pasword.
2. Configuring Network Devices
3. Regionalization.
4. Service Activation during startup.
5. Activating the memory dump-Crash Dump.
6. Creating users.

For more detailed information see the HandBook.

6. - End of Installation

Once concluded the previous step, HandBook describes them as Post-Installation Tasks, is presented a summary screen where you can make changes or finish; we choose Exit.
 

Step 6a: Final Review select Exit Setup

Before concluding the installation, bsdinstall asks if we want to open a shell to make changes to the new environment; ANSWER NO, it is an environment where the chroot command has been executed and is useless to us.

STEP 6b: Manual Configuration: Answer NO

Finally, bsdinstall asks if you want to restart the system or access the Live CD,  ANSWER  "Live CD"

STEP 6c: BSDINSTALL Final Step: Answer "Live CD"

Step 6d: Access the Live CD environment before restarting.

We login as root, right now is where we proceed to execute the third of the scripts that make ​​the last configuration steps and then reboot the system.

# umount /media
# cd /var/tmp
# ./post_install_fbsd_rpool.sh

 

Post-Installation Tasks

After booting the system for the first time, we proceed to make 2 steps I always do on my FreeBSD installations.

Updating the Ports Collection

First we update the Ports collection, these steps are only done the first time.

root@morsa:/root # portsnap fetch
root@morsa:/root # portsnap extract
root@morsa:/root # portsnap update

From this time to update the Ports Collection use:

root@morsa:/root # portsnap fetch update

Install portmaster

This tool simplifies the management of Ports collection.

We include support for the new package manager. 

root@morsa:/root # cd /usr/ports/ports-mgmt/pkg && make install clean
root@morsa:/root # cd /usr/ports/ports-mgmt/portmaster && make install clean

root@morsa:/root # echo 'WITH_PKGNG=yes' >> /etc/make.conf
root@morsa:/root # cp /usr/local/etc/portmaster.rc.sample /usr/local/etc/portmaster.rc
root@morsa:/root # pkg2ng

Edit the file portmaster.rc and check the values ​​of variables.

// BEGIN OF EXCERPT
# Always delete stale distfiles without prompting (-d)
ALWAYS_SCRUB_DISTFILES=dopt
# Be verbose (-v)
PM_VERBOSE=vopt
# Install packages for build-only dependencies (--packages-build)
PM_PACKAGES_BUILD=pmp_build
# Delete build-only dependencies when finished (--delete-build-only)
PM_DEL_BUILD_ONLY=pm_dbo
#Suppress the build confirmation message (--no-confirm)
PM_NO_CONFIRM=pm_no_confirm
/ / BEGIN OF EXCERPT

 

Install beadm

This installation example of root over ZFS is designed to fit beadm tool, with which we manage boot environments (Boot Enviroments).

1.- Install beadm with portmaster

root@morsa:/root # portmaster sysutils/beadm

2.-  create the first snapshot after the installation

root@morsa:/root # beadm create 91_30062013@intall_r243825

3.-  list the BEs and snapshots

root@morsa:/root # beadm list -s

BE/Dataset/Snapshot                                              Active Mountpoint            Space Created

91_30062013
  fbsdzpool1/ROOT/91_30062013                                    NR     /                    378.1M 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/usr                                -      /usr                 357.0M 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/usr/home                           -      /usr/home            144.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/usr/ports                          -      /usr/ports             1.4G 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/usr/ports/distfiles                -      /usr/ports/distfiles   1.7M 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/usr/ports/packages                 -      /usr/ports/packages  144.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/usr/src                            -      /usr/src             509.0M 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/var                                -      /var                 692.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/var/crash                          -      /var/crash           148.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/var/db                             -      /var/db              149.1M 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/var/db/pkg                         -      /var/db/pkg          280.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/var/empty                          -      /var/empty           144.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/var/log                            -      /var/log             316.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/var/mail                           -      /var/mail            152.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/var/run                            -      /var/run             452.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013/var/tmp                            -      /var/tmp             256.0K 2013-06-30 14:39
  fbsdzpool1/ROOT/91_30062013@intall_r243825                     -      -                    120.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/usr@intall_r243825                 -      -                      8.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/usr/home@intall_r243825            -      -                      0.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/usr/ports@intall_r243825           -      -                      8.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/usr/ports/distfiles@intall_r243825 -      -                      8.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/usr/ports/packages@intall_r243825  -      -                      0.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/usr/src@intall_r243825             -      -                      8.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/var@intall_r243825                 -      -                    124.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/var/crash@intall_r243825           -      -                      0.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/var/db@intall_r243825              -      -                    132.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/var/db/pkg@intall_r243825          -      -                      8.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/var/empty@intall_r243825           -      -                      0.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/var/log@intall_r243825             -      -                    108.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/var/mail@intall_r243825            -      -                      8.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/var/run@intall_r243825             -      -                    184.0K 2013-06-30 16:00
  fbsdzpool1/ROOT/91_30062013/var/tmp@intall_r243825             -      -                    100.0K 2013-06-30 16:0

 

Final Thoughts

This installation guide is aimed to optimize ZFS over modern disks, which are of 4K physical sectors, compared to traditional 512 bytes, although it still presented to the BIOS and OS as 512 byte sectors. 

If not taken this into account when aligning the partitions and create filesystems, is a performance degradation of I/O to disk.
 
Finally we will analyze briefly the ZFS hierarchy defined:
fbsdzpool1: The ZFS pool where FreeBSD BEs. resides.
fbsdzpool1/ROOT: It is the Root file system common to all BEs.
fbsdzpool1/swap: the swap area is defined directly under fbsdzpool1 and is common to all BEs.
fbsdzpool1/tmp: The file system /tmp is also common to all BEs.
fbsdzpool1/ROOT/91_30062013: It is the initial BE dond We install FreeBSD. From this point start the standard directories /, /usr, /usr/home, /var.
 
Each BE has its own copy of /usr/ports and /var. I do so because after a major version upgrade we must rebuild the ports collection as part of the upgrade process. Therefore if something goes wrong,  we need to rollback the Ports collection and installed  packages, not just the OS.

Another option, very popular, is that /var and /usr/ports directly begin from  fbsdzpool1 and so remain shared between all BEs. Sharing /var and /usr/ports is not supported  by the scripts in its current version.

In any case, /var and /usr/ports should go hand in hand, or are shared between BEs or are local to each BE, but both together.

It is also common to see the directory /usr/home shared between  BEs, beginning directly from fbsdzpool1. I prefer to have each BE with  his own copy of /usr/home. Shared /usr/home is not supported  by the scripts in its current version.