Netbooting FreeBSD installation CD

freebsd pxe

Last month, I started the project simple-pxe, which is a set of netboot setup scripts for USTC netboot service. To support UEFI netboot, I used GRUB2 instead of PXELINUX. I targeted to unify legacy PXE and UEFI netboot facilities of our university.

After finishing the scripts for Linux distros' LiveCD/installers, I tried to setup netboot for FreeBSD installation ISOs. There are many online tutorials about FreeBSD netboot, but most won't work in our enviroment. Our netboot facility should support multi-OS booting, so I cannot use FreeBSD-only PXE loader (pxeboot). And I have no control of our university's DHCP/BOOTP server, so I cannot push root-path options to FreeBSD loader.

Generally, Syslinux's MEMDISK utility can be used to boot ISO images. In GRUB2, following commands boot FreeBSD 11.2 installation image with MEMDISK:

linux16 MEMDISK
initrd16 FreeBSD-11.2-RELEASE-amd64-disc1.iso

GRUB2 loads everything to memory. For PXE booting, I can put the ISO file in the TFTP or HTTP directory, or even let GRUB2 download the ISO from FreeBSD official site.

However, MEMDISK seems to be incompatible with FreeBSD 12.0 image. The format of the ISO image is different with previous versions, and the kernel fails to detect CD device simulated by MEMDISK. Since I had no idea how to debug it, I moved to other solutions.

In fact, GRUB2 supports booting FreeBSD kernels with kfreebsd command. This post shows how to boot mfsBSD images (a FreeBSD-based LiveCD) from GRUB2. The core part is:

kfreebsd_module (loop)/mfsroot.gz type=mfs_root
set kFreeBSD.vfs.root.mountfrom="ufs:/dev/md0"

which specifies an image to be loaded at boot (mapped to /dev/md0) and use it as the root filesystem. More details about MFS root can be found in man 4 md and man 8 loader.

An UFS image can be used as MFS root, so what about an ISO image? It indeed works! I extracted FreeBSD kernel from the installation ISO, and provided the ISO file as MFS root:

kfreebsd ${freebsd_root}/boot/kernel/kernel
kfreebsd_loadenv ${freebsd_root}/boot/device.hints
kfreebsd_module ${freebsd_root}/cd.iso type=mfs_root
set kFreeBSD.vfs.root.mountfrom="cd9660:/dev/md0"
set kFreeBSD.vfs.root.mountfrom.options=ro

I have tested this method in PXE booting with FreeBSD 11.2 and 12.0 ISOs and found no issue.

Unfortunately, above method doesn't work in UEFI booting. GRUB2's kfreebsd command doesn't work well in UEFI mode. GRUB2's doc says that it supports headless booting of FreeBSD kernel in UEFI mode, but I just observed hung console and cannot find any signs that the kernel is running.

So I tried to chainload FreeBSD's EFI loader (boot/loader.efi) from GRUB2. In this way, GRUB2 cannot pass loader and kenv config to FreeBSD loader (as what kfreebsd_loadenv does. The EFI loader does support netboot. Its function is the same as FreeBSD's legacy PXE loader pxeboot. You need prepare a NFS FreeBSD root, and push the NFS root path setting to the loader with DHCP option root-path. However, as is stated in the begining, I have no control of DHCP server. Even if I can set the DHCP option, there is no way to push different root-path setting for different versions of FreeBSD images.

By observing behaviour of the EFI loader, I found that the default NFS path is set to the root of the TFTP server of netboot. The defination of default root path can be found in stand/libsa/globals.c of FreeBSD source code:

char    rootpath[FNAME_SIZE] = "/"

and rootpath is set to DHCP option root-path in stand/libsa/bootp.c:

if (tag == TAG_ROOTPATH) {
    if ((val = getenv("dhcp.root-path")) == NULL)
        val = (const char *)cp;
    strlcpy(rootpath, val, sizeof(rootpath));
}

So the default rootpath is hardcoded in the EFI loader. It came to me that I can directly patch the binary to change this default value.

First, let's find the location of rootpath variable. It is a 128-byte long string with first byte set to / and other bytes set to zero. GNU grep utility can be used:

$ grep -obUaP "/\x00{127}" boot/loader.efi
380576:/

There is only one appearance so I am sure it is rootpath. Then let's write our NFS path to the loader with dd utility:

printf '%s\x00' "/nfs/path/to/freebsd/root" \
    | dd of=boot/loader.efi bs=1 seek=380576 conv=notrunc
# If NFS server is not TFTP server, NFS path can be set in this way:
#   "ip:/nfs/path/to/freebsd/root"

Now I have a customized loader with my NFS path hardcoded. The loader will load kernel in this path and set initial root path to it.

Page created on 2019-02-06