Installing FreeBSD via serial console and PXE

— by Jeremy Chadwick <jdc@koitsu.org>

Choose one:

An author's preface...

Before I get started, I want to take a moment to remind readers of an obvious fact which, apparently, some have forgotten:

The year is 2010, not 1989.

FreeBSD was originally considered the "die-hard server operating system", which means it was intended for servers. It was used as such back when the 2.2.x series was available. Datacenters today are usually massive (read: have a very large number of servers), and require the need of 3 simple things, all of which are well-established and common:

Installation of an entire OS without physical access to the box
Heavy reliance on the network for installation (e.g. network install)
Use of high speed serial console (9600+bps) from start to finish to assist or automate said installation.

For sake of comparison, both Linux and Solaris (sparc and i386) have the above needs addressed, but FreeBSD trails behind. Accomplishing the above on FreeBSD requires the knowledge which I think many administrators lack — and that's not their fault. Things like:

Intricate knowledge of how the FreeBSD multi-stage (boot0/boot1/boot2/loader) bootstrap behaves, and what pxeboot(8) really is
Rebuilding above bootstraps with custom options during compile
Implementation of said bootstraps into PXE-based installation environments
Knowledge of how TFTP and DHCP work, and how to debug them if they break,
Familiarity with NFS (configuration and security awareness), and any one-off quirks that pertain to a specific FreeBSD release/version
Familiarity with tftpd (specifically permissions quirks it has in its code
Intricate knowledge of configuring a DHCP server (common question: "what's the 'next-server' and 'option root-path' stuff? Is it needed? Why?")
Disk label/GEOM changes (particular to 8.x)
Caveats/bugs with all of the above, and more. Examples: how out-of-the-box FreeBSD sets the maximum serial port speed to 9600bps, not being able to do a complete 100% TFTP-based (e.g. no NFS) install, security issues surrounding use of rpcbind/mountd/nfsd (daemons binding to INADDR_ANY despite use of -h, using arbitrary port numbers which make it difficult to firewall off, etc.), undocumented pxeboot(8) options which are more or less and many other things.

If you're not "acceptably" familiar with all of the above, you'll end up smashing your head against your monitor for days before posting to a mailing list — only to hear leaves rustle, or possibly someone saying "me too!" Most administrators do not know how to program, and even if they did (like myself), lack knowledge of the inner-workings of the above. Therefore, we can't realistically expect administrators of systems to provide patches or enhancements to make this process easier (for sake of comparison, it took me 7 hours just to get it all up and working) — we can only expect them to report the problems and hope someone more technical steps up to the plate.

With the introduction of FreeBSD 8.x into the fray, things became more irritating. The procedure for 8.x is more or less the same as 7.x, except there are numerous things which will catch you off-guard. For example, I received a couple of reports from other administrators following my 7.x guide for 8.x, but were running into issues. It wasn't until I tried it myself on 8.x that I was able to share their pain, thus working out the quirks.

I hope the above documents help anyone/everyone who is looking to do a remote installation of present-day FreeBSD via PXE using serial console rather than over VGA.

Itemised list of annoyances with FreeBSD installs with serial console

The below is an itemised list of the (unacceptable) annoyances or nuances that an administrator has to deal with when trying to install FreeBSD remotely with serial console. These items sadden/depress me greatly, given that FreeBSD has an established history and is toted (socially) as "a server OS". Comparatively, well-established Linux distributions (ex. Ubuntu) have addressed these shortcomings long ago, both at the bootloader level and the kernel level. Likewise, DragonflyBSD has also addressed these shortcomings.

Serial port speed in the boot blocks and the kernel is "hard locked" at 9600bps and cannot be increased unless boot blocks and kernel are rebuilt for a higher speed. Meaning: an administrator cannot boot a stock FreeBSD ISO and achieve more than 9600bps without rebuilding the boot blocks and kernel.
Belief in the developer/user community that setting comconsole_speed in loader.conf is sufficient; it is not. For PXE-based installs, admins absolutely want serial output as soon as possible in the boot sequence (read: within pxeldr), which is especially useful when debugging/troubleshooting why PXE booting may not work correctly. Hence, boot blocks must be rebuilt to achieve this functionality early on.
During post-install configuration, there is no way to adjust contents of /boot.config. This is needed by administrators to add flags like -S115200 -D -h to the file to switch to serial-based console as early as possible during the boot-up. Instead, admins are expected to drop their serial port software or console server down to 9600bps temporarily, just so they can log in and create the file.
During post-install configuration, there is no way to adjust contents of /boot/loader.conf. This is often needed by administrators to add lines like ahci_load="yes", otherwise the admin has to remember to do this twice (during the initial boot of the ISO/PXE setup, and again once the OS is installed on-disk.
No way to drop to an "Emergency Shell" within the installer when using a serial port (there's no way to press Alt-F4 via serial console) -- and even if one could, the "Emergency Shell" contains zero useful utilities for accomplishing simple tasks. Both of these shortcomings are addressed in FreeBSD 9.x where sysinstall has been replaced entirely with bsdinstall (and is therefore now a menu item/option during start-up). I have no experience as of this writing with FreeBSD 9.x and serial-based installs, however, so whether or not this works reliably via serial console is unknown to me.

It is my personal and professional belief that an administrator should be able to download a standard FreeBSD image (think ISO/memstick), PXE boot it, and be able to accomplish an entire OS install via serial console, with the end result being a fully working system with 115200bps serial console. Physical access, or manual power-cycling of the system, should not be required at any point to accomplish this task.