Netboot PXE Live CD Multiple Releases

This article details how to configure a stream-lined net-boot server for PXE clients using DHCP, TFTP and NFS to run live-CD images, custom-debugging, and diskless clients. Network clients can use PXE to boot using files on a server. The aim is to loop-mount the ISO images so that both TFTP and NFS have access to them.

Because TFTP uses a chroot jail the mounts are done inside the jail and then bound (mount --bind) to locations in the NFS server tree. TFTP needs access to the (compressed) kernel image vmlinuz and initial RAM-disk image initrd.gz. It also needs a minimum of three PXE boot files: the boot-strap code (pxelinux.0), the menu code (menu.c32), and the menu text configuration (pxelinux.cfg/default).

NFS needs access to all other files from the CD/installed images.

In this article the sub-net DHCP server is on 10.254.251.1 (WRT54GL running dd-wrt). The TFTP/NFS server is 10.254.251.52. The net-boot client will get a dynamic IP in the range 10.254.251.51 through 10.254.251.199.

DHCP options

DHCP and the Bootp protocols provide an IP address for a client to network boot (performed by client's PC BIOS).

DHCP Daemon

DHCP and TFTP on Same Server

The TFTP server is running on the same host and interface as the DHCP/Bootp server:

subnet 10.254.251.0 netmask 255.255.255.0 {
    range 10.254.251.51 10.254.251.199;
    filename "pxelinux.0";
}

TFTP on Separate Server

The TFTP server is installed on 10.254.251.52 which is a separate server:

subnet 10.254.251.0 netmask 255.255.255.0 {
    range 10.254.251.51 10.254.251.199;
    filename "pxelinux.0";
    next-server 10.254.251.52;
}

Dnsmasq

Provide the file-name and IP address of the TFTP server where the client will get the boot images.

dhcp-boot=pxelinux.0,,10.254.251.52

Note: If using dd-wrt on a router and adding this setting via the web interface, it should be added in the Additional DNSMasq Options text-area not the Additional DHCPd Options text-area.

TFTP Server

sudo apt-get install tftpd-hpa

The network-accessible files will be installed in /var/lib/tftpboot/.

PXE Configuration

Executables

Copy the PXE boot-loader and menu binary executables from the syslinux package (the package may need installing first):

sudo apt-get install syslinux
sudo cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/
sudo cp /usr/lib/syslinux/menu.c32 /var/lib/tftpboot/

Menu

Create a sub-directory

sudo mkdir -p /var/lib/tftpboot/pxelinux.cfg

and create the file /var/lib/tftpboot/pxelinux.cfg/default with an entry for each bootable image the PXE server will offer:

Note: The contents of indented lines following each append should be all on one line in the file. They are wrapped here to prevent ultra-long lines and scrolling to read.

DEFAULT menu.c32
TIMEOUT 600
ONTIMEOUT localboot
MENU TITLE PXE Network Boot

LABEL localboot
        MENU LABEL ^Local Boot (from CD or Hard Disk)
        MENU DEFAULT
        LOCALBOOT 0
        TIMEOUT 600

LABEL 9.04 Jaunty 32-bit Live
        kernel live/ubuntu-9.04-desktop-i386/casper/vmlinuz
        append boot=casper netboot=nfs
          initrd=live/ubuntu-9.04-desktop-i386/casper/initrd.gz
          nfsroot=10.254.251.52:/srv/boot/live/ubuntu-9.04-desktop-i386 --

LABEL 9.04 Jaunty 64-bit Live
        kernel live/ubuntu-9.04-desktop-amd64/casper/vmlinuz
        append boot=casper netboot=nfs
          initrd=live/ubuntu-9.04-desktop-amd64/casper/initrd.gz
          nfsroot=10.254.251.52:/srv/boot/live/ubuntu-9.04-desktop-amd64 --

LABEL 8.10 Intrepid 32-bit Live
        kernel live/ubuntu-8.10-desktop-i386/vmlinuz
        append boot=casper netboot=nfs
          initrd=live/ubuntu-8.10-desktop-i386/initrd.gz
          nfsroot=10.254.251.52:/srv/boot/live/ubuntu-8.10-desktop-i386 --

LABEL 8.10 Intrepid 64-bit Live
        kernel live/ubuntu-8.10-desktop-amd64/vmlinuz
        append boot=casper netboot=nfs
          initrd=live/ubuntu-8.10-desktop-amd64/initrd.gz
          nfsroot=10.254.251.52:/srv/boot/live/ubuntu-8.10-desktop-amd64 --

LABEL 8.04 Hardy 32-bit Live
        kernel live/ubuntu-8.04-desktop-i386/vmlinuz
        append boot=casper netboot=nfs
          initrd=live/ubuntu-8.04-desktop-i386/initrd.gz
          nfsroot=10.254.251.52:/srv/boot/live/ubuntu-8.04-desktop-i386 ---

LABEL 8.04 Hardy 64-bit Live
        kernel live/ubuntu-8.04-desktop-amd64/vmlinuz
        append boot=casper netboot=nfs
          initrd=live/ubuntu-8.04-desktop-amd64/initrd.gz
          nfsroot=10.254.251.52:/srv/boot/live/ubuntu-8.04-desktop-amd64 --

LABEL 7.10 Gutsy 32-bit Live
        kernel live/ubuntu-7.10-desktop-i386/casper/vmlinuz
        append root=/dev/nfs boot=casper netboot=nfs
          initrd=live/ubuntu-7.10-desktop-i386/casper/initrd.gz quiet splash
          nfsroot=10.254.251.52:/srv/boot/live/ubuntu-7.10-desktop-i386 --

If a PC (or group of PCs) requires a customised set of options a menu can be created where the menu file-name matches all or part of (in order) the BIOS GUID, MAC address, or IP address. See /usr/share/doc/syslinux/pxelinux.txt.gz for details. Here's an example:

If the boot file name is /var/lib/tftpboot/pxelinux.0, the UUID is b8945908-d6a6-41a9-611d-74a6ab80b83d, the Ethernet MAC address is 88:99:AA:BB:CC:DD and the IP address 192.0.2.91 (192.0.2.91 == hexadecimal C000025B), it will try:

        /var/lib/tftpboot/pxelinux.cfg/b8945908-d6a6-41a9-611d-74a6ab80b83d
        /var/lib/tftpboot/pxelinux.cfg/01-88-99-aa-bb-cc-dd
        /var/lib/tftpboot/pxelinux.cfg/C000025B
        /var/lib/tftpboot/pxelinux.cfg/C000025
        /var/lib/tftpboot/pxelinux.cfg/C00002
        /var/lib/tftpboot/pxelinux.cfg/C0000
        /var/lib/tftpboot/pxelinux.cfg/C000
        /var/lib/tftpboot/pxelinux.cfg/C00
        /var/lib/tftpboot/pxelinux.cfg/C0
        /var/lib/tftpboot/pxelinux.cfg/C
        /var/lib/tftpboot/pxelinux.cfg/default

CD Images

To avoid needing to extract ISO CD/DVD images they can be mounted directly into the file system. The paths must match what is specified in the PXE menu entries.

In my system all the downloaded ISOs are kept together, along with exploded images in sub-directories that can be altered for debugging:

ls -1 /home/all/Downloads/Ubuntu-CD-Images/
ubuntu-8.04-desktop-amd64.iso
ubuntu-8.04-desktop-i386.iso
ubuntu-8.10-desktop-amd64.iso
ubuntu-8.10-desktop-i386.iso
ubuntu-9.04-desktop-amd64.iso
ubuntu-9.04-desktop-i386.iso
ubuntu-9.04-desktop-i386-DEBUG

Because the TFTP daemon runs in a chroot jail the ISO images must be mounted as descendants of /var/lib/tftpboot/.

Manual Mount/Export Creation

Create a holding directory:

sudo mkdir -p /var/lib/tftproot/live

Mount each bootable CD image:

sudo mkdir -p /var/lib/tftproot/live/ubuntu-9.04-desktop-i386
sudo mount -t iso9660 -o loop /home/all/Downloads/Ubuntu-CD-Images/ubuntu-9.04-desktop-i386.iso /var/lib/tftpboot/live/ubuntu-9.04-desktop-i386

NFS Server Configuration

It is recommended to use the NFS server built into the Linux kernel so install the user-space administration package:

sudo apt-get install nfs-kernel-server

Create directories that will be used as mount-points for other paths (this avoids putting files in an unusual location or using unusual long paths in NFS mount references):

sudo mkdir -p /srv/boot/live

Link this directory to the TFTP daemon's chroot jail:

sudo mount --rbind /var/lib/tftpboot/live /srv/boot/live

Note: Use --rbind here since the sub-mount(s) of /var/lib/tftpboot/live/ need to be exposed (--bind won't expose sub-mounts).

Export the file-systems for each mount --rbind:

sudo exportfs -i -o async,no_root_squash,no_subtree_check,ro 0.0.0.0/0.0.0.0:/srv/boot/live/ubuntu-9.04-desktop-i386  

Network Boot

Automated Start/Stop Script

The TFTP and NFS exports can be done automatically using my attached pxe-prepare.sh script. The script is called with either the start or stop command and will create or remove the mount and export for each ISO image or sub-directory in the CD image root directory.

sudo pxe-prepare.sh start
# PXE/NFS net boot available
sudo pxe-prepare.sh stop

Client Boot

On the client PC enable PXE net-boot in BIOS. Restart. The BIOS will either ask for a particular key to be pressed to attempt a net-boot or will try it as part of the boot order.

The PC will report the progress of the PXE boot attempt. If successful the PXE menu will be shown. Selecting an option will cause the kernel image (vmlinuz) and the initial ram-disk image (initrd.gz) to be fetched. The casper scripts will handle the mounting of the NFS server CD image at /cdrom and will then proceed to boot. It can take some time if the network is slow (e.g. a WiFi? network).

Netconsole Debugging

Sometimes it is useful to capture the kernel log messages during boot-time. If the client and manager PC have serial ports (many PCs do not) a null-modem cable can be used.

However, seeing as the PC is booting over the network it makes sense to make use of the kernel's netconsole facility to echo the messages to another PC. That could be a syslog server or simply the netcat process.

Log Server

Use netcat to listen for UDP packets on the default netconsole target port (6666). Optionally specify the server's interface to listen on (useful when multi-homed):

nc -vv -u -l -p 6666 -s 10.254.251.52

Modified live-CD Initial RAM-disk Image

Now the live-CD's casper/initrd.gz needs netconsole adding to the list of modules it should load. The 'regular' way would be to add module entries to the initrd /conf/modules file and ensure the corresponding modules are in its /lib/modules/$(uname -r)/ directory.

However, I've been experimenting with an alternative that alters the initrd /init script so that both prerequisite modules (network device drivers) and netconsole itself can be demand-loaded via the kernel command-line. I'd like to get the mechanism accepted into the initramfs-tools package so that future live-CD's (and installed systems) have this additional flexibility.

For now, which-ever mechanism is used, the initrd image has to be expanded, altered, and rebuilt.

Mount ISO Image

Mount the ISO CD image into the file-system:

sudo mkdir /mnt/live-cd
sudo mount -t iso9660 -o loop /home/all/Downloads/Ubuntu-CD-Images/ubuntu-9.04-desktop-i386.iso /mnt/live-cd

Create a Working Copy

The image needs to be altered so a working copy will be used.

mkdir /home/all/Downloads/Ubuntu-CD-Images/ubuntu-9.04-desktop-i386-DEBUG
cp -a /mnt/live-cd/* /home/all/Downloads/Ubuntu-CD-Images/ubuntu-9.04-desktop-i386-DEBUG/

Extract Initrd

Create a temporary location and extract the image contents:

mkdir /tmp/initrd
cd /tmp/initrd

For initrd images compressed using gzip + tar (pre-Karmic 9.10):

zcat /home/all/Downloads/Ubuntu-CD-Images/ubuntu-9.04-desktop-i386-DEBUG/casper/initrd.gz | cpio -i -d

For initrd images compressed using lzma (Karmic 9.10 and later):

lzma -dc -S .lz /home/all/Downloads/Ubuntu-CD-Images/ubuntu-10.04-desktop-i386-DEBUG/casper/initrd.lz | cpio -id

Patch to /init Script

Download the init.patch file attached to this article. It adds an additional kernel command-line option (essential=) and detects the netconsole= option and when found, loads the module with the given parameters.

cd /tmp
wget -N http://tjworld.net/raw-attachment/wiki/Linux/Ubuntu/NetbootPxeLiveCDMultipleReleases/init.patch

Apply it using:

cd /tmp/initrd
patch -p3 < /tmp/init.patch

Ensure Network Module is Included

This step is dependent on the network device in the PC. First identify the network chip-set and from that the kernel module that drives it. Check the module is included in the initrd image. For example, the Intel e100 is used by the PC so:

cd /tmp/initrd
find ./lib/modules/ -name 'e100*'

./lib/modules/2.6.28-9-generic/kernel/drivers/net/e100.ko

If the kernel module isn't in the initrd image copy it from an installed system that uses the same kernel version or mount the live-CD's squashfs file-system and copy it from there:

mkdir /mnt/squash
sudo mount -t squashfs -o loop /home/all/Downloads/Ubuntu-CD-Images/ubuntu-9.04-desktop-i386-DEBUG/casper/filesystem.squashfs /mnt/squash

Now it is possible to locate the module. Let's use e100 again as an example - in practice this step is only needed if the module isn't in the initrd image:

find /mnt/squash/lib/modules -name 'e100.ko'

/mnt/squash/lib/modules/2.6.28-9-generic/kernel/drivers/net/e100.ko

cp /mnt/squash/lib/modules/2.6.28-9-generic/kernel/drivers/net/e100.ko /tmp/initrd/lib/modules/2.6.28-9-generic/kernel/drivers/net/e100.ko

sudo umount /mnt/squash

Rebuild Initrd

Collect the list of file names, re-create the cpio archive and compress it.

cd /tmp/initrd

For initrd compressed with gzip (pre-Karmic 9.10):

find . | cpio -o -H newc | gzip > /tmp/initrd.gz

For initrd compressed with lzma (Karmic 9.10 and later):

find . | cpio --dereference -o -H newc | lzma -7 > /tmp/initrd.lz

Copy Back to live-CD Working Copy

Replace the original (pre-Karmic 9.10):

cp /tmp/initrd.gz /home/all/Downloads/Ubuntu-CD-Images/ubuntu-9.04-desktop-i386-DEBUG/casper/

or (for Karmic 9.10 and later):

cp /tmp/initrd.lz /home/all/Downloads/Ubuntu-CD-Images/ubuntu-10.04-desktop-i386-DEBUG/casper/

Modified PXE Boot Menu Entry

The netconsole parameters need adding to the kernel command line. This could be done manually by editing the menu at boot-time but it makes more sense to add a custom debug entry to the PXE menu.

With the additional essential= parameter it is possible to ensure the network device driver is loaded before netconsole starts - a requirement for netconsole to work. In this example the Intel e100 driver is being loaded. When using netconsole it is useful to increate the verbosity of kernel messages using debug (Note: when using debug the upstart /init script writes a boot-log to /dev/.initramfs/initramfs.debug and /casper.log - copied to /var/log/ when the root file-system is ready).

Add an entry to /var/lib/tftpboot/pxelinux.cfg/default:

LABEL 9.04 Jaunty 32-bit Live DEBUG netconsole
        kernel live/ubuntu-9.04-desktop-i386-DEBUG/casper/vmlinuz
        append debug boot=casper netboot=nfs
          initrd=live/ubuntu-9.04-desktop-i386-DEBUG/casper/initrd.gz
          nfsroot=10.254.251.52:/srv/boot/live/ubuntu-9.04-desktop-i386-DEBUG
          essential=e100 netconsole=@10.254.251.95/,@10.254.251.52/ --

The format is netconsole=[local-port]@[local-ip-address]/[interface],[target-port]@<target-ip-address>/[target-mac-address] (<required>, [optional]).

Note: Although netconsole will use the default local-ip-address 0.0.0.0 if one isn't given, it doesn't appear to work directly or with broadcast unless an IP address is specified. This can be a problem if multiple PCs might use the PXE netconsole facility and may require manual intervention at the PXE boot menu by the user to set a free IP address.