Netboot PXE Live CD Multiple Releases

This article details how to configure a stream-lined net-boot server for PXE clients using DHCP, TFTP and NFS to run live-CD images, custom-debugging, and diskless clients. Network clients can use PXE to boot using files on a server. The aim is to loop-mount the ISO images so that both TFTP and NFS have access to them.

It also shows how to configure net-boot PowerPC ISO images for Apple iMac/iBook NewWorld PowerPC using a modified yaboot (Yet Another Boot loader) that supports kernel and initrd images larger than 6MB. (This section will be added soon).

Because TFTP uses a chroot jail the mounts are done inside the jail and then bound (mount --bind) to locations in the NFS server tree. TFTP needs access to the (compressed) kernel image vmlinuz and initial RAM-disk image initrd.gz. It also needs a minimum of three PXE boot files: the boot-strap code (pxelinux.0), the menu code (menu.c32), and the menu text configuration (pxelinux.cfg/default).

NFS needs access to all other files from the CD/installed images.

In this article the sub-net DHCP server is on 10.254.251.1 (WRT54GL running dd-wrt). The TFTP/NFS server is 10.254.251.2. The net-boot client will get a dynamic IP in the range 10.254.251.51 through 10.254.251.199.

DHCP options

DHCP and the Bootp protocols provide an IP address for a client to network boot (performed by client's PC BIOS).

DHCP Daemon

DHCP and TFTP on Same Server

The TFTP server is running on the same host and interface as the DHCP/Bootp server:

subnet 10.254.251.0 netmask 255.255.255.0 {
    range 10.254.251.51 10.254.251.199;
    filename "pxelinux.0";
}

TFTP on Separate Server

The TFTP server is installed on 10.254.251.52 which is a separate server:

subnet 10.254.251.0 netmask 255.255.255.0 {
    range 10.254.251.51 10.254.251.199;
    filename "pxelinux.0";
    next-server 10.254.251.2;
}

Dnsmasq

Provide the file-name and IP address of the TFTP server where the client will get the boot images.

dhcp-boot=pxelinux.0,,10.254.251.2

Note: If using dd-wrt on a router and adding this setting via the web interface, it should be added in the Additional DNSMasq Options text-area not the Additional DHCPd Options text-area.

TFTP Server

sudo apt-get install tftpd-hpa

The network-accessible files will be installed in /var/lib/tftpboot/.

PXE Configuration

Executables

Copy the PXE boot-loader and menu binary executables from the syslinux package (the package may need installing first):

sudo apt-get install syslinux
sudo cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/
sudo cp /usr/lib/syslinux/menu.c32 /var/lib/tftpboot/

Menu

Create a sub-directory

sudo mkdir -p /var/lib/tftpboot/pxelinux.cfg

and create the file /var/lib/tftpboot/pxelinux.cfg/default with an entry for each bootable image the PXE server will offer:

Note: The contents of indented lines following each append should be all on one line in the file. They are wrapped here to prevent ultra-long lines and scrolling to read.

DEFAULT menu.c32
TIMEOUT 600
ONTIMEOUT localboot
MENU TITLE PXE Network Boot

LABEL localboot
        MENU LABEL ^Local Boot (from CD or Hard Disk)
        MENU DEFAULT
        LOCALBOOT 0
        TIMEOUT 600

LABEL 10.04 Lucid x86 32-bit Ubuntu Alternate
    kernel iso-image/ubuntu-10.04-alternate-i386/install/netboot/ubuntu-installer/i386/linux
    append netboot=nfs nfsroot=10.254.251.2:/srv/boot/iso-image/ubuntu-10.04-alternate-i386 initrd=iso-image/ubuntu-10.04-alternate-i386/install/netboot/ubuntu-installer/i386/initrd.gz 
LABEL 10.04 Lucid x86 64-bit Ubuntu Live
    kernel iso-image/ubuntu-10.04-desktop-amd64/casper/vmlinuz
    append boot=casper netboot=nfs nfsroot=10.254.251.2:/srv/boot/iso-image/ubuntu-10.04-desktop-amd64 initrd=iso-image/ubuntu-10.04-desktop-amd64/casper/initrd.lz 
LABEL 10.04 Lucid x86 32-bit Ubuntu Live DEBUG netconsole
    kernel iso-image/ubuntu-10.04-desktop-i386-DEBUG/casper/vmlinuz
    append boot=casper netboot=nfs nfsroot=10.254.251.2:/srv/boot/iso-image/ubuntu-10.04-desktop-i386-DEBUG initrd=iso-image/ubuntu-10.04-desktop-i386-DEBUG/casper/initrd.lz essential=e100 netconsole=@10.254.251.95/,@10.254.251.2/
LABEL 10.04 Lucid x86 32-bit Ubuntu Live
    kernel iso-image/ubuntu-10.04-desktop-i386/casper/vmlinuz
    append boot=casper netboot=nfs nfsroot=10.254.251.2:/srv/boot/iso-image/ubuntu-10.04-desktop-i386 initrd=iso-image/ubuntu-10.04-desktop-i386/casper/initrd.lz 
LABEL 10.04 Lucid x86 32-bit Xubuntu Alternate
    kernel iso-image/xubuntu-10.04-alternate-i386/install/netboot/ubuntu-installer/i386/linux
    append netboot=nfs nfsroot=10.254.251.2:/srv/boot/iso-image/xubuntu-10.04-alternate-i386 initrd=iso-image/xubuntu-10.04-alternate-i386/install/netboot/ubuntu-installer/i386/initrd.gz 
LABEL 10.04 Lucid x86 32-bit Xubuntu Live
    kernel iso-image/xubuntu-10.04-desktop-i386/casper/vmlinuz
    append boot=casper netboot=nfs nfsroot=10.254.251.2:/srv/boot/iso-image/xubuntu-10.04-desktop-i386 initrd=iso-image/xubuntu-10.04-desktop-i386/casper/initrd.lz 

If a PC (or group of PCs) requires a customised set of options a menu can be created where the menu file-name matches all or part of (in order) the BIOS GUID, MAC address, or IP address. See /usr/share/doc/syslinux/pxelinux.txt.gz for details. Here's an example:

If the boot file name is /var/lib/tftpboot/pxelinux.0, the UUID is b8945908-d6a6-41a9-611d-74a6ab80b83d, the Ethernet MAC address is 88:99:AA:BB:CC:DD and the IP address 192.0.2.91 (192.0.2.91 == hexadecimal C000025B), it will try:

        /var/lib/tftpboot/pxelinux.cfg/b8945908-d6a6-41a9-611d-74a6ab80b83d
        /var/lib/tftpboot/pxelinux.cfg/01-88-99-aa-bb-cc-dd
        /var/lib/tftpboot/pxelinux.cfg/C000025B
        /var/lib/tftpboot/pxelinux.cfg/C000025
        /var/lib/tftpboot/pxelinux.cfg/C00002
        /var/lib/tftpboot/pxelinux.cfg/C0000
        /var/lib/tftpboot/pxelinux.cfg/C000
        /var/lib/tftpboot/pxelinux.cfg/C00
        /var/lib/tftpboot/pxelinux.cfg/C0
        /var/lib/tftpboot/pxelinux.cfg/C
        /var/lib/tftpboot/pxelinux.cfg/default

CD Images

To avoid needing to extract ISO CD/DVD images they can be mounted directly into the file system. The paths must match what is specified in the PXE menu entries.

In my system all the downloaded ISOs are kept together. I use separate sub-directories for the different Ubuntu sub-projects such as Xubuntu (this is especially helpful for the current development images since the ISO names for sub-project ISOs can be the same as the Ubuntu ISOs) and use symbolic-links in the top-most directory, along with extracted images in sub-directories that can be altered for debugging:

/home/all/iso-image/ubuntu/
/home/all/iso-image/xubuntu/
/home/all/iso-image/ubuntu-10.04-desktop-i386-DEBUG/
/home/all/iso-image/ubuntu-10.04-alternate-i386.iso -> ubuntu/ubuntu-10.04-alternate-i386.iso
/home/all/iso-image/ubuntu-10.04-alternate-powerpc.iso -> ubuntu/ubuntu-10.04-alternate-powerpc.iso
/home/all/iso-image/ubuntu-10.04-desktop-amd64.iso -> ubuntu/ubuntu-10.04-desktop-amd64.iso
/home/all/iso-image/ubuntu-10.04-desktop-i386.iso -> ubuntu/ubuntu-10.04-desktop-i386.iso
/home/all/iso-image/ubuntu-10.04-desktop-powerpc.iso -> ubuntu/ubuntu-10.04-desktop-powerpc.iso
/home/all/iso-image/xubuntu-10.04-alternate-i386.iso -> xubuntu/xubuntu-10.04-alternate-i386.iso
/home/all/iso-image/xubuntu-10.04-desktop-i386.iso -> xubuntu/xubuntu-10.04-desktop-i386.iso

Because the TFTP daemon runs in a chroot jail the ISO images must be mounted as descendants of /var/lib/tftpboot/.

Manual Mount/Export? Creation

Create a holding directory:

sudo mkdir -p /var/lib/tftproot/iso-image

Mount each bootable CD image:

sudo mkdir -p /var/lib/tftproot/iso-image/ubuntu-10.04-desktop-i386
sudo mount -t iso9660 -o loop /home/all/iso-image/ubuntu-10.04-desktop-i386.iso /var/lib/tftpboot/iso-image/ubuntu-10.04-desktop-i386

NFS Server Configuration

It is recommended to use the NFS server built into the Linux kernel so install the user-space administration package:

sudo apt-get install nfs-kernel-server

Create directories that will be used as mount-points for other paths (this avoids putting files in an unusual location or using unusual long paths in NFS mount references):

sudo mkdir -p /srv/boot/iso-image

Link this directory to the TFTP daemon's chroot jail:

sudo mount --rbind /var/lib/tftpboot/iso-image /srv/boot/iso-image

Note: Use --rbind here since the sub-mount(s) of /var/lib/tftpboot/iso-image/ need to be exposed (--bind won't expose sub-mounts).

Export the file-systems for each mount --rbind:

sudo exportfs -i -o async,no_root_squash,no_subtree_check,ro 0.0.0.0/0.0.0.0:/srv/boot/iso-image/ubuntu-10.04-desktop-i386  

Network Boot

Automated Start/Stop? Script

The TFTP and NFS exports can be done automatically using my attached pxe-prepare script . The script is called with either the start or stop command and will create or remove the mount and export for each ISO image or sub-directory in the CD image root directory but only when there is an associated .pxelabel file.

 sudo pxe-prepare start
stop: Unknown instance: 
mounting ISO ubuntu-10.04-alternate-i386
 TFTP: iso-image/ubuntu-10.04-alternate-i386
 NFS:  10.254.251.2/srv/boot/iso-image/ubuntu-10.04-alternate-i386
mounting ISO ubuntu-10.04-alternate-powerpc
 TFTP: iso-image/ubuntu-10.04-alternate-powerpc
 NFS:  10.254.251.2/srv/boot/iso-image/ubuntu-10.04-alternate-powerpc
mounting ISO ubuntu-10.04-desktop-amd64
 TFTP: iso-image/ubuntu-10.04-desktop-amd64
 NFS:  10.254.251.2/srv/boot/iso-image/ubuntu-10.04-desktop-amd64
mounting directory ubuntu-10.04-desktop-i386-DEBUG
 TFTP: iso-image/ubuntu-10.04-desktop-i386-DEBUG
 NFS:  10.254.251.2/srv/boot/iso-image/ubuntu-10.04-desktop-i386-DEBUG
mounting ISO ubuntu-10.04-desktop-i386
 TFTP: iso-image/ubuntu-10.04-desktop-i386
 NFS:  10.254.251.2/srv/boot/iso-image/ubuntu-10.04-desktop-i386
mounting ISO ubuntu-10.04-desktop-powerpc
 TFTP: iso-image/ubuntu-10.04-desktop-powerpc
 NFS:  10.254.251.2/srv/boot/iso-image/ubuntu-10.04-desktop-powerpc
mounting ISO xubuntu-10.04-alternate-i386
 TFTP: iso-image/xubuntu-10.04-alternate-i386
 NFS:  10.254.251.2/srv/boot/iso-image/xubuntu-10.04-alternate-i386
mounting ISO xubuntu-10.04-desktop-i386
 TFTP: iso-image/xubuntu-10.04-desktop-i386
 NFS:  10.254.251.2/srv/boot/iso-image/xubuntu-10.04-desktop-i386
tftpd-hpa start/running, process 7248

sudo pxe-prepare stop
unmounting ubuntu-10.04-alternate-i386
unmounting ubuntu-10.04-alternate-powerpc
unmounting ubuntu-10.04-desktop-amd64
unmounting ubuntu-10.04-desktop-i386
unmounting ubuntu-10.04-desktop-i386-DEBUG
unmounting ubuntu-10.04-desktop-powerpc
unmounting xubuntu-10.04-alternate-i386
unmounting xubuntu-10.04-desktop-i386
tftpd-hpa stop/waiting

Client Boot

On the client PC enable PXE net-boot in BIOS. Restart. The BIOS will either ask for a particular key to be pressed to attempt a net-boot or will try it as part of the boot order.

The PC will report the progress of the PXE boot attempt. If successful the PXE menu will be shown. Selecting an option will cause the kernel image (vmlinuz) and the initial ram-disk image (initrd.gz) to be fetched. The casper scripts will handle the mounting of the NFS server CD image at /cdrom and will then proceed to boot. It can take some time if the network is slow (e.g. a WiFi? network).

Netconsole Debugging

Sometimes it is useful to capture the kernel log messages during boot-time. If the client and manager PC have serial ports (many PCs do not) a null-modem cable can be used.

However, seeing as the PC is booting over the network it makes sense to make use of the kernel's netconsole facility to echo the messages to another PC. That could be a syslog server or simply the netcat process.

Log Server

Use netcat to listen for UDP packets on the default netconsole target port (6666). Optionally specify the server's interface to listen on (useful when multi-homed):

nc -vv -u -l -p 6666 -s 10.254.251.52

Modified live-CD Initial RAM-disk Image

Now the live-CD's casper/initrd.gz needs netconsole adding to the list of modules it should load. The 'regular' way would be to add module entries to the initrd /conf/modules file and ensure the corresponding modules are in its /lib/modules/$(uname -r)/ directory.

However, I've been experimenting with an alternative that alters the initrd /init script so that both prerequisite modules (network device drivers) and netconsole itself can be demand-loaded via the kernel command-line. I'd like to get the mechanism accepted into the initramfs-tools package so that future live-CD's (and installed systems) have this additional flexibility.

For now, which-ever mechanism is used, the initrd image has to be expanded, altered, and rebuilt.

Mount ISO Image

Mount the ISO CD image into the file-system:

sudo mkdir /mnt/live-cd
sudo mount -t iso9660 -o loop /home/all/iso-image/ubuntu-10.04-desktop-i386.iso /mnt/live-cd

Create a Working Copy

The image needs to be altered so a working copy will be used.

mkdir /home/all/iso-image/ubuntu-10.04-desktop-i386-DEBUG
cp -a /mnt/live-cd/* /home/all/iso-image/ubuntu-10.04-desktop-i386-DEBUG/

Extract Initrd

Create a temporary location and extract the image contents:

mkdir /tmp/initrd
cd /tmp/initrd

For initrd images compressed using gzip + tar (pre-Karmic 9.10):

zcat /home/all/iso-image/ubuntu-9.04-desktop-i386-DEBUG/casper/initrd.gz | cpio -i -d

For initrd images compressed using lzma (Karmic 9.10 and later):

lzma -dc -S .lz /home/all/iso-image/ubuntu-10.04-desktop-i386-DEBUG/casper/initrd.lz | cpio -id

Patch to /init Script

Download the init.patch file attached to this article . It adds an additional kernel command-line option (essential=) and detects the netconsole= option and when found, loads the module with the given parameters.

cd /tmp
wget -N http://tjworld.net/raw-attachment/wiki/Linux/Ubuntu/NetbootPxeLiveCDMultipleReleases/init.patch

Apply it using:

cd /tmp/initrd
patch -p3 < /tmp/init.patch

Ensure Network Module is Included

This step is dependent on the network device in the PC. First identify the network chip-set and from that the kernel module that drives it. Check the module is included in the initrd image. For example, the Intel e100 is used by the PC so:

cd /tmp/initrd
find ./lib/modules/ -name 'e100*'

./lib/modules/2.6.32-22-generic/kernel/drivers/net/e100.ko

If the kernel module isn't in the initrd image copy it from an installed system that uses the same kernel version or mount the live-CD's squashfs file-system and copy it from there:

mkdir /mnt/squash
sudo mount -t squashfs -o loop /home/all/iso-image/ubuntu-10.04-desktop-i386-DEBUG/casper/filesystem.squashfs /mnt/squash

Now it is possible to locate the module. Let's use e100 again as an example - in practice this step is only needed if the module isn't in the initrd image:

find /mnt/squash/lib/modules -name 'e100.ko'

/mnt/squash/lib/modules/2.6.32-22-generic/kernel/drivers/net/e100.ko

cp /mnt/squash/lib/modules/2.6.32-22-generic/kernel/drivers/net/e100.ko /tmp/initrd/lib/modules/2.6.32-22-generic/kernel/drivers/net/e100.ko

sudo umount /mnt/squash

Rebuild Initrd

Collect the list of file names, re-create the cpio archive and compress it.

cd /tmp/initrd

For initrd compressed with gzip (pre-Karmic 9.10):

find . | cpio -o -H newc | gzip > /tmp/initrd.gz

For initrd compressed with lzma (Karmic 9.10 and later):

find . | cpio --dereference -o -H newc | lzma -7 > /tmp/initrd.lz

Copy Back to live-CD Working Copy

Replace the original (pre-Karmic 9.10):

cp /tmp/initrd.gz /home/all/iso-image/ubuntu-9.04-desktop-i386-DEBUG/casper/

or (for Karmic 9.10 and later):

cp /tmp/initrd.lz /home/all/iso-image/ubuntu-10.04-desktop-i386-DEBUG/casper/

Modified PXE Boot Menu Entry

The netconsole parameters need adding to the kernel command line. This could be done manually by editing the menu at boot-time but it makes more sense to add a custom debug entry to the PXE menu.

With the additional essential= parameter it is possible to ensure the network device driver is loaded before netconsole starts - a requirement for netconsole to work. In this example the Intel e100 driver is being loaded. When using netconsole it is useful to increate the verbosity of kernel messages using debug (Note: when using debug the upstart /init script writes a boot-log to /dev/.initramfs/initramfs.debug and /casper.log - copied to /var/log/ when the root file-system is ready).

Add an entry to /var/lib/tftpboot/pxelinux.cfg/default:

LABEL 10.04 Lucid 32-bit Live DEBUG netconsole
        kernel iso-image/ubuntu-10.04-desktop-i386-DEBUG/casper/vmlinuz
        append debug boot=casper netboot=nfs
          initrd=iso-image/ubuntu-10.04-desktop-i386-DEBUG/casper/initrd.gz
          nfsroot=10.254.251.2:/srv/boot/iso-image/ubuntu-10.04-desktop-i386-DEBUG
          essential=e100 netconsole=@10.254.251.95/,@10.254.251.2/ --

The format is netconsole=[local-port]@[local-ip-address]/[interface],[target-port]@<target-ip-address>/[target-mac-address] (<required>, [optional]).

Note: Although netconsole will use the default local-ip-address 0.0.0.0 if one isn't given, it doesn't appear to work directly or with broadcast unless an IP address is specified. This can be a problem if multiple PCs might use the PXE netconsole facility and may require manual intervention at the PXE boot menu by the user to set a free IP address.