Building a FreeBSD PXE HTTP Image with Jenkins

I have spent some time building a set of Jenkins job to run automated network tests using PXE and even IPMI to control the test nodes, and I wanted to share what I came up with others.

This method is unique because it does not use NFS at all. Only HTTP and TFTP. We will use Poudriere to build a tiny image that only contains enough to download a bigger image that is also built by poudriere. So we need a TFTP root that has the required PXE boot bits from /boot, such as pxeboot and the forth code to process configs.

Requirements

  • A Webserver (such as Nginx)
  • DHCP Server (such as dnsmasq)
  • Jenkins
  • Poudriere (installed on the builders)

Jenkins Configuration

Install the following plugins:

Jenkins Build Job

This job builds and archives the objects and source of the build for later use.

I used git and had it configured to clone the repo into the subdir head.

#!/bin/sh

SRCDIR=/b/usr/src
OBJDIR=/b/usr/obj
JOBS=$( sysctl -n hw.ncpu )

mount_nullfs head ${SRCDIR} && \
mount_nullfs obj  ${OBJDIR} && \
make -j${JOBS} -C ${SRCDIR} buildworld buildkernel && \
tar -cv head obj | xz -T0 buildartifacts.tar

Make sure the job is setup to archive the buildartifacts.tar

Jenkins Create Image Job

You might be wondering, why is this done as a separate step? The answer is, to make it easier to have jobs that can be run without doing a full build and to reuse the already built artifacts in other jobs.

Copy the following artifacts from the build job:

buildartifacts.tar

This is the execute shell part of the job:

#!/bin/sh

EXIT=0
SRCDIR=/b/usr/src
OBJDIR=/b/usr/obj

mkdir -p dest
INSTALLDIR=`realpath dest`
[ -z ${INSTALLDIR} ] && echo "Variables undefined"
[ ! -d ${OBJDIR} ] && mkdir -p ${OBJDIR}
[ ! -d ${SRCDIR} ] && mkdir -p ${SRCDIR}
[   -f kernel.gz ] && rm kernel.gz
[   -f poudriereimage.txz ] && rm poudriereimage.txz
[   -f poudriereimage-miniroot.gz ] && rm poudriereimage-miniroot.gz
[   -d repo ] && rm -fr repo
[ ! -d repo ] && mkdir repo

set +e
tar -xf buildartifacts.tar && \
mount_nullfs head ${SRCDIR} && \
mount_nullfs obj  ${OBJDIR} && \

# Build the Poudriere Jail
env MAKEOBJDIRPREFIX=${OBJDIR} poudriere jail -c -j ${BUILD_TAG} -m src=${SRCDIR} -K GENERIC-NODEBUG && \

# Copy in packages from last build
[ -f repo.txz ] && ( cd repo && tar --strip-components 6 -xvf ../repo.txz ) && \
mv repo /usr/local/poudriere/data/packages/${BUILD_TAG}-default

# Build packages
poudriere ports -u && \
poudriere bulk -j ${BUILD_TAG} -f ${WORKSPACE}/netboot/test-pkgs && \

# Build the image
sed -i.bak -e s/BUILD_NUMBER/${BUILD_NUMBER}/ ${WORKSPACE}/netboot/miniroot-overlay/etc/rc && \
poudriere image -j ${BUILD_TAG} -t tar -n ${BUILD_TAG} -m ${WORKSPACE}/netboot/miniroot-overlay -c ${WORKSPACE}/netboot/overlay -f ${WORKSPACE}/netboot/test-pkgs -h "" && \
mv ${WORKSPACE}/netboot/miniroot-overlay/etc/rc.bak ${WORKSPACE}/netboot/miniroot-overlay/etc/rc && \

# Save the Kernel
cp obj/b/usr/src/amd64.amd64/sys/*/kernel kernel && \
cp obj/b/usr/src/amd64.amd64/sys/*/modules/b/usr/src/sys/modules/tmpfs/tmpfs.ko . && \
gzip -9 kernel tmpfs.ko && \

# Save the image
[ -f /usr/local/poudriere/data/images/${BUILD_TAG}.txz ] && mv /usr/local/poudriere/data/images/${BUILD_TAG}.txz poudriereimage.txz && \
[ -f /usr/local/poudriere/data/images/${BUILD_TAG}-miniroot.gz ] && mv /usr/local/poudriere/data/images/${BUILD_TAG}-miniroot.gz poudriereimage-miniroot.gz && \

# Save the pkgs
tar -cf - /usr/local/poudriere/data/packages/${BUILD_TAG}-default | xz -T0 > repo.txz

EXIT=$?

umount -t nullfs ${SRCDIR}
umount -t nullfs ${OBJDIR}
poudriere logclean -j ${BUILD_TAG} -a
poudriere jail -d -j ${BUILD_TAG}
[ -d /usr/local/poudriere/data/packages/${BUILD_TAG}-default ] && rm -fr /usr/local/poudriere/data/packages/${BUILD_TAG}-default
set -e
return ${EXIT}

Jenkins Test Job

PXE Root

Contains the following:

  • kernel.gz
  • tmpfs.ko.gz
  • miniroot.tar.gz

Thanks to Sean Bruno, Emmanuel Vadot and Baptiste Daroussin for sharing tips and tricks that made this possible.