Building CI with Jenkins on FreeBSD with Jails: Part 1

FreeBSD provides some handy tools to build clean environments for testing code. This first installment will show how to setup the tooling for the CI part of the CI/CD pipeline for a Python app.

Install and configure Jenkins

pkg install jenkins

Start Jenkins

service jenkins start

A plugin called Scripted Cloud, will handle spinning up the Jails.

Install and configure ezjail

Using ZFS will make cloning the jails very fast, so I created a small zpool called scratch. Replace scratch with the name of your zpool.

pkg install ezjail

In /usr/local/etc/ezjail.conf make sure the following variables are set to enable ZFS:

ezjail_use_zfs="YES"

And to enable ZFS for each jail:

zjail_use_zfs_for_jails="YES"

And then for which zpool and dataset to use:

ezjail_jailzfs="scratch/ezjail"

Create the base jail to clone from:

ezjail-admin install

Create the jail startup and shutdown scripts

These scripts will be used by the Scripted Cloud plugin to create and tear down the jails as needed.

Change the INTERFACE at the top of the script to match the interface used on the Jenkins host. Note the name of the jails will be jenkins_XX. Where the XX will be the last octet (LO) of the IP. The network used here is 192.168.1.0/24, modify the IP variable as appropriate for the network the Jenkins host is on. So this means a Jenkins node called jenkins_51 will have the IP 192.168.1.51.

In /usr/local/bin/ezjail_spinup.sh:

#!/bin/sh

INTERFACE=vtnet0

[ -z "${SCVM_NAME}" ] && echo "SCVM_NAME not set" && exit 2

LO=$( echo ${SCVM_NAME} | cut -d_ -f2 )
IP="192.168.1.${LO}"

# Check hosts file
grep ${SCVM_NAME} /etc/hosts > /dev/null
if [ $? -ne 0 ]; then
        echo "${IP}  ${SCVM_NAME}" >> /etc/hosts
fi

# Configure IP
ifconfig ${INTERFACE} inet ${IP} alias

# Create a jail for use by Jenkins
ezjail-admin create jenkins_${LO} ${IP}

/usr/local/etc/rc.d/ezjail start jenkins_${LO}

# Install basic stuff for Jenkins to use
jexec jenkins_${LO} pkg install -y openjdk8 git-lite

# Install pkgs for testing
jexec jenkins_${LO} pkg install -y python devel/py-pytest www/py-flask devel/py-freezegun

# Add a user for Jenkins
jexec jenkins_${LO} pw useradd jenkins -m -d /usr/local/jenkins

jexec jenkins_${LO} mkdir /usr/local/jenkins/workspace

# Add an ssh key
jexec jenkins_${LO} mkdir /usr/local/jenkins/.ssh
echo "### INSERT YOUR PUB KEY HERE ###" > /usr/jails/jenkins_${LO}/usr/local/jenkins/.ssh/authorized_keys
jexec jenkins_${LO} chown -R jenkins /usr/local/jenkins
jexec jenkins_${LO} chmod 750 /usr/local/jenkins/.ssh
jexec jenkins_${LO} chmod 600 /usr/local/jenkins/.ssh/authorized_keys

This script is where any dependencies needed for the jobs are being installed. In the future it would be nice to install them as part of the job itself.

For the jail shutdown script, as before modify the INTERFACE variable to match the inteface of the Jenkins host.

In /usr/local/bin/ezjail_spindown.sh:

#!/bin/sh
INTERFACE=vtnet0

IP=$( jls | grep ${SCVM_NAME} | awk '{ print $2 }' )

ezjail-admin delete -f -w ${SCVM_NAME}

if [ ! -z "${IP}" ]; then
        ifconfig ${INTERFACE} ${IP} delete
fi

Configure sudo

This will allow the jenkins user account to execute them with the needed root privilages to configure IPs and create Jails.

Add the following to /usr/local/etc/sudoers.d/jenkins:

Defaults  env_keep += SCVM_*
jenkins  ALL=(ALL) NOPASSWD: /usr/local/bin/ezjail_spinup.sh, /usr/local/bin/ezjail_spindown.sh

Configure Jenkins to spin up instances

  1. Go to the Jenkins Web UI
  2. Navigate to ‘Manage Jenkins’
  3. Select ‘Manage Nodes and Clouds’
  4. Choose ‘Configure Clouds’ from the left navigation
  5. Select ‘Add a new cloud’ and choose ‘scripted Cloud’
  6. Set ‘Name to use for this scripted Cloud’ to “Jail”
  7. Set ‘Batch or a shell script files to start machine’ to sudo /usr/local/bin/ezjail_spinup.sh
  8. Set ‘Batch or a shell script files to stop machine’ to sudo /usr/local/bin/ezjail_spindown.sh
  9. Save the configuration

Now to setup each node:

  1. Navigate to ‘Manage Jenkins’
  2. Select ‘Manage Nodes and Clouds’
  3. Choose ‘New Node’ from the left navigation
  4. Set the name as jenkins_51, replace 51 with each IP you want to use
  5. Select ‘Slave virtual computer running under scripted Cloud’
  6. Choose ‘OK’
  7. On the node configuration page, from the ‘scripted Cloud Instance’ choose ‘jail’
  8. Set the ‘Virtual Machine Name’ to jenkins_51, replace 51 with each IP you want to use
  9. Set the ‘Remote FS root’ to /usr/local/jenkins
  10. Set the label to jail
  11. From the ‘Usage’ drop down select ‘Only build jobs with label expressions matching this node’
  12. From the ‘Slave launch method’ drop down select ‘Launch agents via SSH’
  13. Set the ‘Host’ to jenkins_51, replace 51 with each IP you want to use
  14. Set the ‘Credentials’ to the key that matches the pub key from the ezjail_spinup.sh script above
  15. From the ‘Host Key Verification Strategy’ drop down select ‘Non verifying Verification Strategy’
  16. From the ‘Availability’ drop down select ‘Bring this agent online when in demand, and take offline when idle’
  17. Set the ‘In demand delay’ to ‘1’
  18. Set the ‘Idle delay’ to ‘1’
  19. From the ‘What to do when the slave is disconnected’ drop down select ‘Shutdown’
  20. Save the configuration

Repeat this process as many times as necessary to have agents running at the same time. In this example two to three agents will be enough.

Create a new job

In the new job set the following properties:

  1. Check the ‘Restrict where this project can be run’ box
  2. Set the ‘Lable Expression’ to ‘jail’ and it should match the agents created above
  3. For the build, add a new ‘Excute Shell’ build step
  4. Set the command to something basic like echo "Test jail job"
  5. Save the job

Test by running the job manually.

This process has been built for testing a Python app using pytest, in a future article the CD part of deploying the Python app will be explored.