The Singularity SPANK plugin provides the users with an interface to launch an application within a Linux container. The plug-in adds multiple command-line options to the salloc, srun and sbatch commands. These options are then propagated to a shell script slurm-singularity-wrapper.sh customizable by the cluster administrator. This plugin is compatible to both Apptainer ¹ and SinguarityCE ² (Sylabs Inc.) as container engine.

The Slurm SPANK plug-in mechanisms ³ dynamically modifies the runtime behavior of Slurm jobs:

SPANK provides a very generic interface for stackable plug-ins which may be used to dynamically modify the job launch code in Slurm. SPANK plugins may be built without access to Slurm source code. They need only be compiled against Slurm's spank.h ⁴ header file, added to the SPANK config file plugstack.conf, and they will be loaded at runtime during the next job launch.

All SPANK plug-ins should be recompiled when upgrading Slurm to a new major release. ³

Build this plug-in using g++ from the GNU Compiler Collection (GCC) version 8 or newer. The plug-ins are compiled against this header file spank.h [02]. Fedora distributes this file in the slurm-devel RPM package ⁵. CMake is available via the cmake package.

cmake -S . -B build # configure the project and choose a build dir
cmake --build build # build the Singularity SPANK plug-in
sudo cmake --install build # install the binary and configuration files
# on older CMake: cmake --build build --target install

By default the plug-in singularity-exec.so is installed to /usr/lib64/slurm.

Restart slurmd in order to load the plug-in after installation.

Basic configuration to enable the plug-in:

# configure the plug-in mechanism to load configurations from a sub-directory
mkdir /etc/slurm/plugstack.conf.d
cat > /etc/slurm/plugstack.conf <<EOF
include /etc/slurm/plugstack.conf.d/*.conf'
EOF
# reference the path to the plug-in and the wrapper script
cat > /etc/slurm/plugstack.conf.d/singularity-exec.conf <<EOF
required /usr/lib64/slurm/singularity-exec.so default= script=/usr/libexec/slurm-singularity-wrapper.sh bind= args=disabled
EOF

Note that the configuration illustrated above will be deployed by make install. Modification to the plug-in configuration described below does not required a restart of slurmd:

Passing -DINSTALL_PLUGSTACK_CONF=ON to the CMake configure command will automate the above configuration.

The plugin adds following command-line options to salloc, srun and sbatch:

The implementation of slurm-singularity-wrapper.sh adds additional environment variables:

Following srun command use the options and environment variables described above:

SLURM_SINGULARITY_DEBUG=true SLURM_SINGULARITY_GLOBAL=--silent \
      srun --singularity-container=/tmp/debian10.sif \
           --singularity-bind=/srv \
           --singularity-args="--no-home" \
           -- /bin/grep -i pretty /etc/os-release

Executing it will generate output similar to:

Start Singularity container /tmp/debian10.sif
Debug: SLURM_SINGULARITY_CONTAINER=/tmp/debian10.sif
Debug: SLURM_SINGULARITY_ARGS=--no-home
Debug: SLURM_SINGULARITY_BIND=/etc/slurm,/var/run/munge,/var/spool/slurm
Debug: SLURM_SINGULARITY_GLOBAL=--silent
Debug: singularity --silent exec --bind=/etc/slurm,/var/run/munge,/var/spool/slurm,/srv --no-home /tmp/debian10.sif /bin/grep -i pretty /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"

Similar use with the sbatch command:

cat > job.sh <<EOF
#!/usr/bin/env bash
#SBATCH --singularity-container /tmp/debian10.sif
#SBATCH --singularity-bind=/srv
#SBATCH --singularity-args="--no-home"
/bin/grep -i pretty /etc/os-release
EOF
SLURM_SINGULARITY_DEBUG=true SLURM_SINGULARITY_GLOBAL=--silent sbatch job.sh

Build the required singularity containers with the script containers.sh. (This requires the singularity command installed on the host). The containers generated by the script are stored under /tmp/*.sif.

Start a test environment using the included Vagrantfile:

• Installs the apptainer package from Fedora EPEL
• Copies the SIF container images to /tmp
• Builds, installs and configures the Slurm Singularity plug-in

Start a Vagrant box to build an RPM package:

./containers.sh && vagrant up el8 && vagrant ssh el8 # for example...

# synced from the host
cd /vagrant

cmake -S . -B build # configure the project and choose a build dir
cmake --build build # build the Singularity SPANK plug-in
sudo cmake --install build # install the binary and configuration files

sudo systemctl enable --now munge slurmctld slurmd