Nikhef Containers at Scale

Dennis van Dok

2019-04-09

Scientists have a lot of data to process

carve up data in bite-size chunks \(O(GB)\)
process individual chunks in batch jobs
scale up number of batch jobs to reduce overall processing time—sometimes from years to weeks

Users rely on container images for their software distribution

But it also introduces security risks and scalability issues from the site's point of view.

Singularity (https://sylabs.io/docs/) is a popular choice for container use.

Current coding practices of Singularity are not up to highest security standards.
From the user's point of view it is simple (just point to the container image or URL)
Security complications because the privileged mode relies on the loop device mechanism in the Linux kernel; this requires root privileges by making the singularity binaries setuid root.
Alternatively, Singularity non-privileged mode relies on linux control groups to restrict users to a subdirectory

In the context of running \(O(1000)\) jobs this could incur high costs
Especially in non-privileged mode, the unpacking of container images could take up to 10s of minutes.
Modern-day worker nodes have 32 or more job slots, meaning potentially unpacking and running 32 containers simultaneously.
Could be high fraction of the overall processing time per job for short jobs (there is variance depending on use case).

Singularity : run in 2 modes: privileged vs unprivileged
Points singularity to the docker image/singlualrity image (supposed to work with both images).
Security complications for privileged mode — setuid binaries. Not happy with the degree of security scrutiny that they have for this mode.
Security mistakes by programmers — doesn’t belong — opens too many back doors
Non privileged mode works fine, but instead of using a singularity container — now need to point the image to a directory.Because the mechanism
Privileged mode relies on the loop back interface to mount the image as a virtual disk
Non-Privileged version uses a linux control group mechanism to restrict the users access to a subdirectory
However this subdirectory needs to already exist somewhere.
This could be a problem if this service were to scale up if all your grid jobs would have to start by unpacking your image into a subdirectory — time consuming operation and disk space needs to be made available. 40 odd cores for a 40 odd jobs — will fill up the local disk quickly
- Unpacked size is much larger than the packed image files

The challenge:

Reduce the set up time for a container (10s of minutes eating up your wall time for a job slot)
Scale up to running 100 containers on a single machine in under 5 minutes
Verify the security of the solution

One approach is to make the unpacked images available via CVMFS.

It would have to be transparent for the users, as simple as their current use case.
CVMFS is a proven system to massively scale software distribution
Details about typical container sizes and machine sizes to be discussed with the Nikhef team.