, CVMFS for the masses

Dennis van Dok

Generic Components of the eScience Infrastructure Ecosystem — 14th IEEE eScience Conference Amsterdam, Monday 2018-10-29

Grid computing a.k.a. PaaS

Large scale common science infrastructure for high throughput batch computing.

  • the only guaranteed environment is the base OS and some middleware
  • no persistent local storage between jobs
  • bring your own software

Challenges of software distribution

  • Bringing software with every job incurs much overhead
  • Projects to develop common software distributions have a slow upgrade cycle
  • Negotiating a locally writable software area for each site takes time, effort and coordination

Software distribution with CVMFS

  • CVMFS spun off the the CERN Virtual Machine
  • content delivery based on http
  • data is distributed as objects referenced by hashes
  • read-only, so trivial to replicate massively
  • transactionally consistent indices
  • garbage collected


Sorry, your browser does not support SVG.


CVMFS is great for large organisations. But for small teams it can be a real challenge:

  • set up and maintain a repository
  • take care of a Stratum-0 server
  • negotiate the replication at Stratum-1 sites
  • negotiate with sites to include the repository in their CVMFS configuration

I imagined dozens of small e-science groups knocking on my door to get their repositories mounted.

Our solution:

Nikhef and SURFSara have jointly set up /cvmfs/ to offer a single CVMFS repository for all e-science users in the Netherlands.


The system consists of

  • a user interface system, where users can log on (with ssh) and upload their software
  • a Stratum-0 server which copies the user's files at regular intervals
  • Stratum-1 at Nikhef and RAL
  • mounted by default on all grid resources in the Netherlands

Sorry, your browser does not support SVG.



  • User requests account at SURFSara
  • Standard quota of 2GB (could be extended)
  • Manage software on
  • Copy software to /cvmfs/$USER
  • Run the publish command which touches the softdrive.modified file


  • Automated rsync from Stratum-0 server at Nikhef
  • Two stage process:
    1. rsync the softdrive.modified files
    2. rsync those directories with updated softdrive.modified files


Catalog size exploded when monitoring was put in place. The monitoring triggered an update every five minutes and thereby a completely new, full catalog of all files.

This was ultimately understood and remedied by making subcatalogs per user.

User experience

To complement the technical implementation, the total user experience was taken care of by having proper documentation, monitoring and guidance.


The user documentation is right there when logging on to the system. The message of the day, printed for login shells, gives a summary of the workings of the system and how to publish data.

More extensive documentation was written and placed on-line.


End to end monitoring of the system is done by automatically triggering a change to the system every hour and measuring the time it takes for the data to reach a client machine. Alerts are raised if the delay reaches a certain threshold, prompting the technicians to inspect what went wrong.


The softdrive model has proven to be succesful; it is easy for users to maintain their own software; the software is lightweight and the maintainance burden on the administrators is very light.

There is no plan at this point to add more bells and whistles to the system.

Even as the PaaS infrastructure dwindles in favour of IaaS (infrastructure as a service), the CVMFS system could still be a viable component for delivering software.

Some numbers

  • 25 active users last 6 months
  • 393k files, 178 GB



Some other national grid infrastructures offer something similar to softdrive, but I've not heard of anyone interested in cloning our setup. If you have plans to provide CVMFS to your users, and would perhaps like to use (parts of) the softdrive system, don't hesitate to contact me.