Using Containers

Containers allow you to run another Linux Distro inside the host. As we don’t have superuser privileges on Nikhef machines (except our laptops), Apptainer in user mode is the preferred solution.

Setting up

Apptainer is available on cvmfs from the Open Science Grid. It makes sense to add the path to Apptainer to your path to avoid very long commands.

Apptainer downloads images to a cache directory, which can quickly fill up your home directory - where the default cache directory is located. To avoid this, set APPTAINER_CACHEDIR to a path on /data

export APPTAINER_CACHEDIR=/data/my_group/my_username/.apptainer_cache
export PATH=$PATH:/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin

Running a Container

Apptainer supports running docker containers:

$> apptainer run docker://ubuntu:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 7b1a6ab2e44d done
Copying config e7132beceb done
Writing manifest to image destination
Storing signatures
2021/12/03 13:36:54  info unpack layer: sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54
INFO:    Creating SIF file...
INFO:    Converting SIF file to temporary sandbox...
WARNING: underlay of /etc/localtime required more than 50 (66) bind mounts
Apptainer> pwd
/user/my_username
Apptainer> cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Accessing other directories

To access directories outside the working directory, Apptainer can be told to bind-mount them using the -B argument. The value is a comma-separated list of outside_directory:inside_directory:

$> apptainer run -B /project/my_project/my_username:/project,/dcache:/dcache docker://ubuntu:latest
INFO:    Using cached SIF image
INFO:    Converting SIF file to temporary sandbox...
WARNING: underlay of /etc/localtime required more than 50 (66) bind mounts
Apptainer> ls /dcache
alice  antares  atlas  auger  bfys  datagrid  detrd  etseis  gravwav  hisparc  km3net  test  theorie  xenon

Repeatedly Running Containers

To speed up the start of Apptainer by avoiding repeated builds of the containers, you can build an image:

$> apptainer build --sandbox --fix-perms /data/my_group/my_username/ubuntu_latest docker://ubuntu:latest

And then start it with

$> apptainer run /data/my_group/my_username/ubuntu_latest

The --writable argument can be passed to apptainer run, which will allow installation of additional software in the image. Important caveat: the installation of software shouldn’t try to change user (which is not allowed).

Building and installing software in shared directories is possible, but in this case, the path inside and outside of the container should be the same to avoid issues with relocation of built libraries and hardcoded paths.

GPUs

To get access to the GPUs in the system, pass either --rocm or --nv to apptainer run as needed.

Example: Tensorflow on AMD GPUs

$> ssh wn-lot-001
$> apptainer build --sandbox --fix-perms /data/my_group/my_username/tensorflow_rocm docker://rocm/tensorflow:latest
$> apptainer run --rocm /data/my_group/my_username/tensorflow_rocm
Apptainer> python
Python 3.9.13 (main, May 23 2022, 22:01:06)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorflow.python.client import device_lib
>>> device_lib.list_local_devices()
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 6969131800811270632
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 33818673152
locality {
  bus_id: 2
  numa_node: 1
  links {
    link {
      device_id: 1
      type: "StreamExecutor"
      strength: 1
    }
  }
}
incarnation: 15068533579864484951
physical_device_desc: "device: 0, name: Vega 20, pci bus id: 0000:83:00.0"
, name: "/device:GPU:1"
device_type: "GPU"
memory_limit: 33818673152
locality {
  bus_id: 2
  numa_node: 1
  links {
    link {
      type: "StreamExecutor"
      strength: 1
    }
  }
}
incarnation: 16034552143571767030
physical_device_desc: "device: 1, name: Vega 20, pci bus id: 0000:43:00.0"
]