Using Containers¶
Containers allow you to run another Linux Distro inside the host. As we don’t have superuser privileges on Nikhef machines (except our laptops), Apptainer in user mode is the preferred solution.
Setting up¶
Apptainer is available on cvmfs from the Open Science Grid. It makes sense to add the path to Apptainer to your path to avoid very long commands.
Apptainer downloads images to a cache directory, which can quickly
fill up your home directory - where the default cache directory is
located. To avoid this, set APPTAINER_CACHEDIR
to a path on
/data
export APPTAINER_CACHEDIR=/data/my_group/my_username/.apptainer_cache
export PATH=$PATH:/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin
Running a Container¶
Apptainer supports running docker containers:
$> apptainer run docker://ubuntu:latest
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 7b1a6ab2e44d done
Copying config e7132beceb done
Writing manifest to image destination
Storing signatures
2021/12/03 13:36:54 info unpack layer: sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54
INFO: Creating SIF file...
INFO: Converting SIF file to temporary sandbox...
WARNING: underlay of /etc/localtime required more than 50 (66) bind mounts
Apptainer> pwd
/user/my_username
Apptainer> cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Accessing other directories¶
To access directories outside the working directory, Apptainer can
be told to bind-mount them using the -B
argument. The value is a
comma-separated list of outside_directory:inside_directory
:
$> apptainer run -B /project/my_project/my_username:/project,/dcache:/dcache docker://ubuntu:latest
INFO: Using cached SIF image
INFO: Converting SIF file to temporary sandbox...
WARNING: underlay of /etc/localtime required more than 50 (66) bind mounts
Apptainer> ls /dcache
alice antares atlas auger bfys datagrid detrd etseis gravwav hisparc km3net test theorie xenon
Repeatedly Running Containers¶
To speed up the start of Apptainer by avoiding repeated builds of the containers, you can build an image:
$> apptainer build --sandbox --fix-perms /data/my_group/my_username/ubuntu_latest docker://ubuntu:latest
And then start it with
$> apptainer run /data/my_group/my_username/ubuntu_latest
The --writable
argument can be passed to apptainer run
, which
will allow installation of additional software in the image. Important
caveat: the installation of software shouldn’t try to change user
(which is not allowed).
Building and installing software in shared directories is possible, but in this case, the path inside and outside of the container should be the same to avoid issues with relocation of built libraries and hardcoded paths.
GPUs¶
To get access to the GPUs in the system, pass either --rocm
or
--nv
to apptainer run
as needed.
Example: Tensorflow on AMD GPUs¶
$> ssh wn-lot-001
$> apptainer build --sandbox --fix-perms /data/my_group/my_username/tensorflow_rocm docker://rocm/tensorflow:latest
$> apptainer run --rocm /data/my_group/my_username/tensorflow_rocm
Apptainer> python
Python 3.9.13 (main, May 23 2022, 22:01:06)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorflow.python.client import device_lib
>>> device_lib.list_local_devices()
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 6969131800811270632
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 33818673152
locality {
bus_id: 2
numa_node: 1
links {
link {
device_id: 1
type: "StreamExecutor"
strength: 1
}
}
}
incarnation: 15068533579864484951
physical_device_desc: "device: 0, name: Vega 20, pci bus id: 0000:83:00.0"
, name: "/device:GPU:1"
device_type: "GPU"
memory_limit: 33818673152
locality {
bus_id: 2
numa_node: 1
links {
link {
type: "StreamExecutor"
strength: 1
}
}
}
incarnation: 16034552143571767030
physical_device_desc: "device: 1, name: Vega 20, pci bus id: 0000:43:00.0"
]