HEPiX San Diego, Spring 2019

Dennis van Dok


HEPiX San Diego 25–29 March 2019

All presentations and recordings available at




North America 31
Europe 39
other 4
total 88


Site Reports 19
Storage & Filesystems 10
Grids, Clouds & Virtualization 8
Networking & Security 7
Computing & Batch Systems 7
IT Facilities & Business Continuity 7
End User IT Services & OS 3
Basic IT Services 2
Other 1
Total 64

Site Reports


  • HTCondor shared pool model on HTC Linux Farm for more efficient utilization (no preemption implemented yet)
  • OpenLDAP to RedHat IPA migration
  • Jupyter analysis portal (Jupyterhub)
  • xcache xrootd cache in front of dCache

Canada T1

Move from TRIUMF to SFU (30km). Still operated by TRIUMF T1 personnel.


Wigner remote site decommissioning

ATLAS Great Lakes Tier 2

Analysis of low utilization. ATLAS jobs vary in CPU efficiency from 40% to 90%.

Implemented backfilling with ATLAS@home (BOINC).


IPv6-only worker node still not working but getting closer (no USCMS factory support for IPv6 yet).

Brian Bockelman no longer with us.


Data Intensive Computing for climate and earth-system science
Data & Analysis Facility for National Infrastructure

HTCondor cluster runs jobs inside docker images managed by RAL. Running latest mainline Linux kernels for best overlayfs support.

4kcore Openstack cloud to double in 2019Q2.


New research division for astroparticle physics, headed by Christian Stegmann.

New computing facility, passively cooled racks

Upgrade to Windows 10: No DESY accounts or information in Microsoft Clouds

End-User IT Services & OS


Consolidation of cluster management and provisioning to have a more consistent end-user experience and simplify staff effort.

Networking & Security

CERN security outlook

Mostly same as before (i.e. still bad).

Interesting suggesting: e-mail detonation chambers.

Also: check for Monero outgoing links on clusters.


How to break things with a few small UDP packets.

Take home message: pay attention to the ASICS vendors and don't mix 100 Gbit and 40 Gbit.

Storage & Filesystems


7 PiB Ceph cluster spanning three sites; dynamic user enrollment through Grouper and COmanage.


oVirt virtualization cluster with Gluster underneath ; SR-IOV (single root input/output virtualization) to virtualize I/O to Infiniband.

no live migration though.


Still wants to phase out AFS before RUN3.

Computing & Batch Systems

Benchmarking WG

The HS06 benchmark is obsolete, evaluating CPU Spec2017. Experiment-based benchmarks look promising; they use LHC workloads (other contributions are welcome).

Fair share at IN2P3

A closer look at the realities of pledges, quota and the throughput at the intermediate time scale \(O(\text{days})\)

Spark on Kubernetes and SWAN

On-demand provisioning and elastic scaling at CERN.

DESY photon sciences

New experiments bring challenges to computing and processing

IT Facilities & Business Continuity

HSF/WLCG Systems performance and cost modelling WG

Still a factor of 2 gap between requirements for HL LHC and available resources.

Technology watch WG

Recent market developments:

  • the return of AMD
  • shrinking market for HDD, increased use of SSD
  • concerns with shrinking competition in tape storage market
  • Rollout of 5G will contribute to rising network traffic

Grids, Clouds & Virtualization

Entonos experience with HTC at public clouds

Gregory Parker, Former WD, now independent contractor.

Adjustments to computing model to get the most out of AWS; use of spot pricing.

(Also: why do HDDs even work these days?)

Running an opportunistic OSG site inside PRP Kubernetes

Lots of science got done for OSG IceCube.

Upcoming HEPiX

HEPiX is coming to Amsterdam

HEPiX Amsterdam, 14–18 October 2019

Congress Centre


Organising committee:

  • Dennis van Dok
  • Bart van der Wal
  • Paul Kuipers


  • Joan Berger + secretaries

Sponsor liaison:

  • Tristan Suerink


Welcome drinks on Monday

In the Heineken Experience

  • drinks
  • Dutch snacks (borrelhapjes)

Conference dinner on Wednesday

Walking dinner at the Amsterdam museum + visiting museum exhibit.



15×10³ €


Covers the dinner on Wednesday


8 × 10³ €


Covers the welcome drinks on Monday


5 × 3 × 10³ €

Covers the lunches

Several potential sponsors still in the pipeline


  • Website
  • Registration/payment
  • Budget
  • Sponsors
  • Scoping out the venues

Willing to volunteer?