HiLumi LHC Project (5): Computers and Networks That Keep Pace with the new CERN Accelerator

The numbers are staggering but give an idea of the computing power involved with the accelerators at CERN. The ATLAS detector, for example, generates approximately 60 terabytes of ones and zeros per second at full capacity. That’s the equivalent of about a thousand 4K Blu-ray DVDs worth of data that must be processed, transferred, and stored every second.

Also impressive: a 400 Gigabit/s fiber-optic connection links CERN and Nikhef. That line is a thousand times wider than a home internet connection, yet it is nonetheless regularly overloaded with scientific data traveling to and from CERN.

Keep in mind, says David Groep, head of Physics Data Processing at Nikhef, that these are just the figures for the recently shut-down LHC. “For the upcoming high-luminosity LHC, a massive scaling up of IT infrastructure is needed; otherwise, the computing and networks will never be able to keep up with the accelerator.”

Computers play a major role throughout experimental particle physics. Near the collision points in the accelerator, computers select correlated signals from the detectors and transmit those patterns to databases, where researchers use computers to reconstruct particle tracks from the signals, thereby testing physical theories and searching for new phenomena.

The challenge posed by the arrival of the HiLumi LHC is the sheer volume of signals in the detectors. Every time beams cross, there are about 200 proton collisions, each of which produces signals in the detector. To identify correlations among them, faster sensors are used, which provide more precise timing of the events. Nikhef had previously worked on such sensor chips as part of the FASTER program.

To quickly distinguish interesting patterns from random background noise, graphics GPUs—chips from the gaming industry designed to perform countless tasks simultaneously—have been used for several years now. In the ALICE and LHCb detectors, these are already being used in the so-called triggers—systems that detect and record relevant signal patterns in real time. Nikhef was one of the pioneers in the field of GPUs within LHCb.

With the arrival of enormous luminosity in the CERN accelerator, graphics-based computing is becoming the standard, especially in the triggers where classic CPUs are still laboring step by step. Nikhef’s Physics Data Processing group is now intensively investigating the best use of hardware and software, in part with support from the FASTTRACK roadmap program.

But everything is a trade-off, says Groep. “You can tackle the overlapping images from those 200 collisions at a time with a lot of computing power at the end of the pipeline, or ensure—through better timing and smarter signal selection up front—that they overlap less, and less overlap means less computing is needed. Both are options, both are necessary, but we’re looking for the optimal setting for all the parameters.”

In this context, smart selection is increasingly becoming a matter of artificial intelligence, but how to do this effectively and affordably is still the subject of extensive research, including at Nikhef. And at the other end of the pipeline as well—when reconstructing particle tracks—artificial intelligence is once again the way to produce results that are both faster and better. And what about after HiLumi? Quantum computing might even play a role then—something Nikhef is already exploring.

But there, too, the question is what will be best and most cost-effective; due to the enormous demand for chips driven by AI, computers have become significantly more expensive rather than cheaper, as was long thought. And there’s also the question of whether all the necessary computing power will actually be available in time for the HiLumi era, which begins in 2030 and will last ten years, Groep adds.

An experimental GPU cluster will soon be set up at Nikhef to develop and test techniques and strategies, with the aim of determining the optimal approach for experiments at CERN and elsewhere.

And shortly thereafter, the first large cluster for AI-based analysis is planned. To keep energy consumption within limits, liquid cooling will be used for the equipment for the first time—something Nikhef has been researching for some time. This is a quarter more energy-efficient than the current air cooling system.

Nikhef’s data center in Amsterdam has traditionally been a so-called Tier-1 center within the computing network surrounding CERN in Geneva, where a copy of a vast amount of research data from the experiments is maintained.

This is not only secure for CERN but also more efficient for Dutch particle physicists. As part of the recently funded MERGE program, Nikhef and SURF will further equip that infrastructure to meet the growing computational needs of particle physicists, including AI and supercomputing on the national Snellius computer.

That growing ocean of experimental data, says Groep, will soon also play a role in training artificial intelligence for particle physics research. Since the beginning of this year, Nikhef has been establishing a special task force to promote and facilitate the use of AI in particle physics research.

While the arrival of HiLumi creates enormous time pressure for the computing needs of the experiments at CERN, Groep is less concerned about data transport. The current 400 Gigabit/s connections to CERN are already being expanded to 800 Gigabit/s using better fiber optics and smarter protocols. But tests are already planned for bandwidths of up to 1.2 Terabits/s. “Much more than what we expect to get out of the HiLumi LHC at maximum capacity. That’s a comforting thought.”

This is the fifth installment in a summer series on work on the HiLumi LHC accelerator.