Inria software in the world's fastest supercomputer

A supercomputer that occupies a 700 m² room

Installed in the Oak Ridge (Tennessee) laboratory of the U.S. Department of Energy, Frontier occupies a 700 m² room, consumes 21 megawatts of electricity and must be cooled by 350 pumps that inject more than 22 m³ of water per minute.

But it is its speed that has the international HPC (High Performance Computing) community buzzing: in one second, it performs 1.1 billion billion operations (addition, subtraction, multiplication or division of 20-digit numbers). This is more than the mythical barrier of the exaflop (1 billion billion calculations/second), which it is the first to officially break.

State-of-the-art hardware architecture

"Every two or three years, a new supercomputer makes it to the top of the TOP500, explains Brice Goglin, director of research in the TADaaM project-team at the Inria center at the university of Bordeaux. It's rarely due to a hardware revolution. Sometimes it's just a clever mix of processors, graphics cards and networks not yet commercially available." Important note: some states may have more powerful supercomputers... but keep their existence a secret.

Why is Frontier so fast? First, because it uses the latest state-of-the-art computing processors and graphics cards. Second, because the 145 kilometers of interconnecting cables between its 9,400 servers (!) make up a network that's cleverly designed to speed up data transfers. Finally, in these servers, the connections between processors have been optimized. "This is the key to performance, explains Brice Goglin. The bottleneck is not the computation itself, but the time it takes for the data to reach us."

hwloc from Inria, a standard in the HPC world

In addition, Frontier benefits from the latest version of Inria's hwloc software, which was created in 2009 by researchers at the Inria center at the university of Bordeaux and has become a standard in the HPC world. "hwloc describes the hardware architecture of the machine and physically locates each resource: processor, graphics card, memory, etc. This allows us to distribute the parallel computing tasks of a program in the best possible way. For example, if two tasks need to exchange a lot of data, they are executed in processors located close to each other."

hwloc works a bit like suggesting to a general, before a battle, an optimal distribution and mode of communication between ground troops, artillery and aviation. The difference is that it is not about leading an army, but about executing a highly complex computer program. By the way, the designers of this program do not have to rewrite it every time they change supercomputers: hwloc does the work for them!

© Inria / Photo H. Raguet

Six months of work to adapt to Frontier

To develop version 2.8, which propels Frontier to the top of the world rankings, Brice Goglin exchanged countless emails with AMD and HPE Cray, the industry groups that designed the supercomputer, over a period of six months. "hwloc is open source software and we explain the reasons for every change in its code. But vendors bring in the latest technology. And sometimes they couldn't disclose certain details yet."

Still, it was essential to know these trade secrets. In particular, to describe how the processors (capable of all kinds of calculations), graphics cards (very fast for simple and repetitive operations) and network cards (which send data to other servers) are connected in the 9,400 servers.

Beyond the exaflop wall, exascale applications

Indeed, the speed of data flow is so decisive that it relies on highly specialized interconnections. "It's as if instead of supplying a factory with water via a single point of entry, we connect the fire system, air conditioning, sanitary facilities and production equipment separately to the public network. The installation is more complex, but the water gets to where it's needed faster."

Frontier, which takes this idea to the extreme, needed to be finely modeled to better distribute tasks and data. The designers at hwloc rose to the challenge, and the result is a new world speed record.

In a few months, the supercomputer will be open to the scientific community. Breaking the exaflop barrier makes previously unthinkable calculations possible. The next step will be exascale, i.e. achieving this level of performance for complex applications: validating the fundamental laws of nature, simulating the functioning of the brain, analyzing the structure of molecules and proteins, studying nuclear fusion and fission, analyzing the mechanisms of climate change, and so on.

More information

• The world's first exaflopian supercomputer is American, L’Usine digitale, 30/5/2022.
• Towards a better control of processor architectures with hwloc 2.0, Inria, 21/2/2018.