Hybrid cooling for supercomputer

STULZ Modular has completed a prestigious installation at the University of Göttingen for the Emmy supercomputer, which employs an innovative combination of direct to chip liquid and air cooling. One of the top 100 most powerful supercomputers in the world, Emmy is named after renowned German mathematician, Emmy Noether, who was described by Albert Einstein as one of the most important women in the field of mathematics.

STULZ Modular Data Center at the University of Göttingen

The University of Göttingen needed a new data centre to house Emmy, as the existing facilities could not provide the required space and cooling infrastructure. It needed to be a modular construction with a 1.5MW total capacity that could accommodate further expansion, with the deployment of a cooling system that could remove heat density of up to 100kW per rack. Emmy’s power consumption was also a factor, so the implemented solution needed to address this by being as energy efficient and sustainable as possible.

‘We were given less than two months to design and install a two room modular data centre with a cooling infrastructure, which would be installed on a ground slab and connected to the on-site transformer station,’ explained Dushy Goonawardhane, managing director at STULZ Modular. ‘Our solution comprises four prefabricated modules – two larger modules cover an area of 85m2 and are joined along the spine to accommodate the direct to chip liquid cooled supercomputer. Two smaller modules are also joined along the spine to accommodate air cooled IT equipment in 70m2 of space.’

The entire data centre comprises high performance computers, 1,120kW direct to chip liquid cooled systems with approximately 20% residual heat, high-density racks and STULZ CyberAir and STULZ CyberRow precision air conditioning unit with free cooling . With 96kW per full rack and 11 racks currently in-situ, there is available capacity for up to 14 racks in total.

STULZ Modular worked with , CoolIT Systems which specialises in scalable liquid cooling solutions for the world’s most demanding computing environments, to incorporate direct to chip liquid cooling to Emmy’s microprocessors. Comprising two liquid loops, the secondary loop provides a flow of cooling fluid from the cooling distribution unit (CDU) to the distribution manifolds and into the servers, where heat is transferred through cold plates into the coolant. The secondary fluid then flows into the heat exchanger in the CDU, where it transfers heat into the primary loop and the absorbed heat energy is carried to a dry cooler and rejected.

The direct to chip liquid cooled system removes 78% (74.9kW) of the server heat load. A water-cooled STULZ CyberRow (with free cooling option) air cooling unit removes the remaining 22% (21.1kW) of the heat load produced by components within the server. The CyberRow’s return air temperature is specified at approximately 48°C, supply air temperature at 27-35°C and water temperature at 32-36°C.

The University of Göttingen is dedicated to reducing its carbon footprint and overall energy consumption across its campus. The STULZ modular data centre provides 27% electricity savings at an average 75% load, equating to 3.96GWh per year. Furthermore, compared to a standard air-cooled data centre with a Power Usage Effectiveness (PUE) of 1.56 – the current industry average according to the Uptime Institute – the hybrid direct to chip liquid and air-cooling system provides an overall annual facility PUE of 1.13, with a 1.07 PUE for the liquid cooled supercomputer room alone.

STULZ Modular’s Dushy Goonawardhane concluded, ‘This installation demonstrates our commitment to pushing the boundaries of data centre cooling technology. By combining direct-to-chip liquid cooling with our advanced air-cooling systems, we've created a solution that not only meets the extreme demands of supercomputing but also aligns with the University of Göttingen's sustainability goals. We are excited to share the complexity and learnings from this project in a whitepaper we have produced in cooperation with the university of Göttingen.