Building a New Kind of Water-cooled Supercomputer
click to enlarge
|Schematic showing the water-cooling system of Aquasar|
In an effort to achieve energy-aware computing, the Swiss Federal Institute of Technology Zurich (ETH) and IBM are building a first-of-a-kind water-cooled supercomputer that will directly repurpose excess heat for the university buildings. The innovative system, dubbed Aquasar, is expected to decrease the carbon footprint of the system by up to 85 percent and estimated to save up to 30 tons of CO2 per year, compared to a similar system using today’s cooling technologies.1
Making computing systems and data centers energy-efficient is a staggering undertaking. In fact, up to 50 percent of an average air-cooled data center’s carbon footprint or energy consumption today is not caused by computing but by powering the necessary cooling systems to keep the processors from overheating — a situation that is far from optimal when looking at energy efficiency from a holistic perspective.
“Energy is arguably the number one challenge humanity will be facing in the 21st century. We cannot afford anymore to design computer systems based on the criterion of computational speed and performance alone,” explains Dr. Poulikakos of ETH Zurich, head of the Laboratory of Thermodynamics in Emerging Technologies and lead investigator of this interdisciplinary project. “The new target must be high performance and low net power consumption supercomputers and data centers. This means liquid cooling.”
|Aquasar, a first-of-a-kind water-cooled supercomputer with direct energy re-use, will have water-cooled blades like the one pictured. The two microchannel coolers at the center are attached directly to the processors, allowing for unprecedented cooling efficiency. What’s more, the removed heat will be directly repurposed for heating the university’s buildings.|
With an innovative water-cooling system and direct heat reuse, Aquasar is planned to start operation in 2010 and will reduce overall energy consumption by 40 percent. The system is based on long-term joint research collaboration of ETH and IBM scientists in the field of chip-level water-cooling, as well as on a concept for “water-cooled data centers with direct energy re-use” advanced by scientists at IBM’s Zurich Lab.
The water-cooled supercomputer will consist of two IBM BladeCenter servers in one rack and will have a peak performance of about 10 Teraflops.2
Each of the blades will be equipped with a microscale high-performance liquid cooler per processor, as well as input and output pipeline networks and connections, which allow each blade to be connected and disconnected easily to the entire system.
Water as a coolant has the ability to capture heat about 4,000 times more efficiently than air, and its heat-transporting properties are also far superior. Chip-level cooling with a water temperature of approximately 60 degrees celsius is sufficient to keep the chip at operating temperatures well below the maximally allowed 85 degrees celsius. The high input temperature of the coolant results in an even higher-grade heat as an output which, in this case, will be about 65 degrees celsius.
The pipelines from the individual blades link to the larger network of the server rack, which in turn are connected to the main water transportation network. The water-cooled supercomputer will require about 10 liters of water for cooling, and a pump ensures a flow rate of roughly 30 liters per minute. The entire cooling system is a closed circuit: the cooling water is heated constantly by the chips and consequently cooled to the required temperature as it passes through a passive heat exchanger, thus delivering the removed heat directly to the heating system of the university in this experimental phase. This eliminates the need for today’s energy-hungry chillers.
|IBM research team holds a water-cooled blade. Behind the team is the prototype system at the IBM Zurich Research Lab.|
“Heat is a valuable commodity that we rely on and pay dearly for in our everyday lives. If we capture and transport the waste heat from the active components in a computer system as efficiently as possible, we can reuse it as a resource, thus saving energy and lowering carbon emissions. This project is a significant step towards energy-aware, emission-free computing and data centers,” explains Bruno Michel, Manager Advanced Thermal Packaging at IBM’s Zurich Research Laboratory.
Collaborative research in emission-free high performance computing
From the industrial side, the project is part of IBM’s First-Of-A-Kind program (FOAK), which engages IBM’s scientists with clients to explore and pilot emerging technologies that address real-world business problems. It was made possible by the support of IBM Switzerland and the IBM Research and Development Laboratory in Boeblingen, Germany.
This liquid cooled supercomputer research is planned as a three-year collaborative program called “Direct Re-Use of Waste Heat from Liquid-Cooled Supercomputers: Towards Low Power, High Performance, Zero-Emission Computing and Datacenters,” which is funded jointly mainly by IBM, ETH Zurich and the Swiss Competence Center for Energy and Mobility (CCEM). Part of the system will be devoted to further research into cooling technologies and efficiencies by scientists of ETH Zurich, ETH Lausanne, the Swiss Competence Center for Energy and Mobility, and the IBM Zurich Research Lab.
The computational performance of Aquasar is a very important part of the research. Aquasar will be employed by the Computational Science and Engineering Lab of the Computer Science Department at ETH Zurich for multiscale flow simulations pertaining to problems encountered at the interface of nanotechnology and fluid dynamics. Researchers from this laboratory also will optimize the efficiency with which the respective algorithms perform within the system, in collaboration with the IBM Zurich Lab. These activities will be supplemented with algorithms of other research labs participating in the project. With this supercomputer system, scientists intend to demonstrate that the ability to solve important scientific problems efficiently does not need to have an adverse effect on the energy and environmental challenges facing humanity.
1. By making use of a physical carbon offset that fulfills criteria set forth in the Kyoto Protocol. The estimate of 30 tons CO2 is based on the assumptions of average yearly operation of the system and the energy for heating the buildings being produced by fossil fuels.
2. BladeCenter servers with a mixed population of QS22 IBM PowerXCell 8i processors as well as HS22 with Intel Nehalem processor. In addition, a third air-cooled IBM BladeCenter server will be implemented to serve as a reference system for measurements. Please note, all numbers provided are estimates and refer to the water-cooled IBM BladeCenter servers.