duced in 1976, that iconic machine, the Cray 1, was liquid
cooled using Freon. But, as the HPC industry progressed,
less expensive solutions featuring air cooling became the
norm. Data centers sprouted A/C units including chillers,
ducts, heat-exchangers and noisy fans. Thomas Sterling,
Professor, School of Informatics and Computing at Indiana University, may have experienced the ultimate in
HPC-generated decibels. A while back, he visited Moscow
State University’s massive supercomputer, which featured
high density packing and relied on air cooling. “Loudest
machine I have ever experienced,” he recalls.
Today, liquid cooling is having a resurgence as hardware
densities escalate. Cray, for example, offers both liquid and
air-cooled versions of its new XC30 system and CS300
HPC cluster. Sterling points to a recent single-rack, liquid-cooled system from RSC in Russia, which has achieved
a peak performance of one petaflop. And in Japan, NEC
has developed a phase-change approach in which coolant
is evaporated to disperse heat more quickly.
There are innumerable other approaches to liquid cooling underway, some of them involving cryogenics. But the
poster child for the practical application of warm water
liquid cooling has to be the Department of Energy’s National Renewable Energy Laboratory (NREL) in Golden,
CO. (See related story on page 10 of this issue.)
HP and Intel are supplying NREL’s petascale HPC system.
And HP has developed a new component-level liquid cooling system that reflects the company’s design philosophy.
“What we are providing is a fully-integrated solution,”
explains Nicolas Dube, Distinguished Technologist at HP.
The idea, he says, is to go beyond just cooling the supercomputer’s processors and memory to all other system
components, such as the power supplies, voltage regulators, network interconnect silicon, etcetera.
LOSE THE CHILLER
By embracing warm-water cooling, HP is able to get rid of
that standby of air cooled HPC data centers — the chiller.
Chillers and associated computer room air conditioning have become essential for cooling highly concentrated
HPC clusters. But they come with a price — not only are
they expensive, but they are energy hogs, consuming huge
amounts of electricity and requiring their own dedicated
As of last summer, NREL’s data center has been consuming just over one megawatt of power. But it’s in the winter
that this system shines. Water to the servers is supplied at 75
degrees Fahrenheit and returned at 100 degrees Fahrenheit.
Dube notes that HP and NREL are looking beyond
achieving a favorable power usage effectiveness (PUE) —
the metric that measures data center power user efficiency.
“What I’m pushing for is ERE — energy reuse efficiency,”
he says. “The heated water from the supercomputer’s cooling system acts as a furnace, heating the lab and offices and
is even being channeled under walkways to melt snow in
the Colorado winter.”
And speaking of power, at SC13 in Denver this last November, the Green500 published its list of power-efficient
systems, dominated yet again, to no one’s surprise, by heterogeneous supercomputers sporting Intel CPUs combined
with NVIDIA GPUs.
Taking top honors was the TSUBAME-KFC. Developed
at the Tokyo Institute of Technology, the supercomputer,
powered by a combination of Intel Ivy Bridge Processors
and NVIDIA Kepler GPUs, claimed an efficiency of 4. 5
gigaflops/watt. The Japanese supercomputer features a
unique oil cooling system.
Just a few years ago, recalls Steve Keckler, Senior
Director of Architecture Research at NVIDIA, energy
efficiency was not a critical design criteria. For example,
when designing the company’s Fermi class GPUs, energy
considerations were important but not a top priority. “The
jump (in energy efficiency) from Fermi to our Kepler class
systems was monumental,” says Keckler, noting that the 10
top supercomputers on the Green500 list use Kepler GPUs.
He claims that throughput-oriented computing is the way
Cray-1 preserved at the Deutsches Museum