The TSUBAME-KFC system at Tokyo Institute of Technology
immerses its servers and GPUs in dielectric mineral oil. In
November 2013, the system became the first Japanese system to be awarded the top spot in the Green500 List of
energy efficient supercomputers. Courtesy of NVIDIA
Kirk W. Cameron, Ph.D.
Here’s the pitch: “We would like millions of dollars to build a supercomputer capable of calculating 150 trillion floating point operations per second (TFLOPS). Hundreds of scientists will use the
system to investigate the causes of global warming, drugs
that may cure cancer, and the origins of the universe. The
machine will be built from the most advanced equipment
available from NEC, Intel, NVIDIA, Mellanox, and other
manufacturers. This machine will be a prelude to a larger
system worth hundreds of millions of dollars a few years
from now. Oh, <cough> and we plan to submerge the entire
system in mineral oil.”
While this may seem contrived, at some point in the not-
too-distant past, Japanese researchers proposed a similar
scenario to develop a prototype system now known as the
TSUBAME-KFC supercomputer. The system combines the
computing power of two Intel Xeon ES-2620 processors
with four NVIDIA Tesla K20X graphics processing engines
per node. The resulting heterogeneous system is capable of
more than 150 TFLOPS of computation running the LIN-
PACK benchmark. This supercomputer ranks at #311 in the
Top500 List of the world’s fastest supercomputers.
While all supercomputers like TSUBAME-KFC generate
excessive amounts of heat, most others use a form of air
cooling to reject the heat from the confines of the data center. Cold air is piped in, and hot air is piped out. Air cooling
is commonly available and effective, but the infrastructure is
somewhat inefficient and costly to operate.
What makes TSUBAME-KFC special is that all 2,720
CPU cores are submerged in a liquid, namely dielectric
mineral oil. Though the oil itself is a blend marketed as
GreenDEF by Green Revolution Computing, this substance is not all that different (sans perfume) from the
baby oil with which we are all familiar. The GreenDEF
liquid can absorb heat up to 1200 times more efficiently
than air. By constantly circulating the oil across a supercomputer composed from commodity system racks, components can achieve densities previously difficult without
highly efficient, expensive air cooling.
TSUBAME-KFC’s power and cooling advantages are
Is TSUBAME-KFC a Game-changer?
Japan’s prototype marks a significant increase in infrastructure and system efficiencies