GIGABYTE Server Shows Two-Phase Immersion Liquid Cooling on a 2U GPU G250-S88 using 3M Novec
by Ian Cutress on January 17, 2017 10:30 AM EST- Posted in
- Gigabyte
- Enterprise
- Trade Shows
- cooling
- Servers
- CES 2017
- 3M
In the land of immersed systems, there are many ways of doing things. A few intrepid users have gone with oil, still using a CPU cooler but relying on the liquid cycling throughout the system to remove heat energy. Going back over a decade and a half, I recall a system whereby a 35W processor was immersed, without a heatsink, into a bath of a 3M hydrocarbon with a modest boiling point, around 45C, which was then combined in a sealed system with an external thermal electric cooler to initiate the recycling. The demonstration by GIGABYTE at CES this year with a fully-embedded system is more the latter than the former.
Obviously you cannot use water (unless exceptionally pure/distilled) for conductivity reasons, so some inert hydrocarbon is the purpose here. The concept for this design is a two-phase change from liquid to vapor and back, using no pumps but relying on the fact that the gas will condense and fall back into the solution and sink, causing automatic cycling.
As I mentioned before, previously I had only seen this on a small low powered system, but GIGABYTE and 3M had submerged a full 8-GPU, dual CPU system with 24 memory modules and nothing more than large copper heatsinks on the CPU/GPU, and had even removed the power delivery heatsinks.
To cool the vapor as it rises through the system, a cold radiator is placed inside the sealed system. Well, I say sealed, but during the demo it was being opened and the demonstrator was clearly putting his hand inside. There seemed to also be a system in place to add/remove hydrocarbon material through a pump as well.
So the point in all this is more efficient cooling – no need for massive air conditioning units in a data center, no need to pump chilled water into water blocks. I’m surprised that this system was suitable for all that hardware, but it does leave on issue on the table: getting access to replacing hardware. Moving from air to liquid cooling in a data-center always has this issue.
So to keep things under wraps, 3M's Novec line of liquids involve a full array of halogenated hydrocarbon compounds for different uses, and the variant of Novec that is under use here was not specified. However a quick search turns up a likely candidate in Novec 72DA.
Novec 72DA liquid is a solution of 70% 1,2-trans-dichloroethylene, 4-16% ethyl nonafluorobutyl ether, 4-6% ethyl nonafluoroisobutyl ether and trace other similar methyl variants. The liquid has a boiling point of 45ºC at very low viscosity (0.4 cP, compared to 0.89 cP for water), but also a low specific heat capacity (1.33 J/g/K, compared to 4.184 for water). Typically water cooling (with blocks) with the high heat capacity is preferred, but at 1.33 J/g/K for the main ingredient in Novec is interesting: take a CPU that uses 140W, and in 60 seconds it will change 8.4 kJ of energy from electricity to heat. That would raise one kg of liquid (0.8 liters, due to 1.257 kg per liter for density) up by 7.24ºC. Thus it would take around 3 minutes from a slightly chilled start to create one kg of the main component of Novec to boiling point. If we add in the latent heat of vaporization, or the energy it takes to transform a chemical from a liquid at boiling point to vapor, then we need another 350 kJ/kg, or 41.67 minutes.
Now obviously in such a system it doesn’t work on pure kilograms of chemical – energy is transferred at larger doses on smaller amounts of liquid at once, causing the effect we see in the photos.
28 Comments
View All Comments
Meteor2 - Wednesday, January 18, 2017 - link
So what was this thing extracting in total? I make it just over 2 kW coming out of that 2U rack, assuming efficient power supplies. Rejecting that much heat without fans is rather good.petuma - Friday, February 3, 2017 - link
Power in overclocked state was 3.3kWAnato - Wednesday, January 18, 2017 - link
Now sysadmins and maintainers need to get diving course to access servers :-)zodiacfml - Thursday, January 19, 2017 - link
Seems great for cooling all components and I have read somewhere that CPU loads in the data center are pretty low for best practice.Nottheface - Monday, January 23, 2017 - link
I don't understand how this "So the point in all this is more efficient cooling – no need for massive air conditioning units in a data center, no need to pump chilled water into water blocks." removes the need to remove the heat from the servers from the building???Same amount of heat generated from the servers = same general need of removing it from the building. Whether that method is air conditioning, liquid cooling, etc. gets to be fairly similar looking if you look at the entire process.
MAXINFOTECHVISION - Tuesday, January 24, 2017 - link
http://max-infotechvision.blogspot.in/IS A POWER FULL PLATFORM TO TAKE REVIEW
petuma - Friday, February 3, 2017 - link
BTW - For the curious, the fluid used in the demo at CES is not Novec 72DA. Fluorinert FC-72 or perfluorohexane was used. It also runs on Novec 7100, the fluid preferred by Bitcoin companies like Bitfury.- demo co-creator
petuma - Thursday, November 2, 2017 - link
A colleague just reminded me to correct a misstatement in the above article. Since I built the tank with Adachi-San of Gigabyte at my Lab in St. Paul, I feel qualified to do this. The fluid is not Novec 72DA. Novec 72DA is a reasonably aggressive azeotropic solvent that might have damaged the computer. Systems like this have been run in pure Novec hydrofluoroethers like Novec 7100 (popular in Bitcoin) without the dichloroethyene and IPA. Here it was in fact running in Fluorinert FC-72.