<span class="var-sub_title">The Facility Perspective on Liquid Cooling: Experiences and Proposed Open Specification</span> SC18 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

The Facility Perspective on Liquid Cooling: Experiences and Proposed Open Specification


Authors: Dale Sartor (Lawrence Berkeley National Laboratory), Herbert Huber (Leibniz Supercomputing Centre), Anna Maria Bailey (Lawrence Livermore National Laboratory), Dave Martinez (Sandia National Laboratories), Michael Patterson (Intel Corporation), David Grant (Oak Ridge National Laboratory)

Abstract: As compute densities increase, there is growing demand to more effectively cool power-dense equipment and improve energy efficiency with compressor-less cooling. This BoF will explore the steps necessary to take advantage of warm liquid-cooling in the data-center and introduce an open-specification for a secondary fluid warm liquid-cooled rack.. Lawrence Berkeley National Laboratory and China Institute of Electronics steer this initiative and seek input from the HPC community. This BoF will feature a panel of seasoned operations managers from major supercomputing centers to talk about strategies for effectively enabling warm-water cooling, including a discussion on the need for industry standards.

Long Description: While it is possible to tap into existing chilled water distribution systems, using warmer water sources can substantially increase efficiency, simplify the cooling system design, decrease the potential for condensation in the data center, and even raise the potential to capture and reuse the heat generated in a data center. The panel will discuss issues and opportunities associated with cooling high power density IT equipment with warm liquids, and then cooling those liquids with energy efficient compressor-less cooling.

Topics for discussion based on the experience of the panel and audience: 1. Separate warm water distribution system, and get your warm water loop in place: a. Amply size piping for future growth b. Install T’s and valves to facilitate adding and removing components c. Select materials for corrosion resistance and higher temperatures 2. Cool with a cooling tower and/or a dry cooler for water conservation 3. If a chiller is used, select for optimum efficiency at warm water temperatures 4. Option for warm liquid cooling: a. Rear door heat exchangers b. Cold plate and liquid to the chip heat exchangers c. Hybrid systems and systems that cascade the liquid from colder demands to warmer d. Immersion systems placing IT components directly in non-conductive fluids e. Note a data center can be designed for more than one solution, and may sequence from liquid to air cooling solutions to higher temperature direct liquid cooled solutions. 5. Procuring a liquid cooled system a. Specify warm water (at least ASHRAE W3) b. Look at emerging open specifications for liquid cooled racks

The open specification focuses on a non-proprietary multi-vendor rack for warm liquid cooled servers compatible with existing open rack standards such as the Open Compute Project (OCP), Project Scorpio, and Open19. .

Many vendors have entered the market with liquid cooling solutions, however most/all are proprietary and generally non-compatible with each other. This is acceptable for some homogeneous compute environments, but less acceptable for non-homogeneous compute environments where standard multi-vendor solutions are strongly preferred. Buyers in the market would benefit from greater standardization. The hypothesis is that incompatible proprietary systems are a market barrier that will continue to inhibit adoption of warm liquid cooling.

The specification does not include the heat exchange configuration within the IT equipment. These details will be left to solution providers as long as their design is compatible with the rack specification. Likewise the design of the cooling distribution unit (CDU) is not included. Compatibility is driven by the wetted materials, the transfer fluid, connectors, and operating conditions. This will allow innovation and competition in the market while allowing multiple generations of IT equipment hardware to benefit from the same rack infrastructure. Owner/operators will not be tied to one vendor during IT refresh cycles.

In the context of this specification, the liquid cooling loop consists of a secondary fluid pumped through a rack manifold from a CDU. The heat exchangers within the IT equipment are connected to the rack manifold via either rigid or flexible tubing and quick connects.


URL: https://eehpcwg.llnl.gov/conf_sc17a-copy.html



Back to Birds of a Feather Archive Listing