WIXLIB

WHITE PAPERExecutive SummaryTABLE OF CONTENTSExecutive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Enhancing Heat Removal Capability with Immersion LiquidCooling Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2EDA Performance Boost Beyond Power Efficiency . . . . . . . . . . . . . 3Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4SUPERMICROSupermicro is a global leader in high performance, greencomputing server technology and innovation. We provide ourglobal customers with application-optimized servers andworkstations customized with blade, storage, and GPUsolutions. Our products offer proven reliability, superior design,and one of the industry’s broadest array of productconfigurations, to fit all computational needs.Immersion liquid cooling systems have proven theireffectiveness in multiple applications, focusing primarily onimproving power efficiency. This means a tangible reduction inutility costs and an enhancement in the ability to run greener inthe data center. For software applications, such as EDA used inthe semiconductor industry that could be running on per-corelicensing models, the number of cores and clock speedsrequired to achieve the same level of computing powerdetermines the runtime cost-effectiveness. With this type oflicensing model, licensing cost reduction could be achieved byutilizing a CPU with fewer cores of higher clock speeds atequivalent computing power. However, the lower core-countCPU at a comparative level of computing power comes at asubstantially higher processor case temperature, makingconventional air-cooling extremely inefficient.Supermicro, joined by 3M, Intel , Kaori, and Samsung,demonstrates a proof-of-concept single-phase immersionliquid cooling system with our BigTwin server family of products. Supermicro’s partner Kaori developed an immersion-coolingtank filled with 3M Fluorinert Electronic Liquid FC-40. The test results show that Intel high-frequency chips with a lowercore count delivered high LINPACK performance, while Samsung DRAMs and Solid-State Drives achieved uncompromisedelectrical characteristics. Supermicro’s experimental outcome supports the idea that utilizing high clock speed chips with alower core count to drive for the same level of computing performance could benefit from the savings on software licensing

Enhancing Heat Removal Capability withImmersion Liquid Cooling SystemsThe heavy demand for computing power drives thedevelopment of newer processor chips with everincreasing TDP. Compared to older generations, theupcoming processors efficiently run at well over 250Watts TDP. As a result, this has the potential ofcollectively exceeding rack-level power capacity whenclustered. One way to mitigate this is by spreading outthese “hot silos” areas to distribute heat sources better.However, compromising compute density could result inadditive burdens on the network infrastructure. The extracost and heat generated come with the need to rely moreon fiber and high-speed networking. Supermicroconsiders that densely clustered compute nodes wouldstill be the commonly accepted configuration in the datacenter and for high performance computing (HPC)applications as long as the significant challenges ineffectively and economically removing heat can beresolved.Air as a medium for cooling will generally be less efficientand less effective than liquid. For example, one way tothink of this is that conventional air-cooling essentiallycools an object immersed in flowing air, utilizing air as anagent to remove the heat. When comparing an objectimmersed in a flowing liquid, the liquid conjunctionintroduces more intimate and closer contact than thegaseous conjunction. The higher heat transfer coefficientwith liquid than gas manifests the adequate number ofmolecular-scale particles densely engaging with theobject’s surface.Liquid agents can carry more heat than air does of equalvolume and be more thermally conductive than air.However, the main reason conventional cooling systemsutilize air as an agent is not its heat transfer capability butits nonconductivity and compatibility with most of thewidely used electronic devices in the electronic hardwareindustry.management on data center premises more costly.Moreover, given the ever-surging power density at therack level and the growing TDP at the chip level, it isforeseeable that one day air-cooling systems mayeventually hit the wall that may only be breached byadvanced cooling technologies like immersion liquidcooling systems.TYPICAL LIQUID COOLING SOLUTIONSLiquid cooling solutions are beneficial in improvingcooling efficiency, lowering overall wattageconsumption, and mitigating cost. Various field-provenliquid cooling techniques range from dissipating the heatdirectly from the source to physically dissipating the heatoff-premises.Liquid Engagement at Different StagesIn the heat transfer and exchange process, there arestages where liquid coolants could intervene as a coolingagent: Direct-To-Chip Liquid Cooling Solutions ‐ a liquidcoolant removes heat directly from high wattageprocessors. Then the liquid agent may exchange heatwith another liquid agent at the rack level or thefacility level. In-Row or In-Rack Liquid Cooling Solutions ‐ rack-levelor row-level liquid cooling systems where air-to-liquidheat exchange occurs at different system integrationlevels. The rear door heat exchanger (RDHx) is onetypical example of a rack-level liquid cooling solution. Immersion Liquid Cooling Solutions ‐ the whole orpart of a system is immersed in nonconductive inertdielectric liquids. Then these liquids transfer heatwhile remaining in a liquid state via a liquid-to-liquidheat exchanger or convert to a gaseous state whilecarrying the latent heat. Then the gaseous agent istransformed back into a fluid via a condenser.Visit the Supermicro Liquid Cooling Solutions page for moredetail.With climate change, those used-to-be trivial problemssuch as airborne particles, gaseous pollution, globalwarming, and volcanic activities make air-inletEDA Applications Get a Performance Boost Supermicro, Inc. May, 20222

EDA Performance Boost Beyond Power EfficiencyImmersion liquid cooling systems offer efficiency inoperating IT and data center equipment at coolertemperatures, thus lowering total utility costs. Apart fromthe immediate benefit, additional tips brought tangiblecost savings to the data center by saving licensing fees.The software has typically been priced on a per-core orper-socket model at the commercial level. Per-corelicensing model becomes dominant as the marketdemands more compute power per socket or more cores.Even industries that used to stick to the old school persocket pricing model have started to charge higherlicensing fees depending on the number of cores per CPU.As clock speeds increase and higher wattage CPUsbecome a norm, IT workloads that leverage more coresmaximizing software license utilization out of the lessper-socket fees, can no longer play the old trick aftersoftware companies pulled the plug. Thus, gearing up tothe same level of computing power using fewer cores canbe translated into a dollar value by reducing the numberof runtime licenses required. Low-core high-frequencyCPUs will be a rewarding substitute as long as theexcessive heat driven by running the chips faster can beresolved. Given the surging wattages in the nextgeneration of CPUs, a conventional approach using aircooling starts losing its edge.Supermicro developed a proof-of-concept unit based onthe X11 generation of our BigTwin 2029BT-HNTR 2U 4Node high-density server system equipped with Intel Xeon Gold 6250 processors. The baseline comparison isthe same system using the Intel Xeon Gold 6244processors. Both configurations are populated withSamsung M393A2K40DB3-CWE 16GB 3200Mbps RDIMMand PM983 960GB SSD. We immersed our system usingthe Intel Xeon Gold 6250 in a Kaori liquid cooling tankfilled with 3M Fluorinert Electronic Liquid FC-40. Thesystem using the Intel Xeon Gold 6244 is cooled byconventional air-cooling. The CPU calculation powerperformance is benchmarked with LINPACK, whilememory and SSD performance are benchmarked withStream and FIO.Figure 1 - BigTwin 2029BT-HNTRThe resulting conclusions found that the Intel Xeon Gold 6250 in a low temperature chamber generatednotably 8% higher LINPACK results, averaging from thefour server nodes compared to the Intel Xeon Gold 6244in our air-cooled test environment. Notably, theexperiment would have to be performed in an unrealisticlow temperature chamber at freeze point to utilize the fullspeed of the Intel Xeon Gold 6250 processors.Figure 2 – Air Cooling Baseline ResultIn the case of the immersion-cooled Intel Xeon Gold6250, the system at room temperature achieved slightlyhigher LINPACK results than the previous lowtemperature chamber results. Here the immersion liquidcooling configuration delivered 12% lower powerconsumption than the previous low temperaturechamber. However, the actual power savings would haveachieved approximately 54% when accounting for theexcessive power required to bring the temperature tofreeze point in the air-cooled configuration.EDA Applications Get a Performance Boost Supermicro, Inc. May, 20223

Figure 3 – Intel Xeon Gold 6250 Air vs. Liquid Performance ResultThe memory performance results from the Stream tooldemonstrated a consistent memory bandwidth output asopposed to the baseline configurations. In addition,consistency could also be observed in the NVMe Randomand Sequential Read/Write performance based on the FIOtool.The Intel Xeon processors and the Samsung DIMMs andSSDs demonstrated excellent compatibility with the 3M Fluorinert Electronic Liquid FC-40 without any signs ofdamage or degradation. In addition, the Kaori immersionliquid cooling tank with the plated heat exchange systemprovided a solid stand-alone test bench in the labenvironment.Processors with higher frequencies will bring along higherLINPACK results. Utilizing lower core count processors athigher frequencies to achieve similar performance levelsas higher core count processors at lower frequenciescould bring cost saving benefits in the runtime licensing,a cherry on top of the significant power consumptionmerits.ConclusionSupermicro delivers the highest performance systems todata center operators. New technologies in the electronicdesign field, AI, and data analytics will require the fastestand most capable CPUs and GPUs for these newworkloads. In addition, the higher wattage systems mayFigure 4 - Supermicro Proof-Of-Concept ImmersionLiquid Cooling Demo Unit Jointly Developed with3M, Intel, Kaori, and Samsungrequire liquid cooling for those data center environmentspushing the limits of air cooling.EDA Applications Get a Performance Boost Supermicro, Inc. May, 20224