WIXLIB

REIMAGINING IC ENGINEDEVELOPMENT LEVERAGINGNEXT-GENERATION HPC & AISIBENDU SOMManager & Principal Computational ScientistEnergy Systems DivisionArgonne National LaboratoryRONALD O. GROVER, JR.Staff Researcher, General Motors Research & DevelopmentCo-chair, US DRIVE Advanced Combustion and EmissionsControl (ACEC) Tech Team

MODELING ECOSYSTEM THEN What was the common working relationship?1. Centered around a common code platform2. Each entity had particular strengths3. Strong user community and open communicationWhy did this working model change?1. Complexity of proprietary combustionsystems (meshing/sub-models)2. Need for stronger parallelization(flow/chemistry)3. User support & code maintenance4. Growth and emergence of commercialcodes

MODELING ECOSYSTEM NOW 1. Institutions centered around a growing‘web’ of commercial (COTS) and opensource options2. Implementing UDFs into commercialcodes with need for validation3. Heavy investment in nationallaboratory supercomputinginfrastructure (drive to exascale)4. Opportunities to accelerate machinelearning and artificial intelligence3

CAE IS VITAL FOR FUTURE PROPULSIONSYSTEMSHPC Growth & Reduction in Hardware2005 2007 2009 2011 2013 2015 2017 2019Growth in CAE HPC(Cores/FLOPS/Storage)Number ofDevelopment HardwareGoal: First time capable designs to reduce iterations in the physical worldASME ICEF Panel, 2019

INCREASED DEMAND FOR FAST, VALIDATEDMODELS1. Design direction– Directional accuracy is important2. Meet requirements or not– Accuracy is important (not much room for error)3. Hardware out of the loop– Accuracy is important (not much room for error)Can we do it faster? Can we do it cheaper? Can we do something new?

DOE HPC ROADMAP LEADS TOMULTIPLE EXASCALE SYSTEMSFY 2012FY 2016FY Y 2021FY 2022FY 2023ORNLHPE/AMDMIRA*THETAAURORAANLANLANLIBM BG/QCray/Intel KNLHPE/IntelCORIPERLMUTTERLBNLLBNLCray/Intel TYSIERRALLNLLANL/SNLLLNLIBM BG/QLANL/SNLLLNLCray/Intel Xeon/KNLIBM/NVIDIAHPE/IntelHPE/AMD*decommissioned

FOUR PRIMARY ALLOCATION PROGRAMSFor access to DOE Leadership Computing Facilities (OLCF and ALCF)10% Director’sDiscretionaryUP TO20%ECPExascale ComputingProject20% ALCCASCR LeadershipComputing Challenge50%

TOWARDS PREDICTIVE ENGINE SIMULATIONS Gold-standard datafor AI unearthnew physicsEXPERIMENTSOptical & Metal engines,RCM, APS, . Fast solvers Physics based & datadriven models for:Design newexperiments / conditionPELEDNS of engineconditionNEK5000Initial & BoundaryconditionsDNS & LES ofPACE enginesSOFTWARETECHNOLOGIESMath libraries,Co-design, PACE leads: M. Weismiller (DOE), M. McNenly (LLNL), S. Som (ANL), J. Szybist (ORNL), P. Miles (SNL)––––IgnitionSpraysTurbulent combustionKinetic mechanismsCOMMERCIALCFD TOOLS Scales on exascalearchitecture PACE models Used by industryUSCAR TECHTEAMS & OEMSEmphasis with Nek and commercial codesfor hybrid computing architectures

INCREASING COMPUTATIONAL EXPENSEVISIONMULTI-FIDELITY SIMULATION FRAMEWORK 0.1-0.5 mm grid sizeO (103-104) processorsHigh-fidelity wallresolved LES atlimited conditionsCAPABILITY COMPUTING 10 μm grid sizeO (104-105) processorsDNS for generatinggold-standard datasetsat select conditionsLEVERAGE AMULTI-FIDELITYSIMULATIONFRAMEWORK TO: Improve understandingof flow and combustionprocesses Develop physicsbased and data-drivensubgrid modelsHigh-throughputdesign spaceexploration –RANSCAPACITY COMPUTING 0.5-1 mm grid sizeO (100) processorsSurrogate Models fordesign optimization Perform simulationbased designoptimization Develop surrogatemodels for fastdesign optimization

EXISTING EFFORTS IN HPC & AIHPCHPC used as a microscope –illuminating processes that areinaccessible to experimentationHPC simulations providea benchmark for accuracy ofengineering simulationsPACE leads: M. Weismiller (DOE), M. McNenly (LLNL), S. Som (ANL), J. Szybist (ORNL), P. Miles (SNL)AIMachine learning and patternrecognition will be applied to Resolve decades-old problems(e.g. root causes of cyclic variability) Accelerate CFD simulations Develop reduced order modelsenabling optimal CFD-based design Develop data-driven, accurateand efficient sub-models

EXAMPLE OF CAPACITY COMPUTING Direct computation of combustion metrics over several computed engine cyclesFuel surrogate development matching chemical and physical propertiesCombined flame propagation (G-eqn), detailed kinetics simulations, and dynamic meshingNational laboratory supercomputing leveraging GPUs (Zero-RK solver)FlameKnockLES4ECE, 20211 1

EXAMPLE OF CAPABILITY COMPUTINGGPU chemistry allows use of highlydetailed reaction mechanismsFully coupled 3D CHT modeling foraccurate wall temperature distributionsASME ICEF Panel, 2019

COMPARISON OF EMISSIONPREDICTIONS Improvement in NOx & HC emissions Mixed results on CO and SootGeometryCycleMax # cells# species# reactionsNOxSootTurbulenceBaselineSectorClosed 170k4774Zel’dovichBest SectorSectorClosed 1M144900GRI 3.0Full-cylCylinderOpen 2.5M144900GRI 3.0CHT RANSEngineOpen 4.5M144900GRI 3.0Detailed PSMHiroyasuHiroyasuHiroyasuRANSRANSRANSRANSOther changesWall time / cycleCasesSpray and wall-film Intake swirl vaneCHT 2 hr 5 hr 3.5 days 2 weeks500602208DOE Annual Merit Review, 2019

PREDICTION OF ENGINE-OUT EMISSIONS USINGDEEP CONVOLUTIONAL NEURAL NETWORKS600 CFD SimulationsTraining Data: 400 imagesValidation Data: 100 imagesTest or Unseen data: 100 imagesTotal predicted variables: 3 (CO, HC, Smoke)SAE 2021-01-0414

CNN VS CFD MODEL PREDICTIONSCFD model is sector mesh, Hiroyasu Soot model, GRI 3.0 chemistry for NOxSAE 2021-01-0414

NEW DNS/LES DATA LEVERAGING EXASCALE COMPUTING NEK5000 Spectral element method (SEM) code:––––High numerical accuracy: Nth order tensor-product polynomials (N 5-15)Exponential (spectral) convergence with NHandle complex geometries with moving boundariesEfficient scaling on hybrid exascale architectures Objective: Perform gold-standard DNS/LES simulations for flowand develop/improve submodels for engineering simulationsENGINE SIMULATIONS ON THETA & SUMMIT SUPERCOMPUTERSPIs: Muhsin Ameen, Saumil Patel (ANL)

NEW DNS/LES DATA LEADING TOIMPROVED SUB-MODELSOPEN-CYCLE LES1 NEK5000 was used to perform DNS of GM’s TCC engine(at University of Michigan) on Theta– LES 95M grid points, scales on 16K procs– DNS 430M grid points, scales on 51K procsCLOSED-CYCLE DNSDNS NEAR-SPARK PLUGDNS – SACI COMBUSTIONLES – FUEL INJECTION2 DNS enables development of new heat transfer and combustion modelsin industry use codes (like CONVERGE) on DOE Exascale machines LES framework within the higher order code provides a effective crucibleto test efficacy of existing sub-models* ge-eddy-simulation-les1. S. Wu, M. Ameen, S. Patel, ASME ICEF2021-67671. 2. F. Colmenares, M. Ameen, S. Patel, ASME ICEF 2021-67848

INJECTOR-TO-EMISSION PREDICTION TOOLFast and accurateMACHINE LEARNING ACCELERATESX-RAY TOMOGRAPHY SEGMENTATIONCOUPLED INJECTOR-SPRAY SIMULATIONSWITH DETAILED CHEMISTRY2Computed Tomography (CT) SliceCFD-predictedinjection profileCTSegNet1iso-surfaceready for CFDMultiphase Flow Modeling Cavitation & erosion X-ray scanned geometry Transient needle dynamicsCombustion Modeling 2000 species PAH mechanisms Turbulence chemistry interaction Detailed surrogates, soot modelsCoupled FrameworkAbility to link injector performancewith resultant mixing field, combustiondevelopment, and pollutant formation1. Tekawade et al., International Society for Optics and Photonics, 2019. 2. Mondal, Magnotti, Torelli et al., SAE Int. J. Adv. & Curr. Prac. in Mobility, 2021

IN-NOZZLE FLOW SIMULATIONS ACCOUNTINGMANUFACTURING TOLERANCESVideo courtesy of Katie Matusik and Chris Powell (Argonne) Reconstruction of x-ray scanned geometry Extraction of needle motion profiles Account for surface finish CFD simulations capturing these effectsTorelli, Pei, et al. SAE Int. J. Fuels & Lubr. 11(4), 2018

DATA-DRIVEN EMULATOR USED TO PREDICTSPATIOTEMPORAL INJECTION PROFILEAddresses expense of injector simulationsA-M1 INJECTORSide-oriented single-holeinjector geometrySIMULATION – DATA – LEARNING (SDL) Machine Learning models emulate internalflow fields at orifice exit Emulated flowfields coupled with:– Lagrangian spray model1– Eulerian-Lagrangian SprayAtomization (ELSA) model2 Transfer learning underway to extendto other injectors and injection systemsEMULATED FLOWFIELDS ATORIFICE EXIT FOR STEADY ANDTRANSIENT LES SIMULATIONS Gaseous volume fraction (𝛼𝛼)Velocity components (𝑢𝑢, 𝑣𝑣, 𝑤𝑤)Turbulent kinetic energy (k)Liquid mass (𝑚𝑚𝑙𝑙 )1. Mondal. Magnotti, Torelli et al., SAE Int. J. Adv. & Curr. Prac. in Mobility, 2021. 2. Magnotti et al., LES4ECE, 2021.

ACCURATE SPRAY COMBUSTION PREDICTIONSAT A FRACTION OF THE COSTEmulator far less expensive than simulating the next point of interestInjectionMap fromCFD (“Truth”)Injection Mapfrom GP-basedEmulator38 MILLION TIMES LESS EXPENSIVEMax Temperature [K]Heat Release Rate [MJ/s]Error 1%Error 1%1. Mondal, Magnotti, Torelli et al., ASME ICEF 2021-67888, Accepted.

FIRST OF ITS KIND SIMULATION LINKSEROSION FROM AN X-RAY SCANNED INJECTORWITH SPRAY, COMBUSTION AND EMISSIONSNON-ERODED INJECTORERODED INJECTORBaseline InjectorEroded InjectorSootTotal Void FractionVelocity Magnitude [m/s]Internal flow simulations indicate that erosion leads to: Increased orifice exit diameter Reduction in fuel delivery rates of at least 2 – 3% Wider spray spreading angles1. Magnotti et al., ASME ICEF 2021-67775, Accepted.NOxTimeReacting spray simulations indicate that erosion leads to: Similar first and second stage ignition delays Shorter flame lift off length Higher soot and lower NOx production

UNIVERSAL COMBUSTION MODELENABLED BY ML1 New toolbox ML-CEMA (ML-accelerated chemicalexplosive mode analysis) is developed for advancedflame diagnostics and modeling. ML-CEMA for any fuel combustion– Sheds light on flame stabilization, auto-ignition, flamepropagation, extinction, etc.– Speed up turbulent combustion modeling (e.g., in LES) by 4X.Diagnostics of SACI2Modeling of turbulent partially premixed flames33TotalPremixedNon-premixedNormalized HRR210-24-22-20-18-16-14Crank angle, degML-CEMA identifies locations of premixed reaction fronts anddistinguishes between premixed and non-premixed flames1. Chao Xu et al, AIAA SciTech 2020. 2. Chao Xu et al, ASME ICEF 2021. 3. Chao Xu et al, CNF 2018

MULTI-COMPONENT DETAILED CHEMISTRYSIMULATIONS ENABLED WITH ANNArgonne’sFlamelet solverArgonne’s Flamelet Solver together with UnsteadyFlamelet Progress Variable (UFPV) model has beenextensively validated against engine data withdetailed chemistry and soot models1Flamelet tableGeneration of manifoldby solving flameletequations a prioriχ Accurate predictions in autoignition and unsteadyheat release during interaction phase1.2.Kundu et al. Transportation Engineering 2020.Owoyele, Kundu, and Pal, Proceedings of the Combustion Institute, 2020Mass fractionsCFD simulationsZ, Zv, χ, Captures both high temperature ignition andlow temperature chemistry (LTC)Deep learning techniques further circumvent theissues of high memory footprint and retrieval costassociated with large multi-dimensional flamelettables Mixture of Experts (MoE) approach2, combiningregression and clustering, bifurcates combustionmanifolds and learns large flamelet tables Allows for incorporation of high-dimensional tables fromlarge chemistry mechanisms

ACCELERATING ENGINE DESIGNOPTIMIZATION WITH MLActivO: Basic algorithmWEAK LEARNERWEAK LEARNERSTRONG LEARNER(SVM, BFM, etc.)EXPLORATIONEXPLOITATIONOPTIMIZATION PROGRESSSTRONG LEARNER(Committee of Machines, RF, etc.)Owoyele & Pal, ASME J. Energy Res. Technol. 2020