WIXLIB

8 NERSC ANNUAL REPORT 2013Edison Opens New Doorsfor Early UsersFrom improving carbon sequestration models to creatingvirtual universes, NERSC’s newest supercomputer is a hit“The memorybandwidth makesa huge differencein performance forour code.”before any supercomputer is accepted at nersc, the system is rigorously tested as scientistsare invited to put the system through its paces during an “early science” phase. While the main aim of thisperiod is to test the new system, many scientists are able to use the time to significantly advance their work.Here’s a look at the work NERSC’s new Edison Cray XC30 helped some early users accomplish in 2013.The Fate of Sequestered CO2David Trebotich is modeling the effects of sequestering carbon dioxide (CO2) deep underground. Theaim is to better understand the physical and chemical interactions between CO2 , rocks and the minute,saline-filled pores through which the gas migrates. This information will help scientists understandhow much we can rely on geologic sequestration as a means of reducing greenhouse gas emissions,which cause climate change.Unlike today’s large-scale models, which are unable to resolve microscale features, Trebotich models thephysical and chemical reactions happening at resolutions of hundreds of nanometers to tens of microns.His simulations cover only a tiny area—a tube just a millimeter wide and not even a centimeter long—but in exquisitely fine detail.“We’re definitely dealing with big data and extreme computing,” said Trebotich, a computational scientistin Berkeley Lab’s Computational Research Division. “The code Chombo-Crunch generates datasets ofone terabyte for just a single, 100-microsecond time-step, and we need to do 16 seconds of that to matcha ‘flow-through’ experiment.” Carried out by the Energy Frontier Research Center for Nanoscale Controlof Geologic CO2 , the experiment captured effluent concentrations due to injecting a solution of dissolvedcarbon dioxide through a tube of crushed calcite.

Research News 9The fine detail of this simulation—which showscomputed pH on calcite grains at 1 micronresolution—is necessary to better understand whathappens when the greenhouse gas carbon dioxideis injected underground rather than being releasedinto the atmosphere to exacerbate climate change.david trebotich, lawrence berkeley nationallaboratoryEdison’s high memory-per-node architecturemeans that more of each calculation (andthe resulting temporary data) can be storedclose to the processors working on it.Trebotich was invited to run his codes tohelp test the machine and his simulationsran 2.5 times faster than on Hopper, aCray XE6 and the previous flagship system,reducing the time it takes him to get asolution from months to just weeks.“The memory bandwidth makes a hugedifference in performance for our code,”Trebotich said.The aim is to eventually merge such finelydetailed modeling results with large-scalesimulations for more accurate models.Trebotich is also working on extending hissimulations to shale gas extraction usinghydraulic fracturing. The code frameworkcould also be used for other energyapplications, such as electrochemicaltransport in batteries.Better Combustion for New FuelsJackie Chen and her research team at SandiaNational Laboratories are using Edisonto investigate how to improve combustionusing new engine designs and fuels such asbiodiesel, hydrogen-rich “syngas” from coalgasification and alcohols like ethanol. Hergroup models the behavior of burning fuels bysimulating some of the underlying chemicaland mixing conditions found in thesecombustion engines in simple laboratoryconfigurations, using a direct numericalsimulation code developed at Sandia.During Edison’s early science period,Chen and colleagues modeled hydrogenand oxygen mixing and burning in atransverse jet configuration commonlyemployed by gas turbine combustors inaircraft engines and industrial powerplants. Their simulations were performedin tandem with experimentalists tounderstand the complex flow field affectedby reaction and how it manifests itself ininstabilities. This information is criticalto understanding both the safety andperformance of the device.The team was able to run their directnumerical simulation code, S3D, on100,000 processors at once on Edisonand saw a four- to five-fold performanceimprovement over Hopper, Chen said.“In the end, what really matters isn’t thetechnical specifications of a system,” saidChen. “What really matters is the scienceyou can do with it, and on that count,Edison is off to a promising start.”Our Universe: The Big PictureZarija Lukic, a cosmologist with BerkeleyLab’s Computational Cosmology Center (C3),used Edison to model mega-parsecs of spacein an attempt to understand the large-scalestructure of the universe.

10 NERSC ANNUAL REPORT 2013Since we can’t step outside our own universeto see its structure, cosmologists like Lukicexamine the light from distant quasars andother bright light sources for clues. Whenlight from these quasars passes throughclouds of hydrogen, the gas leaves adistinctive pattern in the light’s spectrum.By studying these patterns (which look likeclusters of squiggles called a Lyman alphaforest), scientists can better understandwhat lies between us and the light source,revealing the process of structure formationin the universe.Researchers use a variety of possiblecosmological models, calculating for eachthe interplay between dark energy, darkmatter and the baryons that flow intogravity wells to become stars and galaxies.Cosmologists can then compare these virtualuniverses with real observations. Using theNyx code developed by Berkeley Lab’s Centerfor Computational Sciences and Engineering,Lukic and colleagues are able to create virtual“what-if” universes to help cosmologists fillin the blanks in their observations.“The ultimate goal is to find a single physicalmodel that fits not just Lyman alpha forestobservations but all the data we have aboutthe nature of matter and the universe, fromcosmic microwave background measurementsto results from experiments like the LargeHadron Collider,” Lukic said.Working with 2 million early hours on Edison,Lukic and collaborators performed one ofthe largest Lyman alpha forest simulationsto date: the equivalent of a cube measuring370 million light years on each side.“With Nyx on Edison we’re able—for the firsttime—to get to volumes of space large enoughand with resolution fine enough to createmodels that aren’t thrown off by numericalbiases,” he said. Lukic expects his work onEdison will become “the gold standard forLyman alpha forest simulations.”Large-scale simulations such as these willbe key to interpreting the data from manyupcoming observational missions, includingThis volume rendering shows the reactivehydrogen/air jet in crossflow in a combustionsimulation. hongfengyu, university ofnebraskaH20TH2

Research News 11the Dark Energy Spectroscopic Instrument,he added.Growing Graphene and Carbon NanotubesVasilii Artyukhov, a post-doctoral researcherin materials science and nanoengineeringat Rice University, used 5 million processorhours during Edison’s testing phase tosimulate how graphene and carbon nanotubesare “grown” on metal substrates usingchemical vapor deposition.Graphene (one-atom thick sheets of graphite)and carbon nanotubes (essentially graphenegrown in tubes) are considered “wondermaterials” because they are light but strongand electrically conductive, among otherproperties. These nanomaterials offer theprospects of vehicles that use less energy;thin, flexible solar cells; and more efficientbatteries—that is, if they can be producedon an industrial scale.Today’s methods for growing grapheneoften produce a mish-mash of nanotubeswith different properties; for instance,semiconducting types are useful forelectronics and metallic types forhigh-current conduits. For structuralpurposes the type is less important thanthe length. That’s why the group is alsoinvestigating how to grow longer tubeson a large scale.For Artyukhov’s purposes, Edison’s fastconnections between processor nodes hasallowed him to run his code on twice asmany processors at speeds twice as fast asbefore. The Rice University researcher isalso using molecular dynamics codes,“which aren’t as computationally demanding,but because Edison is so large, you can runmany of them concurrently,” he said.Using the Nyx code on Edison, scientistswere able to run the largest simulation of its kind(370 million light years on each side) showingneutral hydrogen in the large-scale structureof the universe. The blue webbing representsgas responsible for Lyman-alpha forest signals.The yellow are regions of higher density, wheregalaxy formation takes place.casey stark,lawrence berkeley national laboratory

12 NERSC ANNUAL REPORT 2013IceCube IDs ‘Bert’ and ‘Ernie’NERSC computers essential for finding neutrino eventin timely manner at South Pole“The large amountof computingresources at NERSC. was essentialto finding theseneutrino events in atimely manner.”data being collected at “icecube ,” the neutrino observatory buried in the South Pole ice,nd analyzed at NERSC is helping scientists better understand cosmic particle accelerators.In 2013, the IceCube Collaboration announced that it had observed 28 extremely high-energyevents that constitute the first solid evidence for astrophysical neutrinos from outside our solarystem, according to Spencer Klein, a senior scientist with Berkeley Lab and a long-time memberof the IceCube Collaboration. These 28 events include two of the highest energy neutrinos everreported, dubbed Bert and Ernie.NERSC’s supercomputers were used to sift out neutrino signals from cosmic “noise” in the IceCubeobservations, which were published in the journal Science in November 2013.“The large amount of computing resources at NERSC allowed me to set up my simulations and run themright away, which was essential to finding these neutrino events in a timely manner,” said Lisa Gerhardt,who at the time of her discovery was with Berkeley Lab’s Nuclear Science Division and is now with NERSC.These results provide experimental confirmation that something is accelerating particles to energies above50 trillion electron volts (TeV) and, in the cases of Bert and Ernie, exceeding one quadrillion electron volts(PeV). While not telling scientists what cosmic accelerators are or where they’re located, the IceCube resultsdo provide them with a compass that can help guide them to the answers.Cosmic accelerators have announced their presence through the rare ultra-high energy version of cosmicrays, which, despite the name, are electrically charged particles. It is known that ultra-high energy cosmicrays originate from beyond our solar system, but the electrical charge they carry bends their flight pathsPROJECT TITLENERSC PILEAD INSTITUTIONNERSC RESOURCES USEDSimulation and Analysisfor IceCubeChang Hyon HasLawrence BerkeleyNational LaboratoryCarver, Dirac, PDSF

Research News 13This event display shows “Ernie,” one of twoneutrino events discovered at IceCube whoseenergies exceeded one petaelectronvolt (PeV). Thecolors show when the light arrived, with reds beingthe earliest, succeeded by yellows, greens andblues. The size of the circle indicates the number ofphotons observed.as they pass through interstellar magneticfields, making it impossible to determinewhere in the universe they originated.However, as cosmic ray protons and nuclei areaccelerated, they interact with gas and light,resulting in the emission of neutrinos withenergies proportional to the energies of thecosmic rays that produced them. Electricallyneutral and nearly massless, neutrinos are likephantoms that travel through space in astraight line from their point of origin, passingthrough virtually everything in their pathwithout being affected.The IceCube observatory consists of 5,160basketball-sized detectors called DigitalOptical Modules (DOMs), which wereconceived and largely designed at BerkeleyLab. The DOMS are suspended along 86strings embedded in a cubic kilometer ofclear ice starting one and a half kilometersbeneath the Antarctic surface. Out of thetrillions of neutrinos that pass through theice each day, a couple of hundred will collideicecube collaborationwith oxygen nuclei, yielding the bluelight of Cherenkov radiation that IceCube’sDOMs detect.The 28 high-energy neutrinos reported inScience by the IceCube Collaboration werefound in data collected from May 2010 toMay 2012. In analyzing more recent data,researchers discovered another event that wasalmost double the energy of Bert and Ernie,dubbed “Big Bird.”As to the identity of the mysterious cosmicaccelerators, Klein thinks these early resultsfrom IceCube favor active galactic nuclei, theenormous particle jets ejected by a black holeafter it swallows a star.“The 28 events being reported are diffuseand do not point back to a source,” Kleinsaid, “but the big picture tends to suggestactive galactic nuclei as the leading contenderwith the second leading contender beingsomething we haven’t even thought of yet.”DOE PROGRAM OFFICEFULL STORYPUBLICATIONNP—Nuclear ia-icecube/Ice Cube Collaboration, “Evidence forHigh-Energy Extraterrestrial Neutrinos atthe IceCube Detector,” Science, November22, 2013, 342(6161), doi: 10.1126/science.1242856

14 NERSC ANNUAL REPORT 2013Spotting Hotspot VolcanoesNERSC computers helped scientists detect previouslyunknown channels of slow-moving seismic waves in Earth’supper mantle“Without thesenew techniques,the study’s analysiswould have takenabout 19 yearsto compute.”using supercomputers at nersc, scientists have detected previously unknown channels of slow-movingseismic waves in Earth’s upper mantle. This discovery helps to explain how “hotspot volcanoes”—the kind thatgive birth to island chains like Hawaii and Tahiti—come to exist.The researchers found these channels by analyzing seismic wave data from some 200 earthquakes usinghighly accurate computer simulations of how these waves travel through the Earth’s mantle—the layerbetween the planet’s crust and core. These analyses allowed them to make inferences about the structureof convection in the mantle, which is responsible for carrying heat from the planet’s interior to the surfaceto create hotspot volcanoes.“We now have a clearer picture that can help us understand the ‘plumbing’ of Earth’s mantle responsiblefor hotspot volcano islands like Tahiti and Samoa,” said Scott French, a University of California at Berkeleygraduate student and lead author on the study, which was published in the journal Science.Unlike volcanoes that emerge from collision zones between tectonic plates, hotspot volcanoes form in themiddle of plates. Geologists hypothesize that mantle plumes—hot jets of material that rise from deep insidethe mantle, perhaps near the core-mantle boundary—supply the heat that feed these mid-plate volcaniceruptions. But that model does not easily explain every hotspot volcano chain or why large swaths of thesea floor are significantly hotter than expected.To learn more about the mantle’s structure and convection, French looked at seismic data from hundredsof earthquakes around the world. These energetic waves travel for long distances below Earth’s surface andare tracked by a global network of seismographs. Because the waves change as they travel through differentmaterials, scientists can gain insights about the structure and composition of the substances beneath thePROJECT TITLENERSC PILEAD INSTITUTIONNERSC RESOURCES USEDGlobal Full-Waveform SeismicTomographyBarbara RomanowiczUC BerkeleyHopper

Research News 15This 3D view of the top 1,000 kilometers ofEarth’s mantle beneath the central Pacific shows therelationship between seismically slow “plumes”and channels imaged in the UC Berkeley study.Green cones on the ocean floor mark islandsassociated with “hotspot” volcanoes, such asHawaii and Tahiti.berkeley seismologicallaboratory, uc berkeleyplanet’s surface by analyzing the alteredwaves. But because they cannot directlyobserve the structures lurking belowEarth’s surface, seismic tomographycan be harder to interpret.“To simulate seismic data for interpretation,people have traditionally used broad layersof approximation because it would havebeen too computationally heavy to run anexact model using numerical simulations,”said French.As a graduate student at UC Berkeley in2010, Vedran Lekic used supercomputersat NERSC to develop a method to accuratelymodel seismic wave data while keepingcomputing time manageable. French recentlyrefined this approach, further improvingthe accuracy of the tomography, whilealso scaling the technique to solve largerproblems. Together, these efforts resultedin higher-resolution images of patternsof convection in the Earth’s mantle.One simulation of one earthquake takes about144 computer processor hours on NERSC’sHopper system, French said, “But we neededto run this simulation for 200 individualearthquakes to get an accurate seismic model.Our model improves with each run.”This tomographic model eventually allowedthe team to find finger-like channels oflow-speed seismic waves traveling in themantle about 120 to 220 miles beneath thesea floor. These channels stretch about 700miles wide and 1,400 miles apart. Seismicwaves typically travel about 2.5 to 3 milesper second at this depth, but waves in thechannels traveled about 4 percent slower.Because higher temperatures slow downseismic waves, scientists inferred thatmaterial in the channels must be hotterthan surrounding material.Without these new techniques, the study’sanalysis would have taken about 19 yearsto compute, Lekic noted.DOE PROGRAM OFFICEFULL lcanoes/Scott French, Vedran Lekic and BarbaraRomanowicz, “Waveform TomographyReveals Channeled Flow at the Base ofthe Oceanic Asthenosphere,” Science,342(6155), 227-230 (2013), doi: 10.1126/science.1241514

16 NERSC ANNUAL REPORT 2013Unlocking the Secrets of SolarWind TurbulenceNERSC resources, visualizations help scientists forecastdestructive space weather“Thesesimulationsopened upa whole newarea of physicsfor us.”as inhabitants of earth, our lives are dominated by weather. Not just in the form of rain and snow fromatmospheric clouds, but also charged particles and magnetic fields generated by the Sun—a phenomeno

queued up to take advantage of the new supercomputer’s capabilities. Edison was the first Cray supercomputer with Intel processors, a new Aries interconnect and a dragonfly topology. The system was designed to optimize data motion—the primary bottleneck for many of our applicatio