Emerging Opportunities In HPC Cloud & Co-location Services At The .

Transcription

Emerging Opportunities in HPC Cloud & Co-location Servicesat the University of Nevada - Las VegasJoseph LombardoExecutive Director, UNLV National Supercomputing InstituteJim DonovanVP of Product, Wasabi Technologies, Inc.2019 Technology Exchange in New Orleans, December 2019Collaborations with the Cleveland Clinic Lou Ruvo Center for Brain Health and the Nevada Institute of Personalized Medicine are currently funded by 2Institutional Development Awards (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health: #P20GM109025 &#P20GM121325. The content is solely the responsibility of the author(s) and does not necessarily represent the official views of the National Institutes of Health.

Agenda About the National Supercomputing Institute (NSI) About Switch About Altair About Wasabi

About the NSIFull-service supercomputing facilityMission for excellence in education and research insupercomputing and its applicationsProvides supercomputing training and services to academicand research institutions, government and private industryFacilitates high-technology economic diversification inNevada by providing services not available in the privatesector and by promoting partnerships between universityfaculty and external

NSI @ Switch 2014 - UNLV moved its NSIfacilities to the Switch facility inLas Vegas Hosted on Cherry Creeksystem – large Intel system forscientific and economic R&D 30,000 compute cores Intel Xeon E5-2697v2 12C2.700GHz, Intel Truscale, IntelXeon Phi 7120P Dedicated Research Network(DMZ) with 200Gb/s potential

NSI Computing ChallengesNumerous and complex workloads Hundreds of projects worldwide Highly compute-intensive researchMassive data needs Users must access massive data remotely to do their workTime-sensitive projects Many NSI projects have critical governmental and environmentalsignificance, so timely and reliable performance is a key requirement

New DataNIH: Data Managementand Statistics Core(s)Data Flow SummaryPrepared by Joseph LombardoExe. Director, National SupercomputingInstitute (NSI)CNTN & CEPM Data Core PIlombardo@nscee.edu(CEPM)FreeSurferExcel & ExomeData(MRI data, which is converted by an (Screening & Demographic Interviews –NSI application to CSV format) Double Entry is supported by OpenClinica )(NeuroPsych data saved ascomma-separated values - CSVSAM & BAM)CSV Data(comma-separated values)Data & Software Archives(freestanding files for download – no access to theOpenClinca environment is allowed)Firewall SSL EncryptionFirewall SSL EncryptionFirewall SSL EncryptionRemote ResearcherDouble Entry Lou RuvoNSIShared Archive(National Supercomputing Institute )OpenClinica(NSI’s ODM clinical data extractionapplication is applied here tocreate the archive files )1. Reformat data study meta-data(NSI’s multiple data parsing applications automates theconversion of data from other applications (Excel) to a formatthat OpenClinica’s data import application understands)OpenClinica Output(produces XML)(non-exhaustive)XMLSPSSCSVNote: these formats can beexported to programs thatstore data in tables, suchas Microsoft Excel3. XML - Extensible Markup Language)(this data gets imported into OC) 1. CNTN Defined Archive(XML, SPSS, CSV & Data rCNTNresearcher2. OpenClinica (OC) data import applicationFirewall SSL Encryption2. Subscriber Defined DownloadExcelFirewall SSL Encryption(same as CNTN Defined Archive with the exception that theremote user selects specific fields in the record that they need CNTN will use the selected choices to determine which datais most valuable to the remote research community)3. Software ArchiveFreeSurfer Software Suite: An open source software suite for processing and analyzing (human) brain MRI images.Statistical Package for the Social Sciences (SPSS): SPSS is a software package used for logical batched and non-batched statistical analysis.Operational Data Model (ODM)-XML: a vendor-neutral, platform-independent format for exchanging and archiving clinical and translational research data, along with their associated metadata,administrative data, reference data, and audit information.ODM clinical data extraction application: NSI’s software that produces an extract of clinical data from an ODM file produced by OpenClinica and then writes MRI thickness data for left and righthemispheres to a new file.NSI application: Software created by the National Supercomputing Institute (NSI) in support of CNTN.Secure Sockets Layer (SSL): is a protocol for encrypting data transferred between two computers.In collaboration with the Cleveland Clinic Lou Ruvo Center for Brain Health and the Nevada Institute of Personalized Medicine. Funded by two Institutional Development Awards (IDeA) from the National Institute of General MedicalSciences of the National Institutes of Health: #P20GM109025 & #P20GM121325. The content is solely the responsibility of the author(s) and does not necessarily represent the official views of the National Institutes of Health.

Cloud & Co-location

UNLV & Altair CollaborationThe University of Nevada, Las Vegas (UNLV) is home to the“Cherry Creek II” supercomputer, housed in Switch’s Las VegasSUPERNAP data center. The system is among the fastest andmost powerful in the world. It gives scientists around the globeaccess to significant high-performance computing power.Innovation Intelligence 8

UNLV & Altair CollaborationInnovation Intelligence All that computing power doesn’t orchestrate itself — so UNLVenlisted Altair to deploy the Altair PBS Works highperformance computing (HPC) management suite to securelymanage Cherry Creek II’s compute workload, simplifyingaccess to and utilization of the supercomputer’s capabilities andcapacity. Users can easily create, access, and manage physicaland virtual appliances on Cherry Creek II and run Altair’sHyperWorks simulation software as well as third-partyapplications.9

UNLV & Altair CollaborationThe collaboration “sets the stage for UNLV to become an evengreater supercomputing powerhouse for the Southern Nevadacommunity.” — Joseph Lombardo, Executive Director of theUNLV National Supercomputing InstituteInnovation Intelligence 10

Altair SolutionsThe industry-leading Altair PBS Works workload management solution includes all the toolsyou need to schedule, tune, and accelerate your jobs, including: Altair PBS Professional – a fast, powerful workload manager designed to improveproductivity, optimize utilization and efficiency, and simplify administration for HPC clusters,clouds, and supercomputers. Altair Access – a simple, powerful, and consistent interface for submitting and monitoringjobs on remote clusters, clouds, and other resources, allowing engineers and researchers tofocus on core activities. Altair Control – an easy-to-use web application for monitoring and managing jobs andnodes in an HPC environment. Altair SAO – an advanced tool for software asset optimization, built so you can right-size yourorganization’s software portfolio using real data to make informed business decisions.11

Powerful, flexible customization capabilities -- can beeasily extended by adding site-specific processingplugins/hooks Improved system manageability and extensibility:Innovation Intelligence Lightweight solutionVery easy to manageNot dependent on any specific operating system12

Wasabi Cloud Storage ForEmerging Opportunities in HPC Cloud & Co-location Services12 December 201913

Wasabi Introduction Mission: Low Price, High Performance, Secure Object Storage Started in 2015 by Carbonite’s founding team (David Friend & Jeff Flowers) Privately held & well funded with 80M invested to date Available via Internet2 Cloud Exchange since 2018 Product: Cloud Object Storage as a Service Comparable to: AWS S3, Microsoft Azure Blob Storage,and Google Cloud Platform (GCP) Storage Thousands of customers & partners across all verticals14

Wasabi’s Value PropositionOptimal price, performance, and protection for object storage15

PriceLower cost than all other major object storage providers (more info @ wasabi.com/pricing) Wasabi’s flat fee of .0059/GB/mo( 5.99/TB/mo) for storage is adisruptor relative to competitors No charge for egress (downloads)(vs. up to .09/GB with AWS S3) No charge for API requests (unlike allother public cloud storage providers) No complex storage tiers (Wasabiis hot storage at cold storage prices)Storage Costs For 1 PB of storageWith 20% Data Egress Per Year16

PerformanceBuilt-for-speed file system Purpose-built file system leveraging hardware technology Enables significant cost reduction & performance improvements Faster than AWS S3 & meaningful time-to-first-byte (TTFB) advantages Highly distributed architecture providing exabyte-scale storageUser Servers100 GbESwitchingDatabase ServersStorage ServersWasabi-built software deployed onleading-edge hardware in top-tier data centersHigh performanceenabled by Wasabi’ssystem architectureCompute ThreadsSample test results for write performance with1 MB objects across different compute thread counts17

ProtectionBuilt for scale, durability, security and compliance Durability & Availability 11 x 9s data durability with exabyte scale 99.99% availability SLA with multiple data centers & Wasabi Bucket Replication Data integrity checks at time of upload, download, and every 90 days Security All data encrypted in transit and at rest Immutable buckets prevent accidental deletion/modification Strong identity & access management & multi-factor authentication Compliance with industry privacy,security, and data center standards18

Multiple Public Private Interconnect OptionsEnables high-speed exchange between Wasabi storage & customer compute resources Public internet (N x 10 Gb/s) Wasabi or AWS/Azure/GCP ‘Direct Connect’ (N x 1 or 10 Gb/s) Wasabi Ball Transfer Appliance (up to 100 TB per appliance)Wasabi BallTransferApplianceWasabi eu-central-1 region(Amsterdam)Wasabius-west-1 region(Oregon)Wasabius-east-1 &us-east-2 regions(Northern Virginia)Public Cloud orPrivate Data CenterComputeWasabi apac-1 region(Tokyo – Dec 2019)Key networking, compute &data center partners include:19

AWS S3 CompatibilityWill my existing AWS S3 applications work with Wasabi? Wasabi fully supports the AWS S3 & IAM APIsAWS S3 & IAM APIs Wasabi looks just like an Amazon S3 implementation Same AWS API constructs for storage & identity management No need to change apps you may be currently using with AWS S3 Interop Categories Include: Any 3rd-party AWS S3-compatible appor platform should work with WasabiBackup &RecoveryContentDeliveryStorageGateway 200 apps listed at wasabi.com/interopArchivingAnalyticsIoTApp DevTools20

Wasabi Management ConsoleCommon look-and-feel with AWS Management Console Wasabi’s storage and identity access management console ismodeled on AWS S3 (to make it simpler for new users to adopt) Same concepts of storage buckets, access keys, users, policies etc. Demo video @ wasabi.com/helpWasabi UI samelook & feelas AWS S3mgmtconsole21

Cloud Strategies For Leveraging Wasabi (HPC and more )More info @ wasabi.com/solutionsHighPerformance& EdgeComputingOptimize systemperformance &costs with cloudstorageHybridStorageExtend on-prem orprivate cloudinvestments withaffordable publiccloud storageOn-Premto CloudMove all on-premstorage to thecloud andeliminatemaintenance feesMulti-CloudData LakeTape toCloudEliminate cloudlock-in & choosebest-of-breedproviders for priceand performanceEliminate tiers andstore everything inactive archives foryour data analyticsprojectsMigrate your tapearchive/backup toleverageaffordable publiccloud storage22

Use Cases For Leveraging WasabiMore info @ wasabi.com/solutionsBackup and RecoveryArchivingStore more to enable businesscontinuity across hybrid ormultiple cloudsStore more and evolve complexarchive tiers into a single simpleactive archiveContent DeliveryStore more to accelerate audioand video content and softwaredistributionSurveillanceStore more video and photos toimprove security and lawenforcementData AnalyticsStore more data to enable dataanalytics and better businessintelligenceInternet of ThingsStore more data coming frombillions of smart connectedsensors and devicesApplicationDevelopmentStore more to enable customapp development integrated withcompute and cloud partnersAI/MLStore more data to enableartificial intelligence and machinelearning to transform the future23

Thank YouFor more information, please visit wasabi.com24

Questions?Thank you for your attention!Joseph LombardoUNLV National Supercomputing Institutelombardo@nscee.eduJim DonovanWasabi Technologies, Inc.jdonovan@wasabi.comVictor WrightAltair Engineering, Inc.vwright@altair.com2019 Technology Exchange in New Orleans, December 2019Collaborations with the Cleveland Clinic Lou Ruvo Center for Brain Health and the Nevada Institute of Personalized Medicine are currently funded by twoInstitutional Development Awards (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health: #P20GM109025 &#P20GM121325. The content is solely the responsibility of the author(s) and does not necessarily represent the official views of the National Institutes of Health.

About the NSI Full-service supercomputing facility Mission for excellence in education and research in supercomputing and its applications Provides supercomputing training and services to academic and research institutions, government and private industry Facilitates high-technology economic diversification in Nevada by providing services not available in the private-