Transcription
Using SPEC SFS with the SNIA EmeraldProgram for EPA Energy Star DataCenter Storage ProgramIndustry Stakeholders UpdateVernon Miller – IBMNick Principe – Dell EMCv3
AgendaBackground on SNIA Emerald/Energy Star for blockIntroduce NAS/File test addition; introduce SFS 2014Testbed configuration and measurement pointsTest procedureA look at some real data and the derivation of themetrics 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.2
Green PreambleIncreased regulatory and societal pressures to lowerenergy footprintsGrowing awareness of environmental impact of ITequipmentRising energy cost for power and cooling is a large partof the cost of ownershipData centers cannot readily add additional power orcooling 014Overview: Green Storage, Energy Star and SNIA Emerald Program 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.3
EPA ENERGY STAR for DataCenter StorageEnergy Star certification program for storage systems.EPA partnered with SNIA’s Green Storage Initiative(GSI) to develop technical specification s://www.energystar.gov/products/office equipment/data center storage 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.4
SNIA Green ActivitiesGreen Storage Initiative (GSI)Market green storage and manage the Emerald ProgramResearch, educate, leverage SNIA resources, provide directionGreen Technical Working Group (GTWG)Technical body of storage experts developing green storagespecifications, white papers, tutorials, technical guidanceDevelop the SNIA Emerald Power Efficiency MeasurementSpecification (currently 2.1.1) and “how to” User Guide for itEmerald ProgramPromote use of the SNIA Emerald Specification methodologyand test resultsHelp drive green storage decisions for both vendors stomerOverview: Green Storage, Energy Star and SNIA Emerald Program 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.5
Emerald Power EfficiencyMeasurement Specification –Market TaxonomyClassifies storage systems in terms of operational profile andsupported featuresSimplifies comparisons and regulatory efforts 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.6
Emerald Power EfficiencyMeasurement Specification –I/O and MeasurementStandard input voltages anddatacenter conditions requiredPrefill TestPUT Conditioning TestActive Test (Vdbench)Hot BandRandom WriteRandom ReadSequential WriteSequential ReadReady Idle TestCapacity Optimization Test(comgen) 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.PUT7
Emerald Power EfficiencyMeasurement Specification –Power MetricsPrimary metrics ratios of performance / wattRandom access (Transactional) of the data per unit of powerInput Output per Second per Watt (IOPS/W)Sequential access (Streaming) of the data per unit of powerMebibyte per Second per Watt (MiBPS/W)Storage Capacity per unit of powerGigabyte per Watt (GB/W)Secondary metricsCapacity Optimization verification, i.e. existence testSix techniques that reduce the number of storage devices to store the sameamount of data thus reducing the power required to store the verview SNIA Emerald Measurement 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.8
SNIA Emerald Test DataSubmissionSNIA Emerald ProgramRecord results in SNIAEmerald Test Data Report(TDR)Complete Test SubmissionOrder FormEPA ENERGY STAREPA recognized lab mustperform testsObtain certification from anEPA recognizedCertification Body 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.9
DisclaimerThe SNIA Emerald specification with file-access support,as represented in this presentation, is pre-release; thebenchmark framework, workloads, and results andreporting structure are still under internal SNIA reviewand may change before final release of SNIA EmeraldSpecification version 3.0. 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.10
NAS/File Addition toSpecificationVersion 3 of SNIA Emerald Power EfficiencyMeasurement SpecificationAddresses both block and file accessNew workloads and toolkit for file access testingSPEC SFS 2014New methodology for determining power metrics for file accessExpected rollout in 2017 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.11
SPECStandard Performance EvaluationCorporationThe Standard Performance Evaluation Corporation(SPEC) is a non-profit corporation formed to establish,maintain and endorse a standardized set of relevantbenchmarks that can be applied to the newest generation ofhigh-performance computers. SPEC develops benchmarksuites and also reviews and publishes submitted results frommember organizations and other benchmark licenseeswww.spec.orgSPEC and SPEC SFS are registered trademarks of theStandard Performance Evaluation Corporation. Additionalproduct and service names mentioned herein may be thetrademarks of their respective owners. 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.12
Why SPEC SFS 2014?SPEC SFS 2014 is a Storage Solution BenchmarkRealistic, Solution-based, Industry-standard workloadsDATABASE, SWBUILD, VDA, VDIWorkloads based on traces, like previous SFS 2008Modern scenarios based on standard solutionsAdvanced measurement – quality of serviceOps and latency don’t tell the whole story business metricsAbility to measure broad range of products and configurationsTraditional (HDD), Hybrid, All-FlashKey reasons SNIA Emerald is using SFS 2014Vendors likely already running SFS 2014 in-houseWorkloads already agreed upon by multiple vendorsRobust workload generator for file access 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.13
The SPEC SFS 2014WorkloadsDATABASEFor more details, see:- SDC 2014 presentation: SPEC SFS 2014: An Under-the-Hood Review- The SPEC SFS 2014 website http://www.spec.org/sfs2014Simulates OLTP database consolidationRandom/TransactionalSWBUILDSimulates large software project compilationMetadata intensive – lots of file operationsVDASimulates acquisition of streaming dataHeavy sequential writes with some random readsVDISimulates heavy steady-state VDI workloadRandom/Transactional – greater spread of IO sizes 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.14
File vs Block ConfigurationsBlock AccessESXi Server 1WinWinVMVMESXi Server 2WinWinVMVMESXi Server 10WinWinVMVM10 GbE Switch8 Gb FC SwitchEnvironmentalMeterNAS ServerProduct Under TestPowerMeterIO Load Driver SystemsBlock I/O generator:vdbenchCustom workloadsNo client side cachingMinimal impact topower metrics due toload generatorsStorageController 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.15
File vs Block ConfigurationsFile AccessESXi Server 1WinWinVMVMESXi Server 2WinWinVMVMESXi Server 10WinWinVMVMsFlowCollectionEnvironmentalMeter10 GbE Switch8 Gb FC SwitchNAS ServerPowerMeterProduct Under TestStorageController 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.IO Load Driver SystemsFile I/O generator: SPECSFS 20144 official workloadsClient side cachingLoad generators havegreater effect on overallperformance.sFlowIndustry standard forsampled performancestatisticsCollect data rate16
File vs Block Test SequencesBlock AccessPre-fill test, puts datain PUTPUT conditioningActive testReady idle testCapacity optimizationtest (if defined)File AccessCalibration of SFSworkloadsExecution of the 4SPEC SFS 2014workloads, insequenceReady idle testCapacity optimizationtest (if defined) 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.17
File vs Block Primary MetricsBlock AccessPower efficiency foractive phaseHot BandRandom ReadRandom WriteSequential ReadSequential WriteFile AccessPower efficiency for eachworkload (MiB/sec/Watt)VDADATABASEVDISWBUILDPower efficiency for ReadyIdle test phasePower efficiency forready idle test phase 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.18
File Access MeasurementPointsThe SPEC SFS 2014 metrics are only used to calibratethe appropriate load points for each workloadThe Emerald efficiency metric (MiB/s/W) is derived fromthe data collected by the sFlow collector in front of theProduct Under TestSPEC SFS 2014 measures at the application-levelSNIA Emerald measures at the product-level 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.19
Additional Hardware/SoftwareRequirements for File AccessIn addition to the environmental and power meterscommon for both Block and File Access Emerald testing,File Access testing requires:1.2.A network switch that supports sFloworHost sFlow installed on the load generatorsAn sFlow collector that can log for extended periods of time 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.20
File Access Test ProcedureThe four basic phases of file access testingCalibrationMeasurementData ReductionSNIA Emerald Metrics 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.21
File Access Test ProcedureCalibrationFor each SPEC SFS 2014 workload, find maximum PUTperformanceKnown from existing performance testing– Many vendors run SFS 2014 for regression analysisTest to determine as part of Emerald test process– Run several SFS 2014 runs, adjusting load points to probe for the maximum validload pointNot concerned with maximum efficiency at this pointNeed to know the whole efficiency curve as load scales up 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.22
File Access Test ProcedureCalibration ExampleOn a new test system, setup SWBUILD to run from 1 to30 load points, incrementing the load by 1 each stepAfter 20 load points, achieved ops/sec stopped scaling and SFS2014 was reporting INVALID RUNAt this point, the benchmark was manually terminated12,000Achieved Ops/secINVALID RUN LOAD too highMaximum valid load 415161718192021Concurrent BUILDS (SWBUILD) 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.2322232425262728
File Access Test ProcedureMeasurementUsing maximum valid load from calibration dataRun each workloadTen evenly-spaced load points up to the maximum valid load pointCollect environmental, power, and sFlow data for each runReady-idle testCollect environmental and power data while array is idle 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.24
File Access Test ProcedureMeasurement ExampleFrom the calibration phase, we know the system’smaximum valid load point is 20 BUILDSTherefore, for measurement, we ran from 2 through 20 BUILDS,incrementing by 2 BUILDS12,000Maximum valid load pointINVALID RUN LOAD too highAchieved 8910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28Concurrent BUILDS (SWBUILD) 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.25
File Access Test ProcedureData ReductionUsing collected sFlow and Power dataFor each workload, for the measurement phase only at eachload point, calculateAverage powerAverage network throughputUsing the average power and network throughputCalculate efficiency metric at each load point for each workload– MiB/s/WThis process is known as data reductionTime-based data from multiple sources is reduced to calculated metrics perload point, per-workloadAn open-source tool is available from SPEC to assist with thisprocessThis process does not describe additionalrequired stability/validity assessment 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.26
File Access Test ProcedureData Reduction ExampleThe raw power data log will include data for the wholeduration of benchmark executionOnly want data from the measurement phase 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.27
File Access Test ProcedureData Reduction ExampleUsing data reduction techniques, compute averagepower usage during measurement phaseFor each load point1200Average Watts1000800600400200012345678910SWBUILD Iteration Number The same process is used for the sFlow data Average network throughput during measurementphase 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.28
File Access Test ProcedureSNIA Emerald MetricsFind the “sweet-spot” for all four workloadsFor each workload, find the highest efficiency metric, using datafrom the data reduction stepUsually, but not necessarily, the highest valid load point 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.29
File Access Test ProcedureSNIA Emerald MetricsExampleExample data from another systemIn a real Emerald run, there would be ten load points for eachworkload and ready-idle dataEFFICIENCY METRICSLoad PointDATABASE 30.0260.0370.0480.04910DATABASE SWBUILDVDAMaximum Efficiency .05Metrics reported in Test Data Report 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.30This systemdemonstrateshighest efficiencyat the highestvalid load point
Key TakeawaysThe SNIA Emerald program is adding support for fileaccess storage systemsSPEC SFS 2014 and its workloads are used to evaluatefile access storage systemsThe Product Under Test for SNIA Emerald file accesstesting only includes the storage arrayEfficiency metrics are derived from:Network traffic to and from the storage array– Not SPEC SFS 2014 metricsPower consumption by the storage array 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.31
Additional Resources SPEC SFS 2014 Details and Tools SPEC SFS 2014 Website (User’s Guide and Run Rules) http://spec.org/sfs2014/tag2014.py Data Reduction Tool http://spec.org/sfs2014/support/tools/tag2014.tgz SNIA Storage Developer Conference Presentations: SDC 2014: SPEC SFS 2014 An Under-the-Hood Review SDC 2015: Application-Level Benchmarking with SPEC SFS 2014 https://www.youtube.com/watch?v S9RJ rKc3WIhttps://www.youtube.com/watch?v 4wfeM1q0zHASDC 2016: Using SPEC SFS with the SNIA Emerald Program forEPA Energy Star Data Center Storage Program https://www.youtube.com/watch?v 7gDgcDYatvM 2017 Storage Networking Industry Association and Storage Performance Evaluation Corporation.All Rights Reserved.32
the data collected by the sFlow collector in front of the Product Under Test SPEC SFS 2014 measures at the application-level . An open-source tool is available from SPEC to assist with this process 26 This process does not describe additional required stability/validity assessment
