Transcription
NVMe Modern SAN Primerby M ike Kieran, NetApp Technical Marketing EngineerMichael Peppers, NetApp Technical Marketing EngineerMAY 2018
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness OutcomesIntroductionEnterprise data storage is being transformed by newtechnology—again.This time, the emergent technology is NVMe—short fornonvolatile memory express—a new storage access andtransport protocol that delivers the fastest response timesyet for business-critical enterprise applications.Like the arrival of flash storage, NVMe is about to provide amajor speed boost for the enterprise data storage systems.But this time, the transformative effect could be greaterstill, because NVMe isn’t just a storage spec; the broaderNVMe over Fabrics (NVMe-oF) protocol re-architects theentire data path, from server to storage system.In this primer, we’ll examine how NVMe technology canenable organizations to build storage area networks (SANs)with the performance, reliability, availability, and scalabilitythey need to run their core enterprise applications.You’ll learn how NVMe technology: Defines a new high-speed data transfer protocolfor solid-state drives (SSDs) to communicate withenterprise or client systems Provides higher IOPS and reduced latency fromthe host software stack, through the fabric to thestorage array Takes advantage of the ever-increasing speedof the Fibre Channel transport and othernetworking technologies Can provide your organization with worldclass performance for SANs runningbusiness-critical applicationsNVMe Modern SAN Primer2
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness OutcomesNew Business ImperativesIt’s hard to find a large organization today that isn’t goingthrough a “digital transformation”—examining every aspectof their business to see where data can be exploited toaccelerate time to market, reduce error rates, and improvethe customer experience.In some cases, organizations are accomplishing thisby updating existing applications, such as customerrelationship management or enterprise resource planning.In others, they’re deploying entirely new workloads,including mobile, video, automated bots, real-time analytics,artificial intelligence, and the Internet of Things.Across all use cases, the volume of data that’s beingcreated, aggregated, shared, and stored is increasing ata rapid rate. More important, the value of that data isgrowing exponentially over time. Today, the world’s mostvaluable resource is no longer oil, but data.That’s why NVMe arrives in the data center at theperfect time: It empowers IT organizations to provide their internalcustomers with real-time response for crucialbusiness workloads. It enables organizations to reduce time to value byidentifying and exploiting opportunities that materiallyimpact the business.Today, IT leaders are responsible for upgrading theirexisting infrastructure, delivering everything as a service,and spinning up new workloads, often with little in theway of new budget. That’s why NVMe technology is acrucial technology in the modern data center. It reducesresponse times for critical applications by reducinglatency—the time it takes to access application data on aSAN—from milliseconds to microseconds.NVMe Modern SAN Primer3
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness OutcomesReducing latency with NVMe technology can also addbusiness value for many use cases, including: Real-time analytics. With billions of debit and creditcard transactions being processed every day, retailersand credit card companies have a huge incentive toprevent fraud before it happens. NVMe SANs providethe technology for the near-instantaneous responsenecessary to avoid huge potential losses, in-storeand online. Software agents. Intelligent software bots areincreasingly providing high-quality customerengagement, technical support, and other interactiveservices, often faster than a human could. In the future,NVMe technology will enable systems to respond soquickly and accurately that users won’t know whetherthey’re talking with a human or a bot.“NVMe over Fibre Channeldelivered 58% higher IOPSand 34% lower latencythan SCSI FCP.”— Demartek analyst report, Performance Benefits of NVMeover Fibre Channel – A New, Parallel, Efficient Protocol,May 2018 Internet of Things. NVMe-based storage systems canrapidly ingest massive data sets, and they supporthuge numbers of simultaneous client connections. Thatmakes them ideal for use both as edge data collectionpoints, and as centralized storage pools that can hostthe resultant data lakes created by aggregating all thesensor data.NVMe Modern SAN Primer4
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness OutcomesNVMe and Solid-State DrivesWhen flash technology arrived in the data center, first ashybrid disk-flash arrays and then as all-flash arrays, it had ahuge impact on the performance of enterprise applications.Flash storage has also changed performance expectationsfor end users. Users with flash-enabled PCs have becomeaccustomed to ever-faster response times and expect thesame experience for all of their business applications, evenas those applications become more complex and extendfrom the data center to the cloud.However, the speed gains possible from SSDs areapproaching a limit because of bottlenecks in other partsof the data pathway. Both of today’s main networkingprotocols—Fibre Channel and Ethernet—use the SCSIcommand set for the storage protocol. SCSI was developedfor mechanical media almost 40 years ago, and although itcan handle the data flow to and from spinning disks, it canno longer keep up with new flash storage media.SATA or SAS SSDPCIe SSDDATASoftware RAIDSATA DriverSATA ControllerDATA3rd Party FlashControllerFlash Translation(Flash-aware RAID)Flash MediaFlash MediaHIGH LATENCYApplication “far”from dataLOW LATENCYMemory-like accessto dataFigure 1: The NVMe software stack for SATA or SAS versus PCIe.That’s because SCSI puts I/O requests into a single queue,containing a maximum of 256 commands. As I/O requestsarrive from an application, they have to wait in line whileNVMe Modern SAN Primer5
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness Outcomesother requests are completed. In the era of spinning disks,these data access delays didn’t really matter, becausethe drive’s read-write heads were busy anyway, spinningaround seeking tiny magnetic blocks on rapidly rotatingplatters. Flash and other solid-state media have no movingparts, and thus very low innate latency, so delays in thedata path have become the new bottleneck, slowingdata access.Over the past decade, this situation has improved withthe emergence of the Peripheral Component InterconnectExpress (PCIe) bus, which is much faster than SATA andother legacy protocols. For instance, the PCIe bus supportsup to 65,535 queues, each with a queue depth of 65,535commands, and can connect storage directly to the CPU,providing memory-like access. Plus, PCIe requires a muchsmaller software stack than legacy protocols (see Figure 1).Today, PCIe-based devices are used in all kinds ofcomputing environments, which could pose a problem forSSDs, because every vendor’s hardware requires the use ofproprietary drivers. To eliminate compatibility issues, all thekey industry players have agreed on a new modern dataaccess standard for flash and other forms of solid-statememory running on a PCIe bus: the NVMe specification.NVMe is actually two distinct but interwoven things: A protocol and instruction set optimized for solid-statestorage devices A forward-looking set of open-source architecturalstandards for nonvolatile memory componentsand systemsThe NVMe specification was designed from scratch to takeadvantage of nonvolatile memory in all kinds of computingenvironments, from mobile phones to webscale serviceproviders. And thanks to its increased throughput andlower latency, NVMe is rapidly expanding its presence in thedata center (see Figure 2).Among the companies that have made massiveinvestments in NVMe are NetApp, Broadcom, Intel,Samsung, Micron, Seagate, Cisco, Western Digital,Microsoft, and Toshiba. NetApp now uses NVMe storageacross most of its hybrid-flash and all-flash storage systemsto significantly improve performance.In summary, NVMe is driving an architectural shift thatmakes communication with storage systems massivelyparallel. The result is greater bandwidth and lower latencyconnectivity between servers and storage devices.NVMe Modern SAN Primer6
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness OutcomesAnd it’s intended to be future-proof: NVMe supportstoday’s NAND flash and 3D XPoint technologies andwill work with currently in-development and as-yetundiscovered persistent memory technologies asthey emerge. Interrupt handling Internal locking needed to serialize I/O requests Command streamlining Reduced context switchesIO Controller LatencySoftware Latency 80 µsWait, There’s More!Many other factors enable NVMe to provide unparalleledperformance in the data center, including:Drive Latency 10 ms 80 µs 25 µs 25 µs 10 µs 10 µs 5 µsFlashNVMeDiskFigure 2: The impact of NVMe on media access latency. Lockless design Polling modeNVMe Modern SAN Primer7
IntroductionNVMe over FabricsNVME Host SoftwareThese are crucial enhancements, as networks themselvescontinue to get faster and faster. For example, FibreChannel has accelerated, first to Gen 5 at 16Gbps (gigabitsper second) and now to Gen 6 (32Gbps), with Gen 7 onthe horizon. Meanwhile, Ethernet has likewise increased tospeeds of 100Gbps and beyond.Next Gen FabricsBusiness OutcomesiWARPDatabases, SANs,and NVMeThe NVMe specification is continually evolving; withthe arrival of NVMe over Fabrics (NVMe-oF), NVMe’sperformance and latency benefits now extend acrossnetwork fabrics such as Ethernet, Fibre Channel, andInfiniBand. NVMe-oF adds support for a front-end storageinterface, greater distances within a data center over whichNVMe devices and subsystems can communicate, and theability to scale out to large numbers of NVMe devices.RoCENVMe over FabricsHost Side Transport AbstractionInfiniBandNVMe andSolid-State DrivesSo far, we’ve focused on how the NVMe protocol speeds updata storage. But NVMe is also a data transfer protocol—and a key component in the latest generation of high-speeddata fabrics.Fibre ChannelNew BusinessImperativesController Side Transport AbstractionNVMe SSDsFigure 3: NVMe-oF supports multiple networking protocols.NVMe Modern SAN Primer8
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness OutcomesRemoving the Interface BottleneckNVMe-oF is a relatively new technology, but its underlyingcomponents are well established in enterprise datacenters. One important factor is that it’s agnostic about theunderlying transport mechanism. Although many enterpriseIT shops are focused on using Fibre Channel (NVMe/FC), the NVMe-oF standard also supports remote directmemory access (RDMA) transport mechanisms, such asInfiniBand, RoCE, and iWARP (Figure 3).Transitioning to NVMe/FCFor many IT architects making the move to NVMe-oF, FibreChannel remains the preferred networking technologybecause of its performance and reliability, plus its supportfor fabric-based zoning and name services. Better still,Fibre Channel is capable of transporting multiple higherlevel protocols at the same time, such as NVMe/FC and FCP(SCSI on Fibre Channel).For some organizations, the business advantages justifyan end-to-end NVMe system, with multiple NVMe storagedevices using NVMe-oF with either a Fibre Channel orRDMA interface. This would be blazingly fast, in terms ofboth IOPS and latency.NVMe’s multiprotocol support simplifies the buyingdecisions you’ll face in making such an important migration.The key is to take advantage of your organization’s existinginfrastructure. For many organizations, the transitionto NVMe-oF will take a couple of years, and that’s nota problem.Instead of replacing one technology with another, a bettersolution is a dual-protocol Fibre Channel fabric, runningconcurrent FCP and NVMe/FC traffic. This provides a stableand cost-effective upgrade path, removing uncertainty andrisk during a crucial technology transition.NVMe/FC Solution StackWhile NVMe is a disruptive technology, you can introduce itnondisruptively into your organization.As shown in Figure 5, an end-to-end NVMe/FC solutionstack has three main components–the server, the storagesystem, and the fabric infrastructure that connectsthem. Although it’s possible to implement this type ofNVMe Modern SAN Primer9
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness OutcomesSAN infrastructure with 16Gb Fibre Channel switchesand directors, for best results an end-to-end 32Gb FibreChannel infrastructure is recommended.The complete NVMe-oF data path required to achievemaximum performance improvements is available today inNetApp solutions incorporating state-of-the-art networkinginfrastructure from technology partners such as Brocadeand Broadcom. Plus, some customers with recent storagesystems and Fibre Channel hardware bus adapters will beable to upgrade seamlessly, just by updating drivers andother software.A sample configuration listing the server hardware andsoftware, fabric infrastructure, and storage systems isshown in Figure 6.ServerHBA: SCSI-on-FC /NVMe/FCDual protocol array(SCSI-on-FC, NVMe/FC)SCSI-on-FC arrayNVMe/FC arrayExisting enterprisestorage infrastructureFigure 4: Dual-protocol infrastructure eases the transition from Ethernetto NVMe-oF.NetApp and Brocade have collaborated to create a concisee-book, NVMe over Fibre Channel for Dummies, which youcan download free of charge.NVMe Modern SAN Primer10
IntroductionNew BusinessImperativesNVMeSSDsNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness OutcomesNVMeSSDsBandwidth Bottleneck01001010Traditional SAN01001010Host Server01001010Modern SAN withNVMe-oFHost ServerFigure 5: The complete NVMe/FC solution provides a low-latency data path from the server right through to the storage system.Databases, SANs, and NVMeThe increasing popularity of NVMe is being shaped byworkloads that are crucial to organizations today. Theseworkloads range from long-established productivitystandards to cutting-edge new apps. SANs are built on top of the fastest protocols available(that is, Fibre Channel and iSCSI).Many of these business-critical workloads run either exclusively or preferentially on SAN systems: SANs are block-level systems that attach to thehost like an external hard drive, a key architecturaladvantage in many environments. SANs are optimal for many database workloads, whichare crucial business requirements.NVMe Modern SAN Primer11
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesIn fact, most business-critical apps are built on top ofdatabases, especially Oracle Database, Microsoft SQLServer, and SAP HANA.SANs are ideal for database workloads because they’redesigned to exceed an enterprise’s most rigorous keyperformance indicators,which typically center around: PerformanceNVMe over FabricsDatabases, SANs,and NVMeBusiness Outcomes AvailabilityONTAP 9.4 StackSourceServer OSSLES 12 SP3SUSE LinuxServer side HBAEmulex LPe32004 32G FC HBABroadcomServer side FC/NVMedriverBroadcom latest driversBroadcomSUSE Linux Enterprise 12 Inbox driverSLESGen5: Brocade 6505/6510/6520/8500BroadcomGen6: Brocade G610/G620/G630/X6BroadcomCisco MDS 9132TCiscoFirmware 8.1.1–8.2.x (G610/G620)BroadcomFirmware 8.2.x (G630)BroadcomBroadcom firmware downloadsBroadcomFirmware NX-OS 8.2(1), 8.2.(2)CiscoBrocade Network Advisor (BNA) 14.4.0BroadcomControllersA300, A700, A700s, A800NetAppFC/NVMe ProtocollicenseONTAP 9.4NetAppServerFabric InfrastructureSwitch Ease of use Future-proofSwitch firmware level Return on investmentEnterprise Database WorkloadsTo optimize performance, database administratorsconstantly monitor their systems to make sure eachhardware component is being fully utilized. This is especiallyimportant with database applications, where server sprawlor I/O roadblocks can significantly degrade response time.For example, a single database server connected to a SATAbased storage system inevitably runs into performanceSwitch ManagementSoftwareStorage and ONTAPFigure 6: Complete NVMe/FC solution stack that you can implement today.NVMe Modern SAN Primer12
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness Outcomesissues as I/O requests must wait to be processed. To solvethis problem, IT architects sometimes spin up anotherserver and split the workloads between them, or dedicateone server to user interaction and another to back-endprocessing. But this is an expensive alternative—a singleenterprise database license can cost tens of thousands ofdollars per year, not to mention the operational expenses.By the time you add in the costs of power and cooling,NVMe clearly lets you get more done, with less hardware, atlower cost.In-Memory DatabasesNVMe is ideal for SAP HANA, Apache Spark, and otherin-memory database applications that rely on mainmemory for data storage. These workloads operate withhuge datasets that can be larger than the combinedcluster memory, which puts enormous stress on thestorage system.In SATA environments, even with RAID protocols, asignificant risk exists that a hardware failure will result ina node being temporarily degraded, or even worse, outof service. With NVMe, however, the bus is fast enoughto keep up with the CPU, even with today’s incredibly fastmulticore processors.Business OutcomesTechnology research and development continue ata relentless pace. So as NVMe and other disruptivetechnology innovations come to market, one of the bigchallenges facing IT leaders is how to make well-groundedinvestments in the future.For organizations that are on the path of digitaltransformation, the key priorities include: Using flash and the hybrid cloud to migrate to anemergent modern data center architecture Exploiting the strategic value of data, throughadvanced analytics and artificial intelligence Future-proofing core workloads by running them on ahigh-speed modern SAN architectureNVMe Modern SAN Primer13
IntroductionNew BusinessImperativesNVMe andSolid-State DrivesNVMe over FabricsDatabases, SANs,and NVMeBusiness OutcomesToday, NetApp has the technology, products, andexpertise to provide enterprises around the globewith an unparalleled SAN value proposition. In part,it’s the product—blazing fast storage arrays and datafabrics that provide the nonstop availability thatorganizations need. It’s also the ecosystem—systems integrators and value-added resellers who buildcomplete solutions around the products, configuringthem to their customers’ industry-specific needs.configurations for core enterprise workloads,and the unique NetApp Active IQ automatedsupport system.Known for decades as the dominant brand in file-baseddata storage, NetApp has taken the lead over the pastfew years in SAN technology as well, with sales ofAFF SAN systems more than doubling year over year.In fact, most NetApp AFF customers are implementing SANs. A recent report from market research firmIDC confirms that NetApp is now the fastest growing among the top five SAN vendors, with significantmomentum moving forward into the NVMe era.Beyond that, it’s the platform that customers love:NetApp ONTAP provides world-class data management tools for the modern data center, customLEARN -area-network.aspx*Source: IDC, WW Quarterly Enterprise Storage Systems Tracker – 2017 Q4, Mar 2018. 2018 NetApp, Inc. All Rights Reserved. NETAPP, the NETAPP logo, and the marks listed at netapp.com/TM are trademarks of NetApp, Inc.Other company and product names may be trademarks of their respective owners. May 2018
NetApp now uses NVMe storage across most of its hybrid-flash and all-flash storage systems to significantly improve performance. In summary, NVMe is driving an architectural shift that makes communication with storage systems massively parallel. The result is greater bandwidth and lower latency
