Transcription
Campaign StoragePeter Braam 2017-04Co-founder & CEO Campaign Storage
Contents Memory class storage & Campaign storage Object Storage Campaign Storage Search and Policy Management Data Movers & Servers Road Ahead4/1/17Campaign Storage LLC2
Campaign Storage Campaign Storage was invented at Los Alamos National Laboratory 2014- Peter Braam & Nathan Thompson founded Campaign Storage, LLC inMarch 2016 deliver products in this space Software Defined Storage – we will partner with integrators Other companies are addressing parts of Campaign Storage also3/16/17Campaign Storage LLC3
Storage Tiers &Campaign Storage3/16/17Campaign Storage LLC4
CPU or GPU packagesNVRAMe.g. XPOINT,PCM, urst Buffers –DDN IME, Cray Data WarpNode BW (GB/sec)1 TB/s100 GB/s20 GB/s5 GB/s350 MB/sCluster BW(TB/sec)1 PB/s100 TB/s5 TB/s100 GB/s10’s GB/sSoftwareLanguage levelLanguage level, NVM libsHDF5 & DAOSHDF5DAOSParallel FS &Campaign StorageArchive &CampaignKey featurestransparent computationtransparent computationultra-fast storage appsname spacescientific formatsFS style containerbulk data movement- many files- subtrees of MDBW Cost / (GB/s) 10 (CPU included!) 10 300 2K 30K 8 0.3 0.05 0.01Capacity Cost /GB 3/16/17Campaign Storage LLC5
Role of containersFundamentally unlikely:different tiers perform data movement at similar granularityContainers are a must-have3/16/17Campaign Storage LLC6
Tiers and NVRAM ConsiderationsTieringPersistenceRAM tiers are for computationDistinguishing NVM feature is that data stays ifpower is off.Flash storage is 5x faster with large IODisk similarly is very IO size sensitive:NVRAM will be the fastest storage deviceè migrate pointers, pagesè for most demanding storage applicationsèRetrieve & store containers (distributed?)èShow internal structure on faster sideèStream and serialize data to slower sideInternal program data formats not re-usableè computing format to namespace3/16/17NVRAM: what other benefits to computing?Current libraries – transactions, persistent heaps(not so novel – see Camelot & RVM from 1980’s)Campaign Storage LLC7
Example Container Functionality - lower tiers3/16/17Campaign Storage LLC8
Tiers & TransparencyRAMStorage- Demote infrequently used pointers- Promote frequently used pointersSame principle – transparency requiresaccessing data through a handleIf pointers are not first class objects- Promote upon access- Demote finding less used onesOne handle system with locationdatabase allows other objects to moveExpect distributed tiered KV storeLow level languages – HW or OS support3/16/17 Key value lookup Callbacks for invalidationCampaign Storage LLC9
Using Tiers3/16/17Campaign Storage LLC10
Data CenterData CenterComputeData CenterData CenterNodesComputeComputeNodesNodesIdentity and namespacemanagement with e.g. AD or LDAPContainers for global namespaceService Layer- SMB, NFS, otherData Movers- Parallel ingest & restoreEnterprise Data ServicesEnterprise Data Services- Groupsfile serversEnterpriseData ofServices- Groupsof file servers- AvailabilityAvailability- Performance- - Groupsof file serversPerformance- - Availability- Performance3/16/17Campaign StorageCampaign Storage LLCStorage layer- ZFS & object- Integrated search- Analytics Support- Data management- Massive, low 11
Future Exa-Scale Storage Architecture3/16/17Campaign Storage LLC12
Object Storage3/16/17Campaign Storage LLC13
Cloud object stores – pros & conspromassive scalabilityvery good storage managementwidely agreed S3 REST APIruns on cheapest hardwarecon3/16/17data lacks organizationAPI’s don’t allow distributed concurrent access or random writesperformance can be disappointingdifficult to re-use as a component of other storage systemsCampaign Storage LLC14
Too much choice? Caringo Swarm (formerly CAStor) Cleversafe dsNet Cloudian Data Direct Networks Web Object Scaler (WOS) EMC Atmos EMC Centera EMC Elastic Cloud Storage (ECS) HP StoreAll HGST Himalaya HGST Active Archive Hitachi Data Systems HCP NetApp StorageGrid Webscale Quantum Lattus Scality Ring SwiftStack SwiftWhat is needed offers:-Normal read/write IO per objectNon overlapping IO from multiple clients3 tier hierarchical redundancy (box, rack, data center)Transaction protocol to snapshot consistent stateTo mention a few . (others S3, CEPH, SNIA T10, Seagate A200, DDN WOZ .)3/16/17Campaign Storage LLC15
Campaign Storage3/16/17Campaign Storage LLC16
Campaign Storage - a new tierOld WorldNew WorldBurst BufferParallel File SystemHigh BW, high Decreasing capacitiesdecreasingemphasisArchiveParallel File SystemCampaign StoragenewArchiveCloud3/16/17Campaign Storage LLC17
Campaign StorageIt is It is not A file systemFocus: staging and archivingBuilt fromGeneral purpose file system Industry standard object stores Existing metadata storesLowest cost HWHigh capacity, ultra scalableNot highest BW or lowest latency 3/16/17 Wait these don’t exist actuallyUsing object stores has problems Limited set of data moverssupported10-100x higher than archives10x lower than PFSCampaign Storage LLC18
ImplementationOS with VFS and FuseMarFSObject Storage3/16/17Campaign Storage LLCMetadata FS19
Campaign StorageHPC Cluster ASimulation Cluster20PFBurst Buffer5 PB & 5 TB/sHDFSHPCD & Viz ClusterCampaign StorageCampaign StorageMover NodesCampaign StorageMetadata RepositoryCampaign StorageMover NodesCampaign StorageObject RepositoryHPCDCluster20PFBurst Buffer5 PB & 5 TB/s3/16/17Lustre FS 1 TB/sHPC Cluster BParallelStaging & ArchivingSearch & Data ManagementFile System Interfacecustomer infrastructureCampaign Storage LLCOptional other tools: Policy managers (e.g. Robinhood) Workflow managers (e.g. Irods)20
Campaign Storage Data Layout3/16/17Campaign Storage LLC21
Search & Policy Management3/16/17Campaign Storage LLC22
3/16/17Campaign Storage LLC23
Histograms for subtree searchEvery directory has histogram DB recording properties of its subtree: i.e. #files, #bytes in the subtree have a property? Limited granularity, limited relational algebra Store perhaps 100,000 properties in multiple histogramsExamples: Quota in subtree?What fileservers contain files?Geospatial information in file?(file type, size, access time) tuples Allows limited relational algebra User database for subtree – eliminates reliance on external identity managementNot a new idea. Can be added to ZFS & Lustre3/16/17Campaign Storage LLC24
Data Movers & Services3/16/17Campaign Storage LLC25
Data MoversData MovementToday LANL “parallel rsync” – pftool Lustre HSM mover Packing small files & striping big filesCandidates DMAPI HSM mover Gridftp Full POSIX interface3/16/17Metadata MovementToday Traditional metadata API Multiple namespacesComing Bulk integration of containers Accompanying metadataCampaign Storage LLC26
pftool internalsLoadBalancerScheduler3/16/17dirs queuereaddirstat queuestatcp/S/V queuecopy / syncvalidateCampaign Storage LLCDONEQUEUEREPORTER27
Features of DS3 archival data mover Object store moves batches of files New concept: file level I/O vectorization Includes server driven ordering Packing small files into one objectint copy file range fv(copy range *r, uint count, int flags)struct copy range {int source fd;int dest fd;off t source offset;off t dest offset;size t length;}3/16/17Campaign Storage LLC28
ServicesCampaign Storage always exports the MarFS file systemEnterprise services as further exported protocols:- SMB, NFS, HTTP- Data movement can be out of bandIntegration of namespaces, user databases, other plugins3/16/17Campaign Storage LLC29
Campaign Storage Use Cases3/16/17Campaign Storage LLC30
Workflows - HPCHot vs cold Campaign LocationsStaging & De-staging Select destination object stores Migration on campaign storage Schedule migration with pftoolHSM Copy metadata firstUse subtree search indexExecute policiesSpecialized data movers Leverage object bucket replication Leverage ZFS pool replication For transparent retrieval & attributesSingle project extraction Use ZFS namespace and objectbucket per project3/16/17Multi siteCloud Migrate pool and buckets to S3 Use Snowball?Campaign Storage LLC31
Workflows – Data CenterStaging & archive Schedule migration with pftoolService offload to Campaign Data available without stagingSingle project extraction Use ZFS namespace and objectbucket per projectHot vs cold locations Select destination object stores Migration within campaign storage Automatic movement when servicesneed the dataMulti site Leverage object bucket replication Leverage ZFS pool replicationCloud Migrate pool and buckets to S3 Use Snowball?3/16/17Campaign Storage LLC32
Road ForwardUnique opportunity to innovate data managementLANL and Campaign Storage created an “Industry Steering Group”Seek agreement on Data layout handling Attributes used in connection with long term storage Interfaces for workflows3/16/17Campaign Storage LLC33
Conclusions3/16/17Campaign Storage LLC34
ConclusionsHardware diversification è Software SpecializationExpect a rich high speed exa-scale I/O platform to use containersSimilar containers will organize enterprise tiers of storageCampaign Storage: bulk data store, archive & data movement3/16/17Campaign Storage LLC35
Thank you3/16/17Campaign Storage LLC36
Lustre FS 1 TB/s Campaign Storage Campaign Storage Mover Nodes Campaign Storage MetadataRepository Campaign Storage Mover Nodes Parallel Staging & Archiving Campaign Storage Object Repository Search & Data Management Campaign Storage 3/16/17 Campaign Storage LLC 20 File System Interface Optional other tools: Policy managers (e.g. Robinhood)
