Storage On AWS - Imdaysfl

Transcription

Storage on AWSAshley MillerSr. Solutions Architect1 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Agenda 2IntroductionStorage PrimerBlock StorageShared File SystemsObject StoreOn-Premises Storage Integration 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Introduction: Why choose AWS for storageCompellingEconomicsPay as you go3Easy to UseReduce riskSelf serviceadministrationDurable and SecureNo risky capacityplanningSDKs for simpleintegrationNo need toprovisionfor redundancyor overheadNo Commitment 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.Avoid risks ofphysical mediahandlingSpeed,Agility, ScaleReduce time tomarketFocus on yourbusiness, not yourinfrastructureGlobal Scale

0Storage Primer4 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Block vs File vs ObjectBlock StorageRaw StorageData organized as an array of unrelated blocksHost File System places data on diskEx: Hard Disks, Storage Area Network (SAN) Storage ArraysFile StorageUnrelated data blocks managed by a file (serving) systemNative file system places data on diskEx: Network Attached Storage (NAS) Appliances, Windows File ServersObject StorageStores Virtual containers that encapsulate the data, data attributes, metadata and Object IDsAPI Access to dataMetadata Driven, Policy-based, etc.Ex: Ceph, OpenStack Swift5 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Storage - CharacteristicsSome of the ways we look at storageDurabilityMeasure ofexpected dataloss6AvailabilityMeasure ofexpecteddowntimeSecuritySecuritymeasures forat-rest and intransit data 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.CostAmount perstorage unit,e.g. / GBScalabilityPerformanceUpwardflexibility,storage size,number ity tointeract via APIor with otherservices

Understanding DurabilityStandard IA7GlacierTwo copies on one siteCopies on two sitesdesigned fordesigned urability 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.copies in three AZdesigned for

Availability vs Durability%Availability99.9995 minutes 15 seconds 1 in 100,00099.999999.9999999.9999999998Durability 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.31 seconds 1 in 1,000,0003 seconds 1 in 10,000,000300 uSeconds 1 in 100,000,000,000

More choice for more applicationsBlock storageFile storageGeneral Purpose SSDEFS StandardProvisioned IOPS SSDThroughput-Optimized HDDAmazon EC2Cold HDDAmazon AWS StorageGateway FamilyEFSFSx for WindowsFSx for LustreAmazon FSxfor WindowsFile ServerAmazon EBSBackupEFS Infrequent AccessAmazon FSxfor LustreObject storageS3 StandardS3 Standard-IAS3 One Zone-IAS3 Intelligent-TieringAmazon S3S3 GlacierS3 Glacier Deep Archive9 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

1Block Storage10 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

What is Amazon EBS?EC2instanceEBSvolume11 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Block storage as a service Create, attach volumes through an API Service accessed over the network

AWS EBS FeaturesDurableSecureScalableDesigned for 99.999reliabilityIdentity andAccess PoliciesCapacity whenyou need itRedundant storageacross multiple deviceswithin an AZEncryptionEasily scale upand downPerformanceBackupLow-latency SSDPoint-in-time SnapshotsConsistent I/O PerformanceCopy snapshots across AZand RegionsStripe multiple volumes forhigher I/O performance12 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon EBSNetwork attached block device Independent data lifecycleMultiple volumes per EC2 instanceOnly one EC2 instance at a time per volumeCan be detached from an instance and attached to a different oneRaw block devices Unformatted block devicesIdeal for databases, filesystemsMultiple Drive Types 13SSD (gp2 & io1) and HDD (st1 & sc1) 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon EBS volume typesSSDgp2General PurposeSSD14HDDio1Provisioned IOPSSSD 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.st1Throughput Optimized HDDsc1Cold HDD

Amazon EBS use casesSSDHDDRelational DatabasesNoSQL DatabasesBig Data , AnalyticsFile / MediaMySQL, SQL Server,PostgreSQL, SAP,OracleCassandra, MongoDB,CouchDBKafka, Splunk,Hadoop, DataWarehousingCIFS/NFS,Transcoding,Encoding, Rendering15 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon EBS volume types: General Purpose SSDBaseline: 100 to 16,000 IOPS; 3 IOPS per GiBBurst: 3,000 IOPS (for volumes up to 1,000 GiB)Throughput: Up to 250 MiB/sgp2General Purpose SSDLatency: Single-digit msCapacity: 1 GiB to 16 TiBGreat for boot volumes, low-latency applications, and bursty databases16 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon EBS volume types: Provisioned IOPSBaseline: 100–64,000 IOPSThroughput: Up to 1,000 MiB/sLatency: Single-digit msio1Provisioned IOPS17Capacity: 4 GiB to 16 TiBIdeal for critical applications and databases with sustained IOPS 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon EBS volume types: ThroughputProvisionedBaseline: 40 MiB/s per TiB up to 500 MiB/sBurst: 250 MiB/s per TiB up to 500 MiB/sCapacity: 500 GiB to 16 TiBIdeal for large-block, high-throughput sequential workloadsst1Throughput Optimized HDD18 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon EBS volume types: Cold HDDBaseline: 12 MiB/s per TiB up to 192 MiB/sBurst: 80 MiB/s per TiB up to 250 MiB/sCapacity: 500 GiB to 16 TiBIdeal for sequential throughput workloads, such as logging and backupsc1Cold HDD19 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

EBS Snapshots Point-in-time snapshots of volume blocks Stored in Amazon S3 and accessed via EBS APIs Key Features:21 Immediate access to EBS volume data Resizing EBS volumes Sharing EBS Snapshots Copying EBS Snapshots across AWS regions 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

EBS Snapshots – Delta Block ChangesAWS CloudEC2 Availability ZoneEBSEBSEBSEBSEBSAmazon S3Create SnapshotEBS SnapshotEBSClone FromSnapshotEC2EC2EC2Internet22 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.EBS SnapshotEBS Snapshot

How Do EBS Snapshots Work?XSnapshot 1TimeSnapshot 2Snapshot 3S3EBS VolumeBlock 11ChunkBlock 22ChunkBlock 33ChunkBlock 44Chunk23 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

What is Amazon EC2 instance store?EC2 instancesInstance StoreorPhysical Host24 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Local to instance Non-persistent data store Available on certain EC2families– Ex: d2, i3, r5d, m5d Data is not replicated (bydefault) No snapshot support SSD or HDD

2Shared file system25 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Elastic File System (EFS) 26Fully managed file system for EC2 instancesProvides standard file system semanticsWorks with standard operating system APIsSharable across thousands of instancesElastically grows to petabyte scaleDelivers performance for a wide variety of workloadsHighly available and durableNFS v4–basedAccessible from on-premise servers 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

1Amazon EFS is SimpleFully managed-No hardware, network, file layerCreate a scalable file system in seconds!Seamless integration with existing tools and apps-NFS v4.1—widespread, openStandard file system access semanticsWorks with standard OS file system APIsSimple pricing simple forecasting27 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

2Amazon EFS is Elastic File systems grow and shrinkautomatically as you add and removefiles No need to provision storage capacity orperformance You pay only for the storage space youuse, with no minimum fee28 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

3Amazon EFS is Scalable File systems can grow to petabyte scale Throughput and IOPS scaleautomatically as file systems grow Consistent low latencies regardless offile system size Support for thousands of concurrentNFS connections29 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Highly Durable and Highly Available Designed to sustain AZ offlineconditions Resources aggregated across multipleAZ’s Superior to traditional NAS availabilitymodels Appropriate for Production / Tier 0applications30 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Example use casesBig Data AnalyticsMedia Workflow ProcessingWeb ServingContent ManagementHome Directories31 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

EFS – MountingEFSEC2EC2EC2EC2EC2EFS DNS n.amazonaws.comMount on machinesudo mount -t nfs4 mount-target-DNS:/ /efs-mount-point32 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.EC2

EFS Lifecycle Management EFS offers both Standard and Infrequent Access (IA)storage classes With Lifecycle Management enabled, EFS automaticallymoves files not accessed for 30 days from the Standardstorage class to the EFS IA storage class34 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

FSx for WindowsFully managed Windowsfile systems built on WindowsServer35 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.Integrated withAWS

Native Windows compatibility and featuresNTFSSMBADNative SMBIntegrates withMicrosoft ADand supportsWindows ACLs2.0 to 3.1.1Native WindowscompatibilityWindows Server36 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.DFSNamespacesandDFS Replication

3Object Stores38 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon S3 (Simple Storage Service) 39Web accessible object store (through API or HTTPS)Highly durable (99.999999999% design)Limitlessly scalableMultiple Tiers to match your workloadData Lifecycle RulesStatic Website HostingSecurity, Compliance, and Audit capabilitiesStandard Storage Pricing (us-east-1) - 0.023 per GB 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Your choice of object storage classesS3 StandardS3 Intelligent-TieringS3 Standard-IA40S3 GlacierAccess FrequencyFrequent Active, frequentlyaccessed data Milliseconds access 3 AZ 0.0210/GBS3 One Zone-IA Data with changingaccess patterns Milliseconds access 3 AZ 0.0210 to 0.0125/GB Monitoring fee perObj. Min storage duration Infrequentlyaccessed data Milliseconds access 3 AZ 0.0125/GB Retrieval fee per GB Min storage duration Min object size 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Re-creatable, lessaccessed data Milliseconds access 1 AZ 0.0100/GBS3 GlacierDeep ArchiveInfrequent Archive data Select minutes orhours 3 AZ 0.0040/GBRetrieval fee per GBMin storage duration Retrieval fee per GB Min storage duration Min object size Min object size Archive data Select 12 or 48hours 3 AZ 0.00099/GB Retrieval fee per GB Min storage duration Min object size

S3 Management Features41 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

S3 Access Management & Security Deep integration with AWS Identity and Access Management (IAM)Access Control Lists (ACLs) & S3 bucket policiesQuery String AuthenticationAudit LogsS3 supports both server-side & client-side encryption 42S3 Inventory to check the encryption status of S3 objectsS3 Block Public Access to ensure S3 buckets and objects do not havepublic accessAmazon Macie to discover, classify, and protect sensitive data storedin Amazon S3 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

445On-Premises StorageIntegration 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Many Options for Data Transfer46AWSDirect ConnectAmazonKinesisFirehoseAmazonKinesisData StreamsAmazon KinesisVideo StreamsAmazon MigrationServiceAWSSnowballAWSSnowball EdgeAWSSnowmobileAWSDataSyncAWSTransferfor SFTP 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Storage Gateway hybrid storage solutionsEnables using standard storage protocols to access AWS storage servicesFilesVolumesTapes47 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.Amazon S3Amazon GlacierAmazon EBSsnapshots

Storage Gateway – Files, volumes, and tapesFile gateway NFS (v3 and v4.1) interfaceOn-premises file storage backed by Amazon S3 objectsVolume gateway iSCSI block interfaceOn-premises block storage backed by S3 with EBS snapshotsTape gateway iSCSI virtual tape library interfaceVirtual tape storage in Amazon S3 and Glacier with VTL management48 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Storage Gateway – Common capabilitiesStandard storage protocols integrate with on-premises applicationsLocal caching for low-latency access to frequently used dataEfficient data transfer with buffering and bandwidth managementNative data storage in AWSStateless virtual appliance for resiliencyIntegrated with AWS management and security49 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Tape gatewayiSCSIBackupServerTAPEDRIVECustomer PremisesMEDIACHANGERVirtual tape storage in Amazon S3 and Glacier with VTL managementHTTPSTapeGatewayVirtualTapesstored inAmazon S3Archived Tapesstored inAmazon GlacierVirtual tape storage in S3 and Glacier accessed via tape gatewayData compressed in-transit and at-restUnlimited virtual tape storage, with up to 1PB of tapes active in librarySupports leading backup applications:50 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

File gatewayOn-premises file storage maintained as objects in Amazon S3Customer PremisesNFSv3 / v4.1ApplicationServer 51HTTPSFileGatewayS3StandardS3GlacierStandard InfrequentAccessData stored and retrieved from your S3 bucketsOne-to-one mapping from files-to-objectsFile metadata stored in object metadataBucket access managed by IAM role you own and manageUse S3 Lifecycle Policies, versioning, or CRR to manage data 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Volume gatewayOn-premises volume storage backed by Amazon S3 with EBS snapshotsCustomer orage Gatewaybucket inAmazon S3Block storage in S3 accessed via the volume gatewayData compressed in-transit and at-restBackup on-premises volumes to EBS snapshotsCreate on-premises volumes from EBS snapshotsUp to 1PB of total volume storage per gateway52 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.Amazon EBSsnapshots

Hybrid storage use cases with Storage GatewayEnabling cloud workloadsMove data to AWS storage for Big Data, cloud bursting, ormigrationBackup, archive, and disaster recoveryCost effective storage in AWS with local or cloud restoreTiered cloud storageEasily add AWS storage to your on-premises environment53 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

NEWAWS DataSyncOnline transfer service that simplifies, automates, andaccelerates moving data between on-premises storage and AWSAWSFast datatransferEasy to useSecure andreliableCloudintegratedCosteffectiveCombines the speed and reliability of network accelerationsoftware with the cost-effectiveness of open source tools54 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

NEWHow AWS DataSync worksOn-PremiseRegionAWS storage resourcesNFSSharedfile systemsDeploy onpremises agentfor fast access tolocal storage55Amazon S3bucketTLSAWS DataSyncagentData transferover the WANusing purposebuilt protocol 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.AWSDataSyncAmazon EFSfile systemService in AWSwrites or readsdata from AWSstorage servicesManaged fromAWS Console orCommand LineInterface (CLI)

Amazon Snowball & Snowball Edge 56Terabyte scale data transportUses secure appliancesFaster than Internet for significant data setsImport into S3HIPAA Compliant 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

What is Snowball?Terabyte scale data transportRuggedizedcase“8.5G Impact”E-ink shippinglabel5780 TB 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.10G networkRain & dustresistantTamper-resistantcase & electronicsAll data encryptedend-to-end

How fast is Snowball? Less than 1 day to transfer 250TB via 5x10G connections with 5 Snowballs,less than 1 week including shippingNumber of days to transfer 250TB via the Internet at typical utilizationsInternet Connection 58 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.300Mbps316158105150Mbps632316211

Amazon Snowmobilehttps://www.youtube.com/watch?v 8vQmTZTq7nw59 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Any Questions?60 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Block vs File vs Object Block Storage Raw Storage Data organized as an array of unrelated blocks Host File System places data on disk Ex: Hard Disks, Storage Area Network (SAN) Storage Arrays File Storage Unrelated data blocks managed by a file (serving) system Native file system places data on disk