Transcription
AWS Online Tech TalkModernize Your Applicationswith Containers, Serverless, andAmazon Elastic File System(Amazon EFS)Will OchandarenaPrincipal Product Manager, Amazon EFS 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda What is app modernization, and how does Amazon EFS help? Setting up EFS for modern apps Connecting your application to Amazon EFS Amazon Elastic Container Service (Amazon ECS) Amazon Elastic Kubernetes Services (Amazon EKS) AWS Lambda Best Practices: Performance, Cost, & Ingest 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
What is app modernization,and how does Amazon EFShelp? 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Why modernize?1. Save cost by reducingoperations burden,underutilization ofcompute and storage2. Increase agility byinstantly scaling upaccording to demand 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.3. Develop and deployapplications withgreater efficiency
Modernizing with Containers, AWS Fargate andLambdaMoreAWS LambdaServerless functionsAWS managesCustomer managesData source integrationsApplication codePhysical hardware, software,networking, and facilitiesOpinionatedProvisioningAWS FargateContainer orchestration, provisioningCluster scalingApplication codeData sourceintegrationsServerless containersPhysical hardware, host OS/kernel,networking, and facilitiesAmazon ECS/EKSContainer orchestration control planeApplication codePhysical hardware software, networking,and facilitiesData source n EC2Physical hardware software,networking, and facilitiesInfrastructure-as-a-ServiceLess 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkManagement tasksSecurity config and updatesNetwork configWork clustersApplication codeData source integrationsSecurity config and updates,network config, firewall,management tasksProvisioning, managing scalingand patching of serversScalingSecurity config and updatesNetwork configManagement tasks5
Serverless storage for your application modernizationEC2 instance oron-premises serverApp CodeExistingSAN or NASApp ContainerAmazon ECSAmazon EKSAWS FargateApp CodeAmazon EFS 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.File SystemLambda FunctionApp Code
Applications that need persistent storageLong-runningstateful applicationsShared data setsDevelopertoolsWeb and contentmanagementMachinelearningDatascience FlowJupyterHubAirflow 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Perform AI/ML and Analytics with AWS LambdaImport large AI models quicklyLoad extra code librariesDeploy Modeling and AI solutions with Lambda/tmpAWS LambdaAmazon EFS 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.TensorFlowPyTorch, NumPy, KerasPre trained AI ModelsC Libraries
Streamline Media Processing on the Serverless PlatformSimplify Application ArchitectureProcess files of any sizeReduce CostsAWS LambdaImagesVideosAmazon EFSFile systemAudio RecordingsOne location for ALL files 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Traditional storage is not designed for modern applicationsLack ofscalabilityLack of Agility 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.Administrativeoverhead
Amazon EFSServerless File StorageCloud nativeHighly reliable 20202020, AmazonWebServices,Inc. orits orAffiliates.All rightsAmazon Confidentialand Trademark.AmazonWebServices,Inc.its Affiliates.Allreserved.rights reserved.Amazon Confidentialor TrademarkCost optimized
Simplify Persistent Storage for Modern Applicationswith Amazon EFS for Amazon ECS, EKS, and LambdaSimpleElasticEFS, Fargate, & Lambda are fullyserverless, so no instances tomanage. Amazon EFS configurationis inside Amazon ECS/EKS/Lambdaapp definition. Developers can focuson their applications, notinfrastructure.Amazon ECS, Amazon EKS, AWSFargate, and Amazon EFS are elastic,scale up and down rapidly based ondemand. Customers pay only for whatthey use.Available and DurableAmazon ECS, Amazon EKS, AWSFargate, and Amazon EFS areregional services. Customers canbuild applications that spanmultiple availability zones, withautomatic failover. 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.SecureAccess to Amazon EFS can berestricted based on the IAM role ofthe Amazon ECS task.Amazon EFS Access Points canenforce file system permissionswhen multiple apps share a filesystem.
Asurion builds on-demand MLusing AWS Lambda & Amazon EFSChallengeSolutionBenefitsPerform real-time analysis ofcustomer experience duringsupport calls using ML. Callrecordings didn’t fit in Lambda/tmp space.Asurion uses Amazon EFS togive extended storage space totheir ML functions running inAWS Lambda. ML inference infrastructurescales elastically with callvolume“ Reduced operationaloverhead compared tomaintaining instances andauto-scaling.Company: AsurionIndustry: Insurance ServicesCountry: United StatesEmployees: 10K Website: asurion.comAbout AsurionWe really wanted to use AWS Lambda to make ourML inference elastic, but thought we wouldn’t be ableto because of the size of data the process required.With Amazon EFS, we were easily able to give ourfunction all of the storage space it needs.– JeffSeniorPrincipalSoftwareAsurion 2020, AmazonWebTougas,Services, Inc.or its Affiliates.All rightsreserved. Engineer,Amazon Confidentialand Trademark.”We are the go-to solution for all things tech– our Experts can repair, replace andresolve nearly any tech issue. We’re easyto reach via call, chat, and in-person, too –at one of our convenient uBreakiFix storesor pick a time and place and we’ll to cometo you. With our passion for helping peoplestay connected to their tech, we’re makinglives a little bit easier—and their tech a lotmore amazing.
Acquia Modernizes Web HostingWith Amazon EKS & Amazon EFSChallengeSolutionBenefitsSought to further improveability to elastically scaleacross compute and storage toimprove end customers digitalexperience.Containerize hostingapplication and run on AmazonEKS, using Amazon EFS aspersistent storage. Dynamic scaling of customerenvironmentsCompany: Acquia Lower TCO through improvedstorage and computeutilization.Country: United States“ Reduced administrativeburden by leveraging fullymanaged servicesBy containerizing our hosting applications and runningthem on Amazon EKS and Amazon EFS we have improvedour customer experience while considerably reducing ourinfrastructure and operational maintenance overhead.– JakeFarrell,Inc.SeniorDirectorof Engineering, 2020, AmazonWeb Services,or its Affiliates.All rightsreserved. AmazonAcquiaConfidential and Trademark.”Industry: ITEmployees: 1k Website: acquia.comAbout AcquiaAcquia’s software and services were builtaround Drupal to give enterprise companiesthe ability to build, operate, and optimizewebsites, apps, and other digital experiences.Our products include: Acquia Cloud, DevStudio, Site Studio, Edge CDN, Site Factory,Acquia Lightning, Cloud IDE, Acquia DAM,Personalization, Customer Data Platform,Campaign Studio, Campaign Factory, andseveral others for the developer experience.Get the most out of Drupal and future-proofyour digital strategy.
T-Mobile scales modern applicationdeployments with Amazon EFSChallengeSolutionBenefitsCustomer facing applicationwith large spikes in usagebased on time of day andmonth of year. Existinginfrastructure was not able tosupport the scalability requiredwithout overprovision ofinfrastructure to support peakusage.Modernized applications toemploy microservices.Deployed containers viaKubernetes and Mesos withEFS providing persistentstorage and ability todynamically scale applicationwithout storage managementoverhead 16,000 containers undermanagement“ Reduced cost of NFS storageby 70% compared to DIYwhile reducing storagemanagement overhead Improved cycle time fordeploying application servicesCompany:T-MobileIndustry:Mobile e:www.t-mobile.comAbout T-MobileWe are a large organization that has lots ofapplications with varying requirements for availabilityand performance. EFS provides us with a commonstorage platform that meets these requirements acrossthe board.AmrethChandrasehar,PrincipalT-Mobile 2020, AmazonWeb Services,Inc. or its Affiliates.All rightsArchitect,reserved. AmazonConfidential and Trademark.”As America's Un-carrier, T-Mobile US,Inc. is redefining the way consumersand businesses buy wireless servicesthrough leading product and serviceinnovation. The Company's advancednationwide network deliversoutstanding wireless experiences to79.7 million customers who areunwilling to compromise on qualityand value.
Journey to (and in) the cloud Moved containerized data scienceenvironment to AWS for agility andcost benefits Enabled self-service provisioning ofcontainerized analytics applicationsand compute resources Migrated to a managed service forbetter stability, application scalingand ease of operations, reducingstorage management time by 90% 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Setting up EFS for modern apps 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Goals for Security & Identity1. File systems shouldonly be mountable bythe applications thatneed them2. Apps that mount filesystems should onlyhave access to the datathey needAmazon EFSFile System cat /my app/data### SUCCESS THIS IS MY FILE ### cat /someone elses app/datacat: /someone elses app/data : Permission denied 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Using IAM for File System AccessAmazon ElasticContainer ServiceECSEKSLambdaTask RoleTaskDefinitionAWS IAMAWS LambdaFunctionExecutionRole{Amazon EFSFile System{“Statement” : {“Effect” : “allow”,“Action” : “elasticfilesystem:Client*”,"Resource": ”fs-deadbeef"}}“Statement” : {“Effect” : “allow”,“Action” : “elasticfilesystem:Client*”,“Principal” : { “AWS”: “FargateRole” }}} 20202020, AmazonWebServices,Inc. orits orAffiliates.All rightsAmazon Confidentialand Trademark.AmazonWebServices,Inc.its Affiliates.Allreserved.rights reserved.Amazon Confidentialor Trademark
Handling EFS Authorization Using IAMECSEKSLambda“Effect” : “allow”,“Action” : � : trust”Squashed to 65535“Effect” : “allow”,“Action” : esystem:ClientWrite”],“Principal” : { “AWS”: “semitrust” }“Effect” : “allow”,“Action” : RootAccess],“Principal” : { “AWS”: “fulltrust” } 20202020, AmazonWebServices,Inc. orits orAffiliates.All rightsAmazon Confidentialand Trademark.AmazonWebServices,Inc.its Affiliates.Allreserved.rights reserved.Amazon Confidentialor Trademark
Understanding Container IdentityBy default, POSIX identity comesfrom the container image, notthe task/pod runtime.ECS TaskContainer ImageApp IdentityUser: RootGroup: RootTask Identity(IAM Role)AWS IAM ls –l /efs/homedrwx------ bob.BobHomedrwx------ sally.SallyHomedrwxrwx--- .biusersBI Shared 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Understanding Function IdentityLambda FunctionBy default, Lambda functionshave no pre-determined identityAWS IAM ls –l /efs/homeTask Identity(IAM Role)drwx------ bob.BobHomedrwx------ sally.SallyHomedrwxrwx--- .biusersBI Shared 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Application-specific Access with EFS Access Points ECSEKSLambda{“Name”: “MyApp”,"FileSystemId": ”fs-deadbeef",“PosixUser”: {“Uid”: 123“Gid”: 123,“SecondaryGids”: [100, 200, 300]},“RootDirectory”: {“Path”: “/apps/myapp”,“CreationInfo”: {“OwnerUid”: 123,“OwnerGid”: 123,“Permissions”: “0700”}}Enforces File System IdentityRoot containers can’t escalate accessArbitrary users aren’t locked outCreates App-specific Directory & PermissionsNo EC2 instance required!Apps only see data they need} 20202020, AmazonWebServices,Inc. orits orAffiliates.All rightsAmazon Confidentialand Trademark.AmazonWebServices,Inc.its Affiliates.Allreserved.rights reserved.Amazon Confidentialor Trademark
How EFS Access Points WorkECSEKSLambda{“Name”: “MyApp”,“PosixUser”: {“Uid”: 123“Gid”: 123,“SecondaryGids”: [100, 200, 300]},“RootDirectory”: {“Path”: “/apps/myapp”,“CreationInfo”: {“OwnerUid”: 123,“OwnerGid”: 123,“Permissions”: “0700”}}}“Effect” : “allow”,“Action” : “elasticfilesystem:Client*”,“Principal” : { “AWS”: “approle” },“Condition”* : {“accessPointArn” : “fsap-1234” 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential or TrademarkFile Systemwith POSIXPermissions
Best Practices for IAM and Access Points, & Security Use Access Points, even if single app per file system! Simplifies directory permission setup Consistent experience regardless of user/group setup in container Future-proof for adding apps to share data Use IAM Authorization Use Resource Policies to restrict IAM roles to Access Points Use Identity Policies to give single role “admin” access to file systems Enable Encryption @ Rest and Encryption in Motion Simple setup, no performance penalty 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Connecting your app to EFS 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
ECS 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
How it worksAmazon ElasticContainer ServiceContainer 1EFSVolumeConfigurationContainer 2File systemTaskAmazon EFSAmazon EC2 20202020, AmazonWebServices,Inc. orits orAffiliates.All rightsAmazon Confidentialand Trademark.AmazonWebServices,Inc.its Affiliates.Allreserved.rights reserved.Amazon Confidentialor TrademarkAmazon ECSAWS Fargate
Amazon ElasticContainer Service 20202020, AmazonWebServices,Inc. orits orAffiliates.All rightsAmazon Confidentialand Trademark.AmazonWebServices,Inc.its Affiliates.Allreserved.rights reserved.Amazon Confidentialor Trademark
Amazon ElasticContainer Service 20202020, AmazonWebServices,Inc. orits orAffiliates.All rightsAmazon Confidentialand Trademark.AmazonWebServices,Inc.its Affiliates.Allreserved.rights reserved.Amazon Confidentialor Trademark
"containerDefinitions": [{."mountPoints": [{"readOnly": null,"containerPath": "/data","sourceVolume": "FargateDemoEFS"}],."name": "FileBrowser"}],"taskRoleArn": "arn:aws:iam::.:role/FargateRole",."volumes": [{"efsVolumeConfiguration": {"transitEncryptionPort": null,"fileSystemId": "fs-41c7f3c1","authorizationConfig": {"iam": "ENABLED","accessPointId": "fsap-0f7741bf379626fc2"},"transitEncryption": "ENABLED","rootDirectory": "/"},"name": "FargateDemoEFS", 20202020, AmazonWebServices,Inc. orits orAffiliates.All rightsAmazon Confidentialand Trademark.AmazonWebServices,Inc.its Affiliates.Allreserved.rights reserved.Amazon Confidentialor TrademarkAmazon ElasticContainer Service
EKS 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Amazon EFS & EKS: Concepts Container Storage Interface (CSI)Amazon ElasticKubernetes Service Industry standard interface for connecting storage providers (block or file) to a container. Amazon EFS CSI Driver Implementation of CSI for connecting file systems to containers. Storage Class (SC) Administrator-defined class of storage that Persistent Volumes can be created from. Persistent Volume (PV) Administrator-created unit of storage that can be attached to a container. Has its ownlifecycle. Persistent Volume Claim (PVC) Request to allocate an available PV from a SC to a container. 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Amazon EKS storage – InstallationAmazon EKSAmazon EKS on Amazon EC2# kubectl apply -k ploy/kubernetes/overlays/stable/ecr/?ref release-1.0”#Amazon EKS on Fargate# 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Attaching an Amazon EFS filesystem to a pod (admin)Create storage classAmazon EKSCreate persistent volumekind: StorageClassapiVersion: storage.k8s.io/v1metadata:name: efs-scprovisioner: efs.csi.aws.com 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.apiVersion: v1kind: PersistentVolumemetadata:name: efs-pvspec:capacity:storage: 5GivolumeMode: FilesystemaccessModes:- ReadWriteManypersistentVolumeReclaimPolicy: RetainstorageClassName: efs-sccsi:driver: efs.csi.aws.comvolumeHandle: fs-deadbeef::fsap-deadbeefdead
VolumeHandle path and access pointNewvolumeHandle: {fsid}:{subpath}:{apid}ExamplesMounting a subpath: fs-deadbeef:/myapppath/:Mounting an AP: fs-deadbeef::fsap-feeddeadbeeffeedSubmounting an AP: fs-deadbeef:/subpath:fsap-feeddeadbeeffeed 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.Amazon EKS
Attaching an Amazon EFS file system to apod (user)Create persistent volume claimLaunch podapiVersion: v1kind: PersistentVolumeClaimmetadata:name: efs-claimspec:accessModes:- ReadWriteManystorageClassName: efs-scresources:requests:storage: 5Gi 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.apiVersion: v1kind: Podmetadata:name: efs-appspec:containers:- name: web-containerimage: httpdports:- containerPort: 80name: “http-server”volumeMounts:- name: persistent-storagemountPath: /mnt-efsvolumes:- name: persistent-storagepersistentVolumeClaim:claimName: efs-claimAmazon EKS
Lambda 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Attaching Amazon EFS to a function1. Create1. File System2. Mount Targets3. Access Point2. Configure function for VPC of file system3. Add file system1. Select File System2. Select Access Point 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Best Practices:Performance, Cost,& Ingest 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Best practices – Performance Use General Purpose (GP) formost applications GP lower latency, now supports up to35K read IOPS Max I/O for scale-out analytics / MLthat need 100K IOPS Configure provisionedthroughput for initial need –as file system grows, you’lleventually be given higherthroughput Set up Amazon CloudWatch;monitor throughput, IOPS,and burst credits* 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.* ster/monitoring
When should I use EFS vs EBS?AmazonElastic FileSystemAmazon ElasticBlock Store I need to share data between containers I don’t need shared storage (e.g. Database) I’d like to run across instances or AZs I need point in time snapshots I’d like to take advantage of spot pricingNote: Amazon FSx for Lustre can be used for containers that require ultra-high throughput and very low latency file sharing 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Optimize cost with Amazon EFS Infrequent AccessAmazon EFS IA storage class for infrequently accessed files for 0.025/GB/mo*No changes to existingapplications usingAmazon EFSCostsavings upto 92%Automatedlifecyclemanagement* 20202020, AmazonWebServices,Inc. orits orAffiliates.All rightsAmazon Confidentialand Trademark.AmazonWebServices,Inc.its Affiliates.Allreserved.rights reserved.Amazon Confidentialor TrademarkPricing in the US East (N. Virginia) region
Backup for Amazon EFS EFS file systems can be backedup and restored usingAWS Backup AWS Backup provides automatedbackup scheduling and retentionper user defined policyAmazon EFSBackup encryptionWarm storageAWS BackupCold storage 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. AWS Backup offers two classes ofservice backup storage with theability to lifecycle to cold storage Restore individual filesand directories
Migrating NFS workloads to Amazon EFSOn-PremiseAWS RegionLinux EC2InstancesLinux ApplicationServersInternetNFSNFSNAS FilerDataSyncagentAWSDirect ConnectNFSAWS DataSyncVirtualmachineVPNEFS FileSystemAWS DataSync: Online transfer service that simplifies, automates, andaccelerates moving data between on-premises storage and AWS 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Where to learn moreAmazon EFS: Secure data persistence with Amazon ECS and AWS FargateDevelopers guide to using Amazon EFS with Amazon ECS and AWS FargateDeploying Jenkins on Amazon EKS with Amazon EFSUsing Amazon EFS for AWS Lambda in your serverless applications 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Thank you! 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Acquia's software and services were built around Drupal to give enterprise companies the ability to build, operate, and optimize websites, apps, and other digital experiences. Our products include: Acquia Cloud, Dev Studio, Site Studio, Edge CDN, Site Factory, Acquia Lightning, Cloud IDE, Acquia DAM, Personalization, Customer Data Platform,
