WIXLIB

AWS DataSyncUser Guide

AWS DataSync User GuideAWS DataSync: User GuideCopyright Amazon Web Services, Inc. and/or its affiliates. All rights reserved.Amazon's trademarks and trade dress may not be used in connection with any product or service that is notAmazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages ordiscredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who mayor may not be affiliated with, connected to, or sponsored by Amazon.

AWS DataSync User GuideTable of ContentsWhat is AWS DataSync? . 1Use cases . 1Benefits . 1Additional resources . 2How AWS DataSync works . 3DataSync architecture . 3Transferring between on-premises storage and AWS . 3Transferring between AWS storage services . 3Transferring between cloud storage systems and AWS storage services . 4Components and terminology . 4Agent . 4Location . 5Task . 5Task execution . 5How DataSync transfers files . 6How AWS DataSync verifies data integrity . 6How DataSync handles open and locked files . 6Setting up . 7Signing up for an AWS account . 7Where can I use DataSync? . 7How can I use DataSync? . 7Paying for DataSync . 7Requirements . 8Agent requirements . 8Supported hypervisors . 8Virtual machine requirements . 9Amazon EC2 instance requirements . 9Network requirements . 9Network requirements to connect to your self-managed storage . 9Network requirements when using VPC endpoints . 11Network requirements when using public or FIPS endpoints . 13Required network interfaces for data transfers . 17Getting started . 19Create an agent . 19Deploy your agent . 20Choose a service endpoint . 24Activate your agent . 26Configure a source location . 27Configure a destination location . 28Configure task settings . 29Data verification options . 29Ownership and permissions-related options . 30File metadata and management options . 30Bandwidth options . 31Filtering options . 31Scheduling and queueing options . 31Tags and logging options . 32Review and create your task . 32Start your task . 32Clean up resources . 33Using the AWS CLI . 34Creating an agent . 34Creating locations . 37Creating an NFS location . 37iii

AWS DataSync User GuideCreating an SMB location .Creating an HDFS location .Creating an object storage location .Creating an Amazon EFS location .Creating an Amazon FSx for Windows File Server location .Creating an Amazon FSx for Lustre location .Creating an Amazon FSx for OpenZFS location .Creating an Amazon FSx for NetApp ONTAP location .Creating an Amazon S3 location .Creating a task .Starting a task .Monitoring your task .Monitoring your task in real time .Filtering resources .Parameters for filtering .Filtering by location .Filtering by task .Working with agents .Creating an agent .Using DataSync in a VPC .How DataSync works with VPC endpoints .Configuring DataSync to use private IP addresses for data transfer .Deploying your agent in AWS Regions .Transferring data from a cloud file system to another cloud file system or Amazon S3 .Data transfer from S3 to in-cloud file systems .Editing your agent's properties .Using multiple agents for a location .Agent statuses .Deleting an agent .Configuring your agent for multiple NICs .Working with your agent's local console .Logging in to the agent local console .Obtaining an activation key by using the local console .Configuring your agent network settings .Testing your agent connectivity to the internet .Testing connectivity to storage systems .Viewing your agent system resource status .Synchronizing your VMware agent time .Running AWS DataSync commands on the local console .Getting help with your agent from AWS Support .Working with locations .Supported transfers in the same AWS account .Supported transfers across AWS accounts .Supported transfers across AWS Regions .Creating an NFS location .NFS location settings .NFS server on AWS Snowcone .Creating an SMB location .Creating the location .Understanding the location settings .Creating an HDFS location .Unsupported HDFS features .Creating an object storage location .Prerequisites .Considerations when migrating to or from a Google Cloud Storage bucket .Creating the location .Creating an Amazon EFS location 27273757575767677

AWS DataSync User GuideAccessing Amazon EFS file systems . 77Considerations with Amazon EFS locations . 77Creating the location . 78Using IAM policies to access your Amazon EFS file system . 79Creating an FSx for Windows File Server location . 80Creating an FSx for Lustre location . 82Creating an FSx for OpenZFS location . 83Creating the location . 83Configuring file system authorization . 83Creating an FSx for ONTAP location . 84Accessing FSx for ONTAP file systems . 84Creating the location . 85Creating an Amazon S3 location . 87Considerations when working with Amazon S3 storage classes in DataSync . 88Manually configuring an IAM role to access your Amazon S3 bucket . 90How DataSync handles metadata and special files . 92Metadata copied by DataSync . 92Links and directories copied by DataSync . 96Deleting a location . 96Working with tasks . 97Creating your task . 97Prerequisite: Creating the locations for your DataSync task . 97Creating a task to transfer data between self-managed storage and AWS . 98Creating a task to transfer between in-cloud locations . 98Configuring task settings . 102Filtering data . 103Filtering terms, definitions, and syntax . 103Excluding data from a transfer . 104Including data in a transfer . 105Example filters . 105Scheduling your task . 106Configuring a task schedule . 107Editing a task schedule . 107Task creation statuses . 108Starting your task . 108Queueing task executions . 108Working with task executions . 109Adjust bandwidth throttling . 109Task execution statuses . 109Cancel a task execution . 110Deleting your task . 111Monitoring . 112Accessing CloudWatch metrics . 112DataSync CloudWatch metrics . 112CloudWatch events for DataSync . 113DataSync dimensions . 114Uploading logs to Amazon CloudWatch log groups . 114Security . 116Data protection . 116Encryption in transit . 116Encryption at rest . 118Internetwork traffic privacy . 119Identity and access management . 119Overview of managing access . 119Using identity-based policies (IAM policies) . 124Cross-service confused deputy prevention . 127API permissions reference . 128v

AWS DataSync User GuideLogging .Working with AWS DataSync information in CloudTrail .Understanding AWS DataSync log file entries .Compliance validation .Resilience .Infrastructure security .Quotas and limits .Task quotas .Task execution quotas .File system limits .Filter limits .Request a quota increase .Troubleshooting .I need DataSync to use a specific NFS or SMB version to mount my share .What does the "Failed to retrieve agent activation key" error mean? .I can't activate an agent I created using a VPC endpoint .My task status is unavailable and indicates a mount error .My task failed with an input/output error message .My task is stuck in launching status .My task failed with a permissions denied error message .My task has had a preparing status for a long time .How long does it take to verify a task I've run? .My storage cost is higher than I expected .I don't know what's going on with my agent. Can someone help me? .How do I connect to an Amazon EC2 agent's local console? .My task fails when transferring to an S3 bucket in another AWS account. .My task fails when transferring from a Google Cloud Storage bucket .Tutorials .Transferring from on-premises to S3 in another account .Overview .Prerequisites .Step 1: Create an IAM role for DataSync in Account A .Step 2: Disable ACLs for your S3 bucket in Account B .Step 3: Update the S3 bucket policy in Account B .Step 4: Create a DataSync destination location for the S3 bucket .Step 5: Create and start a DataSync task .Related resources .Transferring from S3 to S3 in another account .Overview .Prerequisites .Step 1: Create an IAM role for DataSync in Account A .Step 2: Disable ACLs for your S3 bucket in Account B .Step 3: Update the S3 bucket policy in Account B .Step 4: Create a DataSync destination location for the S3 bucket .Step 5: Create and start a DataSync task .Related resources .Transferring from Google Cloud Storage to S3 .Overview .Costs .Prerequisites .Step 1: Create an HMAC key for your Google Cloud Storage bucket .Step 2: Configure your network .Step 3: Create a DataSync agent .Step 4: Create a DataSync source location for your Google Cloud Storage bucket .Step 5: Create a DataSync destination location for your S3 bucket .Step 6: Create and start a DataSync task .Additional resources 158159159159160160161162

AWS DataSync User GuideTransferring data from a self-managed storage array .Other use cases .Transferring files in opposite directions .Using multiple tasks to write to the same Amazon S3 bucket .Allowing DataSync to access a restricted Amazon S3 bucket .DataSync API .Actions .

Windows File Server, FSx for Lustre, or FSx for OpenZFS. DataSync includes automatic encryption and data integrity validation to help make sure that your data arrives securely, intact, and ready to use. Archiving cold data - Move cold data stored in on-premises storage directly to durable and secure