How to sync EFS and S3 every 5 min in AWS

DataSync is a powerful tool to move data between different AWS storage options like S3, EFS, and EFx. However, there is a catch, the minimum time you can schedule a task is one per hour, you cannot create a custom cron expression for a lower time like …


This content originally appeared on DEV Community and was authored by Federico Navarrete

DataSync is a powerful tool to move data between different AWS storage options like S3, EFS, and EFx. However, there is a catch, the minimum time you can schedule a task is one per hour, you cannot create a custom cron expression for a lower time like */5 * * * *. My guessing about this restriction is that this feature was planned for Data Warehousing, not for active synchronization.

My challenge started when I had to read some XMLs using RDS for SQL Server. RDS for SQL Server can read the files from S3 natively, but my files came from several micro-services running in Fargate that have only access to EFS as a volume. These files came from external services at different times of the day and represented several gigas to transfer.

In the beginning, I was trying to find a way to read the EFS from SQL Server but it didn't work. RDS doesn't have an option to read EFS because it runs in Windows, and there is not a Linux option available yet, which could potentially give us access to EFS.

However, you cannot do it without a plugin that requires your container to run as privileged and this is not authorized.

After several failed attempts, I created a workaround that involves:

  1. A DataSync task for creating the basic task and synchronizing the data.
  2. A Lambda function for running the task.
  3. A EventBridge rule for triggering the Lambda function every 5 min.

DataSync

Step 1:

Configure your data source (EFS, for instance):

Step 1

Step 2:

Choose the destination (S3, for instance):

Step 2

Step 3:

Configure what you want to move.

Step 4:

Review your new task and create it.

Lambda

This is the Python script that I wrote:

import boto3

client = boto3.client('datasync', region_name='YOUR_REGION')

def lambda_handler(event,context):

    response = client.start_task_execution(
        TaskArn='arn:aws:datasync:YOUR_REGION:YOUR_USER_ID:task/YOUR_TASK_ID'
    )

Where,

  • YOUR_REGION is the location where you want to run it like eu-west-1.
  • YOUR_USER_ID is the user that is going to run the task.
  • YOUR_TASK_ID is the task ID created in the DataSync.

EventBridge

Create a new rule that runs in your expected schedule.

Step 1:

Create a new rule:

New rule

Step 2:

Configure your schedule:

Step 2

Step 3:

Choose your lambda function:

Step 3

Step 4:

Review your new rule and create it.

And that's all. Now, you can run your task DataSync in your required schedule.


This content originally appeared on DEV Community and was authored by Federico Navarrete


Print Share Comment Cite Upload Translate Updates
APA

Federico Navarrete | Sciencx (2021-11-04T11:10:54+00:00) How to sync EFS and S3 every 5 min in AWS. Retrieved from https://www.scien.cx/2021/11/04/how-to-sync-efs-and-s3-every-5-min-in-aws/

MLA
" » How to sync EFS and S3 every 5 min in AWS." Federico Navarrete | Sciencx - Thursday November 4, 2021, https://www.scien.cx/2021/11/04/how-to-sync-efs-and-s3-every-5-min-in-aws/
HARVARD
Federico Navarrete | Sciencx Thursday November 4, 2021 » How to sync EFS and S3 every 5 min in AWS., viewed ,<https://www.scien.cx/2021/11/04/how-to-sync-efs-and-s3-every-5-min-in-aws/>
VANCOUVER
Federico Navarrete | Sciencx - » How to sync EFS and S3 every 5 min in AWS. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2021/11/04/how-to-sync-efs-and-s3-every-5-min-in-aws/
CHICAGO
" » How to sync EFS and S3 every 5 min in AWS." Federico Navarrete | Sciencx - Accessed . https://www.scien.cx/2021/11/04/how-to-sync-efs-and-s3-every-5-min-in-aws/
IEEE
" » How to sync EFS and S3 every 5 min in AWS." Federico Navarrete | Sciencx [Online]. Available: https://www.scien.cx/2021/11/04/how-to-sync-efs-and-s3-every-5-min-in-aws/. [Accessed: ]
rf:citation
» How to sync EFS and S3 every 5 min in AWS | Federico Navarrete | Sciencx | https://www.scien.cx/2021/11/04/how-to-sync-efs-and-s3-every-5-min-in-aws/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts