This content originally appeared on DEV Community and was authored by Kevin Lactio Kemta
Day 003 - 100DaysAWSIaCDevopsChallenge : Part 1
In this article, I am going to create a cloud architecture that allow me to resize and save in DynamoDB tables all objects of type image uploaded inside S3 Bucket following these steps:
- Emit an event after all actions of type
s3:ObjectCreated:Puton the S3 Bucket - A Lambda function captures the above event and then processes it
- The Lambda function get the original object created by its key
- If the object is an image file (with an extension png, jpeg, jpg, bmp, webp or gif), resize the original image using
Sharplib docs - Store the orginal and resized images in the DynamoDB tables.
- Finally store the resized image in another Bucket
All the steps will be achived using Terraform infrastructure as code.
Architecture Diagram
Beautifull heinnn 😎🤩!? I used cloudairy chart to design it.
Create S3 buckets
The event attached to the bucket will be directed to a Lambda Function. To create S3 Event for Lambda function, we first need to create the Bucket. To do this, create a file named main.tf where all our insfrastructre will be coded. Next, let's create our buckets using terraform:
resource "aws_s3_bucket" "pictures" {
object_lock_enabled = false
bucket = "pictures-<{hash}>"
force_destroy = true
tags = {
Name = "PicturesBucket"
}
}
resource "aws_s3_bucket" "thumbs" {
object_lock_enabled = false
bucket = "pictures-<{hash}>-thumbs"
force_destroy = true
tags = {
Name = "PicturesThumbsBucket"
}
}
The object_lock_enabled parameter indicates whether this bucket has an Object Lock configuration enabled, it applies only to news resources.
The force_destroy paramater specifies that all objects should be deleted from the bucket when the buckets is destroyed to avoid errors during the destruction proccess (terraform destroy -target=aws_s3_bucket.<bucket_resource_name>).
Now that the bucket is created 🙂, let's attach a trigger to it that will notify the Lambda Function when new object is uploaded to the bucket.
resource "aws_s3_bucket_notification" "object_created_event" {
bucket = aws_s3_bucket.pictures-bucket.id
lambda_function {
events = ["s3:ObjectCreated:*"]
lambda_function_arn = aws_lambda_function.performing_images_function.arn
}
depends_on = [aws_lambda_function.performing_images_function]
}
Note that performing_images_function is the our Lambda function that will be created later in the function section, and aws_s3_bucket.pictures.id is the bucket previously created.
⚠️ Note: As mentionned in the AWS Docs, an S3 Bucket support only one notification configuration. To bypass this issue, I suggest you if you have more that one notification (Lambda invokation, SNS topic trigger, etc.), create one Lambda notification and inside the Lambda function, dispatch your information to others resources (such as other Lambda, SQS,SNS, etc.).
Block public access
The make the buckt publicly inaccessible, there is another terraform resource named aws_s3_bucket_public_access_block to create to achieve this:
resource "aws_s3_bucket_public_access_block" "private_access" {
bucket = aws_s3_bucket.pictures.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
Lambda function
Now that the bucket is created and event notification trigger is properly configured, let's create our Lambda function to catch all messages emitted by the bucket. The function code will perform the following operations:
- Retrieve object created - The first operation for our Lambda will be to retrieve the created object if it is of image.
-
Resizing the image - Use
sharplibrary to create a miniature (thumb) of the original object. - Upload resized image to s3 bucket dedicated - Upload the thumb image to another s3 bucket.
- Save the original and resized image metadata - After the image is resized without error, the metadata such as URL, Object key, size, etc., will be stored in two dynamoDB tables: one for original image and another for the resized image.
Before creating the Lambda function, we need to grant it the necessaries permissions to interact with others resources and vice versa:
- Allow the bucket to
Invokefunction -lambda:InvokeFunction - Allow the Lambda function to
getobjects inside the bucket -s3:GetObjectands3:GetObjectAcl - Allow the Lambda function to
putitems in the dynamodb tables -dynamodb:PutItem.
Lambda Assume Role
Generate an IAM policy that allow action sts:AssumeRole where identifier of type Service is lambda.amazonaws.com.
data "aws_iam_policy_document" "assume_role" {
statement {
effect = "Allow"
principals {
identifiers = ["lambda.amazonaws.com"]
type = "Service"
}
actions = ["sts:AssumeRole"]
}
}
Now attach the assume_role policy to our future Lambda resource role.
resource "aws_iam_role" "for_lambda" {
assume_role_policy = data.aws_iam_policy_document.assume_role.json
name = "iam_role_for_lambda"
tags = {
Name = "IAM:Role:Lambda"
}
}
Create IAM Policy for lambda
Let's create a new policy to allow Lambda to:
- get S3 objects
- create S3 objects
- put items into DynamoDB tables.
resource "aws_iam_policy" "lambda_policy_for_s3_and_dyanmodb" {
name = "lambda-create-object-and-put-item_policy"
description = "The IAM policy to allow Lambda to get S3 objects, put objects in S3, and put items in DynamoDB tables"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"s3:PutObject",
"s3:GetObject",
"s3:PutObjectAcl"
],
Resource = [
"${aws_s3_bucket.thumbs.arn}/*", # will be created later
]
},
{
Effect = "Allow"
Action = ["s3:GetObject", "s3:GetBucket"]
Resource = [
"${aws_s3_bucket.pictures.arn}/*"
]
},
{
Effect = "Allow"
Action = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
Resource = "*"
},
{
Effect = "Allow"
Action : ["dynamodb:PutItem"]
Resource : [
aws_dynamodb_table.pictures.arn, # will be created later
aws_dynamodb_table.thumbnails.arn # will be created later
]
}
]
})
path = "/"
tags = {
Name = "iam:policy:lambda-for-s3-and-dynamodb"
}
}
In the policy, we have also allowed Lambda to log its activities into CloudWatch, which will permit us to visualize all activities inside the Lambda as shown below:
Action = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
]
Attach Lambda policy to the Lambda role
resource "aws_iam_role_policy_attachment" "attach_policies_to_lambda_role" {
policy_arn = aws_iam_policy.lambda_policy_for_s3_and_dyanmodb.arn
role = aws_iam_role.for_lambda.name
}
And the waiting time over !!! 🧘🏾♂️ let's jump into Lambda creation
Create the lambda function
Before creating the terraform Lambda resource we need first to write code that will be executed inside the function.
Create the files index.ts ans package.json inside assets/lambda directory in the root project. Below is the content the assets/lambda/package.json:
{
"main": "index.js",
"type": "module",
"scripts": {},
"dependencies": {
"@aws-sdk/client-dynamodb": "^3.609.0",
"@aws-sdk/client-s3": "^3.609.0",
"@aws-sdk/lib-dynamodb": "^3.610.0",
"sharp": "^0.33.4",
"uuid": "^10.0.0"
}
}
and run npm install inside the assets/lambda directory to install dependencies. ⚠️ the node_modules is fondamental for the function to execute properly. (in one of my future article I will show you how to optimize it by using node_modules in the layers if you want to launch more than on function for more optimization)
cd assets/lambda
npm install
The function source code assets/lambda/index.ts:
import {GetObjectCommand, PutObjectCommand, S3Client} from "@aws-sdk/client-s3";
import sharp from "sharp";
import {DynamoDBClient} from "@aws-sdk/client-dynamodb";
import {DynamoDBDocumentClient, PutCommand} from "@aws-sdk/lib-dynamodb";
import {v4 as UUID} from "uuid";
const region = process.env.REGION;
const thumbsDestBucket = process.env.THUMBS_BUCKET_NAME;
const picturesTableName = process.env.DYNAMODB_PICTURES_TABLE_NAME;
const thumbnailsTableName = process.env.DYNAMODB_THUMBNAILS_PICTURES_TABLE_NAME;
const s3Client = new S3Client({
reqion: region,
});
const dynClient = new DynamoDBClient({
region: region,
});
const documentClient = DynamoDBDocumentClient.from(dynClient);
export const handler = async (event, context) => {
const bucket = event.Records[0].s3.bucket.name;
const objKey = decodeURIComponent(event.Records[0].s3.object.key.replace("/\+/g", " "));
if(new RegExp("[\/.](jpeg|png|jpg|gif|svg|webp|bmp)$").test(objKey)) {
try {
const originalObject = await s3Client.send(new GetObjectCommand({
Bucket: bucket,
Key: objKey
}));
console.log("Get S3 Object: [OK]");
const imageBody = await originalObject.Body.transformToByteArray();
const thumbs = await sharp(imageBody)
.resize(128)
.png()
.toBuffer();
console.log("Image resized: [OK]");
await s3Client.send(new PutObjectCommand({
Bucket: thumbsDestBucket,
Key: objKey,
Body: thumbs
}));
console.log("Put resized image into S3 bucket: [OK]");
const itemPictureCommand = new PutCommand({
TableName: picturesTableName,
Item: {
ID: UUID(),
ObjectKey: objKey,
BucketName: bucket,
Region: region,
CreatedAt: Math.floor((new Date().getTime()/1000)),
FileSize: event.Records[0].s3.object.size
}
});
await documentClient.send(itemPictureCommand);
console.log("Put original metadata into DynamoDB Table: [OK]");
const itemThumbCommand = new PutCommand({
TableName: thumbnailsTableName,
Item: {
ID: UUID(),
ObjectKey: objKey,
BucketName: thumbsDestBucket,
Region: region,
CreatedAt: Math.floor((new Date().getTime()/1000)),
FileSize: thumbs.byteLength
}
});
await documentClient.send(itemThumbCommand);
console.log("Put resized metadata into DynamoDB Table: [OK]");
console.debug({
statusCode: 200,
body: JSON.stringify({
object: `${bucket}/${objKey}`,
thumbs: `${thumbsDestBucket}/${objKey}`
})
})
} catch (e) {
console.error(e);
console.debug({
statusCode: 500,
body: JSON.stringify(e)
});
}
}
};
Return to the Terraform. Now that the source code is ready, we can now create our Terraform function resource.
Once again, we need to zip our source code, the entire assets/lambda directory including node_modules. To zip the function source code, we will use Terraform archive_file resource like this:
data "archive_file" "function" {
output_path = "./assets/func.zip"
type = "zip"
source_dir = "./assets/lambda"
}
We have zipped the entire content of assets/lambda to assets/func.zip file.
And Terraform function resource:
data "aws_region" "current" {}
resource "aws_lambda_function" "performing_images_function" {
function_name = "performing-images-function"
role = aws_iam_role.for_lambda.arn
handler = "index.handler"
runtime = "nodejs18.x"
filename = "./assets/func.zip"
source_code_hash = data.archive_file.function.output_base64sha256
memory_size = 128
timeout = 10
timeouts {
create = "30m"
update = "40m"
delete = "40m"
}
environment {
variables = {
TRIGGER_BUCKET_NAME = aws_s3_bucket.pictures-bucket.bucket
THUMBS_BUCKET_NAME = aws_s3_bucket.thumbs.bucket
REGION = data.aws_region.current.name
DYANMODB_THUMBNAILS_PICTURES_TABLE_NAME = aws_dynamodb_table.thumbnails.name
DYANMODB_PICTURES_TABLE_NAME = aws_dynamodb_table.pictures.name
}
}
depends_on = [data.archive_file.function]
tags = {
Name = "Lambda:PerformingImages"
}
}
⚠️⚠️ Note: the parameter source_code_hash is important because, if the code changes, it signals Terraform to update Lambda function with the new content.
⚠️ Also it is important to zip source before creating the function, as indicated by the line:
depends_on = [data.archive_file.function]
And the last Terraform resource and the must important one in our Lambda section, is aws_lambda_permission, that grants permission to S3 Bucket to invoke Lambda for all object-created events:
resource "aws_lambda_permission" "allow_bucket" {
statement_id = "AllowExecutionFromBucket"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.performing_images_function.arn
source_arn = aws_s3_bucket.pictures.arn
principal = "s3.amazonaws.com"
}
Create DynamoDB tables
We are now going to create two DynamoDB tables to persist the information about the original object and the resized image. As lambda function is already configured with dynamodb:PutItem, let's define those tables:
resource "aws_dynamodb_table" "pictures" {
name = "PictureTable"
table_class = "STANDARD"
hash_key = "ID"
range_key = "ObjectKey"
billing_mode = "PAY_PER_REQUEST"
dynamic "attribute" {
for_each = local.dynamo_table_attrs
content {
name = attribute.key
type = attribute.value
}
}
tags = {
Name = "dynamodb:PictureTable"
}
}
resource "aws_dynamodb_table" "thumbnails" {
name = "ThumbnailsTable"
table_class = "STANDARD"
hash_key = "ID"
range_key = "ObjectKey"
billing_mode = "PAY_PER_REQUEST"
dynamic "attribute" {
for_each = local.dynamo_table_attrs
content {
name = attribute.key
type = attribute.value
}
}
tags = {
Name = "dynamodb:ThumbnailsTable"
}
}
🥳✨woohaah!!!
We have reached the end of the article.
Thank you so much 🙂
Your can find the full source code on GitHub Repo
Feel free to leave a comment if you need more clarification or if you encounter any issues during the execution of your code.
This content originally appeared on DEV Community and was authored by Kevin Lactio Kemta
Kevin Lactio Kemta | Sciencx (2024-07-19T05:34:04+00:00) Use Lambda and DynamoDB to resize S3 Image Uploaded using TERRAFORM. Retrieved from https://www.scien.cx/2024/07/19/use-lambda-and-dynamodb-to-resize-s3-image-uploaded-using-terraform/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.