AWS SQS to Lambda 429 Errors

Imagine a scenario where a Lambda function polls messages from an SQS queue and uses the payload from those messages to make a POST request to an AWS OpenSearch domain via the REST API endpoint.

Issue

During batch processing, multiple Lam…


This content originally appeared on DEV Community and was authored by MacAndersonUche

Imagine a scenario where a Lambda function polls messages from an SQS queue and uses the payload from those messages to make a POST request to an AWS OpenSearch domain via the REST API endpoint.

Image description

Issue

During batch processing, multiple Lambda instances are triggered simultaneously as soon as messages arrive in the queue. This led to some requests being processed too quickly, causing a 429 (Too Many Requests) error due to rate limiting.

Fix

We implemented two solutions:

Reserved Concurrency: We limited the Lambda function's concurrency to 1 to ensure that only one instance processes messages at a time. This was done using CloudFormation:

   Lambda:
     Type: AWS::Serverless::Function
     Properties:
       ReservedConcurrentExecutions: 1

Queue Delay: We set a value for DelaySeconds in the SQS queue to introduce a delay before the next message is processed, allowing OpenSearch more time to handle requests. This was also configured using CloudFormation.

  Queue:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 40
      DelaySeconds: 30


This content originally appeared on DEV Community and was authored by MacAndersonUche


Print Share Comment Cite Upload Translate Updates
APA

MacAndersonUche | Sciencx (2024-09-18T22:04:56+00:00) AWS SQS to Lambda 429 Errors. Retrieved from https://www.scien.cx/2024/09/18/aws-sqs-to-lambda-429-errors/

MLA
" » AWS SQS to Lambda 429 Errors." MacAndersonUche | Sciencx - Wednesday September 18, 2024, https://www.scien.cx/2024/09/18/aws-sqs-to-lambda-429-errors/
HARVARD
MacAndersonUche | Sciencx Wednesday September 18, 2024 » AWS SQS to Lambda 429 Errors., viewed ,<https://www.scien.cx/2024/09/18/aws-sqs-to-lambda-429-errors/>
VANCOUVER
MacAndersonUche | Sciencx - » AWS SQS to Lambda 429 Errors. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2024/09/18/aws-sqs-to-lambda-429-errors/
CHICAGO
" » AWS SQS to Lambda 429 Errors." MacAndersonUche | Sciencx - Accessed . https://www.scien.cx/2024/09/18/aws-sqs-to-lambda-429-errors/
IEEE
" » AWS SQS to Lambda 429 Errors." MacAndersonUche | Sciencx [Online]. Available: https://www.scien.cx/2024/09/18/aws-sqs-to-lambda-429-errors/. [Accessed: ]
rf:citation
» AWS SQS to Lambda 429 Errors | MacAndersonUche | Sciencx | https://www.scien.cx/2024/09/18/aws-sqs-to-lambda-429-errors/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts