Configure AWS SQS messages to trigger a Lambda function

August 9, 2018 Leave a comment

A new feature has been added to Amazon SQS feature on June 28, 2018. The way feature has been described as

“You can configure incoming messages to trigger a Lambda function.”

Amazon SQS Release Notes

SQS, allowing you to offload tasks into a background process for a faster and more resilient system. To use SQS to trigger serverless lambda functions is another handy essential tool for your toolbox. In this post, I am listing all the required information you may need to put this in action.

Few things to consider

  • AWS Simple Service Queue and Lambda function must be in the same AWS Region.
  • FIFO queues don’t support Lambda function triggers 🙁
  • Only one queue with one or more Lambda functions are allowed to associate
  • An encrypted queue cannot associate with a Lambda function in a different AWS account
  • Account level limits are impacted by other functions

Workflow

 => Standard Queue received the message
 -> direct message to lambda
 -> process the message
 -> delete from queue

Attaching an Amazon SQS queue as an AWS Lambda event source is an easy way to process the queue’s content using a Lambda function. Lambda takes care of:

  • Automatically retrieving messages and directing them to the target Lambda function.
  • Deleting them once your Lambda function successfully completes.

However, when AWS services are using as event sources, the invocation type is predetermined for each service. There is no way to control over the invocation type that these event sources use when they invoke your Lambda function, more.

Map Lambda with SQS

To use this feature Lambda function must be mapped with an Amazon SQS queue

From AWS Console

After login to the console, go to the configuration of your lambda function. Under the designer panel, you will find add triggers option. You may need to scroll and select SQS

Then SQS will appear on the right panel  Now define parameter values and click ADD to create the mapping. Before you save the changes please check Parameters and Execution role below

Parameters

Please note the following configuration parameters:

  • BatchSize: The number of records that AWS Lambda will retrieve from each ReceiveMessage call, details here
  • Enabled: A flag to signal AWS Lambda that it should start polling from SQS queue.
  • EventSourceArn: The ARN of SQS queue that AWS Lambda is monitoring for new messages.
  • FunctionName: The Lambda function to invoke

To find out more about these parameters check Using AWS Lambda with Amazon SQS .

Execution role

To configure Lambda function triggers using the console user must have few roles

Your Amazon SQS role must include the following permissions:

  • lambda:CreateEventSourceMapping
  • lambda:ListEventSourceMappings
  • lambda:ListFunction

Your Lambda role must include the following permissions:

  • sqs:ChangeMessageVisibility
  • sqs:DeleteMessage
  • sqs:GetQueueAttributes
  • sqs:ReceiveMessage

Real benefits of SQS triggers

  • Allow reducing infrastructure footprint
  • Cost less than equivalent workarounds
  • Make things simpler.
  • Improve fault tolerance
  • Easy error handling throw Dead Letter Queue

Advanced Topics

SQS Trigger Batch Size

Batchsize is the largest number of records that will be read from queue at once.

    • Minimum Allowed# 1
    • Maximum Allowed# 10
  • The polling records will vary between  1 and the specified Batch size

With the SQS / Lambda integration, a batch of messages succeeds or fails together. AWS will only delete the messages from the queue if your function returned successfully without any errors. Few different ways to handle this

    • Handle errors within your function code, perhaps by catching them and sending the message to a dead letter queue for further processing.
  • Calling the DeleteMessage API manually within your function after successfully processing a message.
Benefits
    • Reduce API calls
    • So this reduces costs, SQS queue the account will be charged for API calls
  • Can save time and money by processing multiple messages in a single batch, If Lambda function needs to do a costly operation each time it spins up,
      • such as initializing a database connection
    • downloading a dataset to enrich your messages,
Dead Letter Queue

When using Amazon SQS as an event source, configure a DLQ on the Amazon SQS queue itself and not the Lambda function. For more information, see Amazon SQS Dead-Letter Queues. A poison message will push back to queue and will trigger lambda. It will never stop until you have a Dead letter queue with retry policy or manually delete the message.

Scaling Behaviour

Note that Lambda automatically reserves a buffer of 100 concurrent executions for functions without any reserved concurrency limit. This means if your account limit is 1000, you have a total of 900 available to allocate to individual functions formula to estimate your concurrent Lambda function invocations:

 concurrent Lambda function invocations = events (or requests) per second * function duration

Request rate refers to the rate at which your Lambda function is invoked. For stream-based services, AWS Lambda calculates the request rate as follows:

request rate = number of concurrent executions / function duration

For more details, check AWS documentation

  • AWS Lambda’s automatic scaling behavior is designed to keep polling costs low when a queue is empty while simultaneously enabling you to achieve high throughput when the queue is being used heavily.
  • Lambda begins long-polling the Amazon SQS queue
  • Lambda automatically scales up polling activity until the number of concurrent function executions reaches 1000
  • SQS supports an initial burst of 5 concurrent function invocations and increases concurrency by 60 concurrent invocations per minute
  • AWS Lambda will continue to increase the number of concurrent function executions by 500 per minute until your account safety limit has been reached or the number of concurrently executing functions is sufficient to successfully process the increased load.
Few Amazon SQS queue Attribute To consider
Visibility Timeouts

A period of time during which Amazon SQS prevents other consumers from receiving and processing the message

  • Default: 30 Seconds
  • Minimum: 30 Seconds
  • Maximum: 12 Hours

You need to make sure the visibility timeout is more than you Lambda timeout

SQS Delay Queues

Any messages that you send to the queue remain invisible to consumers for the duration of the delay period
For standard queues, the per-queue delay setting is not retroactive—changing the setting doesn’t affect the delay of messages already in the queue.

The difference between the two Visibility Timeout and Delay Queues

  • for delay queues, a message is hidden when it is first added to queue,
  • for visibility timeouts a message is hidden only after it is consumed from the queue.

Next

Considering features, cost, and performance, SQS is the best choice for the task of message distribution between Lambda functions. This is not all. I have added a few helpful blogs regarding the features that helped me to understand how all things are connected

Please leave your feedback at the comment section. Any suggestions to improve the post will be highly appreciated 

I would like to hear your thoughts