Best practices: Configuring AWS Lambda SQS batch size

Why you should avoid setting batch size = 1 for the SQS/Lambda event pattern.

When using Lambda with SQS as an event source, one of the parameters that can be configured is the batch size. This parameter controls the maximum number of records that the Lambda poller service will accumulate before invoking the Lambda function. Default batch size for SQS event source is 10. However, I've seen several teams setting batch size = 1.

SQS batch size configuration example

Setting the batch size = 1 should always be avoided when using Lambda with an SQS event source as it negatively impacts cost and performance. Let us take a look at two scenarios to understand how cost and performance are impacted by  batch size.

Explore #LifeAtCapitalOne

Feeling inspired? So are we.

Batch size configuration scenario and descriptions

A Lambda function is configured with an SQS event source and a burst of 200 messages is pushed to the SQS queue.

  • Scenario 1 uses batch size = 10
  • Scenario 2 uses batch size = 1

Metrics observed for Lambda function using CloudWatch:

 Metric:  Statistic:
 Innovations

 Metrics observed for Lambda function using  CloudWatch

 

 Metric:  Statistic:
 Duration   Average -> Average processing duration of the Lambda function

 

 Metric:  Statistic:
 Concurrency  Max -> Maximum number of Lambda function instances  available for processing the burst of messages

 

Invocation scenario and examples

  1. Scenario 1 ( batch size =10) has 21 invocations.
  2. Scenario 2 ( batch size = 1) has 200 invocations.

Note: Although we set the batch size, AWS doesn’t guarantee every batch size to be exactly the same. However, the batch will never exceed the set batch size limit.

In the case of Scenario 1 ( batch =10),  one may anticipate to have 20 batches,  each of size 10, totalling 200 messages. Instead, we observe 24 invocations. This shows that although we can specify a batch size, it is not guaranteed that each batch will be of the same size. 

Scenario 1 example showing batch=10 with 24 invocations

In the case of Scenario 2  (batch size=1), we have exactly 200 invocations since the batch size limit cannot be exceeded.

Scenario 2 example showing batch=1 with 200 invocations

Lambda duration scenario and examples

Average duration for scenario 1 (batch size=10) is 422 ms. 

  • The duration is higher as each Lambda invocation is processing a batch of 10 messages. 

Scenario 1 example showing batch=10 with 422 ms duration

Average duration for scenario 2 (batch size= 1) is 106 ms. 

  • The duration is lower than Scenario 1 as each Lambda function is processing only one message.

Scenario 2 example showing batch=1 with 106 ms duration

Lambda concurrency scenario and examples

Concurrency increases from 3 to 6 as we change batch size from 10 to 1.

Scenario 1 example showing increase of concurrency from 3 to 6 with batch=1

Lambda cost and performance comparison summary

   

 Scenario 1

 (Batch size = 10)

 200 Messages

Scenario 2

 (Batch size = 1)

 200 Messages

 Lambda invocations  21  200
 Average duration of Lambda  422 ms  106 ms
 Lambda concurrency  3  6

 Lambda cost

 Using AWS price calculator

 (200 request per second and ARM   processor)

 $158.09  $475.82
Approximate overall duration to drain messages from the queue 8,862 ms 21,200 ms

 

  • Based on the above data we see that using batch size =1, is approximately 200% more expensive than using batch size=10.
  • Additionally, using a low batch size could exhaust your Lambda concurrency for the AWS account if there is a high volume of message throughput.
  • By default total Lambda concurrency limit is 1000 per account, however this is a soft limit and can be modified by requesting a quota increase.

SQS batch size considerations

If you are setting batch size =1, so that in case of  failures you would like  to reprocess  a single event, you should consider using batch item failures. BatchItem failures enables Lambda function to reprocess only failed messages instead of reprocessing the entire batch.

Avoid configuring SQS batch size 1 for efficiency

Avoid configuring SQS batch size of 1 for Lambda with a SQS event source as

  • It is expensive
  • It can lead to higher processing times for high message volumes (if Lambda function needs to scale rapidly)
  • It can exhaust available concurrency for Lambda functions in the AWS account

Enable batch item failures  for handling error records and improving efficiency


Sushma Onkar, Distinguished Engineer

Sushma is a distinguished engineer at Capital One focusing on serverless initiatives. She works with engineering teams across the enterprise to design and build serverless data pipelines.

Related Content