Best practices: Configuring AWS Lambda SQS batch size
Why you should avoid setting batch size = 1 for the SQS/Lambda event pattern.
When using Lambda with SQS as an event source, one of the parameters that can be configured is the batch size. This parameter controls the maximum number of records that the Lambda poller service will accumulate before invoking the Lambda function. Default batch size for SQS event source is 10. However, I've seen several teams setting batch size = 1.
SQS batch size configuration example
Setting the batch size = 1 should always be avoided when using Lambda with an SQS event source as it negatively impacts cost and performance. Let us take a look at two scenarios to understand how cost and performance are impacted by batch size.
Batch size configuration scenario and descriptions
A Lambda function is configured with an SQS event source and a burst of 200 messages is pushed to the SQS queue.
- Scenario 1 uses batch size = 10
- Scenario 2 uses batch size = 1
Metrics observed for Lambda function using CloudWatch:
| Metric: | Statistic: |
| Innovations |
Metrics observed for Lambda function using CloudWatch |
| Metric: | Statistic: |
| Duration | Average -> Average processing duration of the Lambda function |
| Metric: | Statistic: |
| Concurrency | Max -> Maximum number of Lambda function instances available for processing the burst of messages |
Invocation scenario and examples
- Scenario 1 ( batch size =10) has 21 invocations.
- Scenario 2 ( batch size = 1) has 200 invocations.
Note: Although we set the batch size, AWS doesn’t guarantee every batch size to be exactly the same. However, the batch will never exceed the set batch size limit.
In the case of Scenario 1 ( batch =10), one may anticipate to have 20 batches, each of size 10, totalling 200 messages. Instead, we observe 24 invocations. This shows that although we can specify a batch size, it is not guaranteed that each batch will be of the same size.
Scenario 1 example showing batch=10 with 24 invocations
In the case of Scenario 2 (batch size=1), we have exactly 200 invocations since the batch size limit cannot be exceeded.
Scenario 2 example showing batch=1 with 200 invocations
Lambda duration scenario and examples
Average duration for scenario 1 (batch size=10) is 422 ms.
- The duration is higher as each Lambda invocation is processing a batch of 10 messages.
Scenario 1 example showing batch=10 with 422 ms duration
Average duration for scenario 2 (batch size= 1) is 106 ms.
- The duration is lower than Scenario 1 as each Lambda function is processing only one message.
Scenario 2 example showing batch=1 with 106 ms duration
Lambda concurrency scenario and examples
Concurrency increases from 3 to 6 as we change batch size from 10 to 1.
Scenario 1 example showing increase of concurrency from 3 to 6 with batch=1
Lambda cost and performance comparison summary
|
Scenario 1 (Batch size = 10) 200 Messages |
Scenario 2 (Batch size = 1) 200 Messages |
|
|---|---|---|
| Lambda invocations | 21 | 200 |
| Average duration of Lambda | 422 ms | 106 ms |
| Lambda concurrency | 3 | 6 |
|
Lambda cost Using AWS price calculator (200 request per second and ARM processor) |
$158.09 | $475.82 |
| Approximate overall duration to drain messages from the queue | 8,862 ms | 21,200 ms |
- Based on the above data we see that using batch size =1, is approximately 200% more expensive than using batch size=10.
- Additionally, using a low batch size could exhaust your Lambda concurrency for the AWS account if there is a high volume of message throughput.
- By default total Lambda concurrency limit is 1000 per account, however this is a soft limit and can be modified by requesting a quota increase.
SQS batch size considerations
If you are setting batch size =1, so that in case of failures you would like to reprocess a single event, you should consider using batch item failures. BatchItem failures enables Lambda function to reprocess only failed messages instead of reprocessing the entire batch.
Avoid configuring SQS batch size 1 for efficiency
Avoid configuring SQS batch size of 1 for Lambda with a SQS event source as
- It is expensive
- It can lead to higher processing times for high message volumes (if Lambda function needs to scale rapidly)
- It can exhaust available concurrency for Lambda functions in the AWS account
Enable batch item failures for handling error records and improving efficiency