AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event sources

AWS Lambda Queues for Apache Kafka

AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event source mappings (ESM) for Kafka clusters that customers run themselves and for managed offerings such as Confluent Cloud. This enables scaling processing for Kafka workloads beyond the partition count of a topic, increasing throughput during traffic spikes and improving processing resilience.

Queue support for self-managed Kafka ESMs is built on Kafka share groups, which are available in Apache Kafka 4.2 or later. This launch enables multiple event pollers to consume concurrently from the same partition, with each event poller supporting up to 10 concurrent invocations. This decouples parallelism from partition count with no changes to producers, topic configuration, or function code.

What to do

• Create a self-managed Kafka ESM with ConsumptionMode set to Queue.
• Use the AWS Management Console, Lambda API, AWS CLI, AWS CloudFormation, AWS SAM, or AWS CDK.
• Configure provisioned mode for EPU usage if needed.

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.