Amazon MSK Replicator now supports enhanced consumer offset synchronization for bidirectional replication

Amazon MSK Replicator Enhancements

Amazon MSK Replicator now provides enhanced consumer offset synchronization for bidirectional replication, enabling applications to resume processing from the correct position when moving across Kafka clusters. This capability allows you to move producer and consumer applications between clusters independently, in any order, without the risk of data loss.

MSK Replicator automates data replication between Kafka clusters, eliminating the need to manage custom replication infrastructure or configure open-source tools. Previously, consumer group offsets were synchronized only when producers and consumers were active on the same cluster, requiring careful sequencing of application migrations between clusters and increasing the risk of duplicate message processing during rollbacks. With this enhancement, MSK Replicator synchronizes consumer group offsets across source and target clusters regardless of where producers are running, enabling applications to move between clusters without coordination constraints or data duplication risks.

What to do

• Enable enhanced consumer offset synchronization when creating a Replicator using the Amazon MSK console, AWS CLI, or AWS CloudFormation.

This capability is supported in all AWS Regions where MSK Replicator is available.

To learn more, visit the MSK Replicator documentation, product page, pricing page, and this AWS blog post.

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.