Announcing the Apache Spark upgrade agent for Amazon EMR

Apache Spark Upgrade Agent for Amazon EMR

AWS introduces the Apache Spark upgrade agent, a tool that accelerates Apache Spark version upgrades for Amazon EMR on EC2 and EMR Serverless. This agent transforms lengthy upgrade processes into weeks-long projects through automated code analysis and transformation.

Engineers can express upgrade requirements in natural language while maintaining full control over code modifications. The agent automatically identifies API changes and behavioral modifications across PySpark and Scala applications. Upgrades can be initiated from SageMaker Unified Studio, Kiro CLI, or any IDE with MCP compatibility.

The agent suggests specific changes, which engineers can review and approve before implementation. It validates functional correctness through data quality validations and supports upgrades from Spark 2.4 to 3.5.

What to do

• Visit SageMaker Unified Studio and select IDE Spaces or install the Kiro CLI to start using the agent.
• Review the documentation for detailed implementation guidance, reference documentation, and migration examples.

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.