Amazon EMR on EKS now supports Apache Spark troubleshooting agent

Amazon EMR on EKS Spark Troubleshooting Agent

Amazon EMR on EKS now supports the Apache Spark troubleshooting agent, enabling data engineers to diagnose job failures through natural language, receiving automated root cause analysis and PySpark code recommendations.

The agent analyzes Spark History Server data, distributed executor logs, and cluster configurations to identify issues such as memory errors, data skew, resource contention, and connectivity failures. This launch extends the agent's coverage to all EMR deployment options: EMR on EC2, EMR Serverless, and EMR on EKS.

What to do

• Access the agent from the EMR on EKS console via the "Troubleshoot with AI" option on failed jobs.
• Use the agent through MCP with compatible AI coding agents like Kiro, Claude Code, and Cursor.
• Ensure all operations are read-only, authenticated with IAM roles, and logged in AWS CloudTrail.

The Spark troubleshooting agent is available in all AWS Regions where SageMaker Unified Studio is available. To get started, go to the EMR on EKS console or set up the MCP server in your preferred AI coding agent. For more details, see the EMR troubleshooting agent documentation.

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.