Run Interactive Workloads on Amazon EMR Serverless with Spark Connect

Published

June 9, 2026

Amazon EMR Serverless Updates

Amazon EMR Serverless now supports interactive sessions with Spark Connect, allowing development and execution of Apache Spark applications from managed notebooks in Amazon SageMaker Unified Studio, Jupyter, and Visual Studio Code. Monitor and debug sessions in the EMR console, with granular cost and usage visibility.

Interactive sessions provide a persistent Spark context across cells and scripts, blending local Python code with remote Spark operations. Spark Connect's client-server architecture decouples the application client from the Spark driver, enabling preferred development environments while Spark infrastructure runs on EMR Serverless. This architecture supports workflows like ad hoc data exploration, debugging, and incremental PySpark job development.

For observability, real-time session monitoring is available via the Spark UI, history tracking through the Spark History Server, and session management from the EMR console or API/CLI/SDK.

Spark Connect on Amazon EMR Serverless is available with EMR release 7.13 in all AWS Regions where Amazon EMR Serverless is available. The SageMaker Unified Studio experience is available in supported regions.