Our Blog

Run interactive workloads on Amazon EMR on EC2 with Spark Connect

Published
August 4, 2026
https://aws.amazon.com/about-aws/whats-new/2026/08/amazon-emr-ec2-spark-connect/

Amazon EMR on EC2 Interactive Spark Sessions with Spark Connect

Amazon EMR on EC2 now supports interactive Apache Spark sessions with Spark Connect, allowing data engineers and data scientists to develop and debug Apache Spark applications interactively from managed notebooks in Amazon SageMaker Unified Studio and their own IDEs, such as Jupyter and Visual Studio Code. Each session runs on dedicated EMR on EC2 clusters, with monitoring and debugging capabilities available in the EMR console.

An interactive session provides a persistent Spark context that spans across cells and scripts, enabling the blending of local Python code execution with remote Spark operations. Spark Connect's client-server architecture decouples the application client from the Spark driver, allowing you to maintain your preferred development environment and tooling while Spark infrastructure runs on the cluster.

This architecture supports workflows including ad hoc data exploration, iterative step-by-step debugging, and incremental PySpark job development before deploying to production. For observability, you get real-time session monitoring via the Spark UI, history tracking through the Spark History Server, and session management from the EMR console or API/CLI/SDK.

What to do

Source: AWS release notes




If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.

Follow our blog

Get the latest insights and advice on AWS services from our experts.

By clicking Sign Up you're confirming that you agree with our Terms and Conditions.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Related posts

Check out other insights from the team at West Loop Strategy.

View all
https://aws.amazon.com/about-aws/whats-new/2025/10/aws-customer-carbon-footprint-tool-scope-3-emissions-data

AWS’ Customer Carbon Footprint Tool now includes Scope 3 emissions data

Today, AWS’ Customer Carbon Footprint Tool (CCFT) has been updated to include Scope 3 emissions data and Scope 1 natural gas and refrigerants, providing AWS customers more complete visibility into their cloud carbon footprint
Read more
https://aws.amazon.com/about-aws/whats-new/2025/10/aws-end-user-messaging-onboarding-progress-alerts-slack-email-eventbridge-destination

AWS End User Messaging now sends onboarding progress alerts via Slack, Email, or any other EventBridge destination

Starting today, AWS End User Messaging customers can be notified of updates to their SMS onboarding progress in Slack, Email, or any other Amazon EventBridge destination
Read more
https://aws.amazon.com/about-aws/whats-new/2026/07/kinesis/on-demand-scale-down

Amazon Kinesis Data Streams now supports scaling down ingest capacity with warm throughput

Amazon Kinesis Data Streams is a serverless streaming data service that makes it easy to capture, process, and store data streams at any scale
Read more
View all