Our Blog

Run interactive workloads on Amazon EMR on EKS with Spark Connect

Published
September 24, 2026
https://aws.amazon.com/about-aws/whats-new/2026/09/emr-eks-spark-connect-interactive/

Amazon EMR on EKS Interactive Spark Sessions

Amazon EMR on EKS now supports interactive Apache Spark sessions with Spark Connect. Data engineers and data scientists can develop and debug Apache Spark applications interactively from managed notebooks in Amazon SageMaker Unified Studio and their own IDEs, such as Jupyter and Visual Studio Code, with Spark running on the Amazon EKS clusters they already operate.

An interactive session provides a persistent Spark context that spans across cells and scripts, letting you blend local Python code execution with remote Spark operations. Spark Connect's client-server architecture decouples your application client from the Spark driver and allows you to maintain your preferred development environment and tooling while Spark runs on your Amazon EKS cluster. This architecture supports workflows including ad hoc data exploration and incremental PySpark job development before deploying to production. Each session runs as pods on a virtual cluster, secured with your AWS Identity and Access Management (IAM) execution role and tagged by project and user.

Spark Connect on Amazon EMR on EKS is available with EMR release 7.14 (Apache Spark 3.5) and emr-spark-8.1.0 (Apache Spark 4.1), in all AWS Commercial Regions. The Amazon SageMaker Unified Studio experience is available in supported AWS Regions.

What to do

Source: AWS release notes




If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.

Follow our blog

Get the latest insights and advice on AWS services from our experts.

By clicking Sign Up you're confirming that you agree with our Terms and Conditions.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Related posts

Check out other insights from the team at West Loop Strategy.

View all
https://aws.amazon.com/about-aws/whats-new/2025/12/security-hub-near-real-time-risk-analytics/

AWS Security Hub is now generally available with near real-time risk analytics

Amazon Web Services (AWS) announces the general availability of AWS Security Hub, a unified cloud security solution that prioritizes your critical security issues and helps you respond at scale, reduce security risks, and improve team productivity
Read more
https://aws.amazon.com/about-aws/whats-new/2025/11/aws-glue-additional-sap-entities-zero-etl-integration-sources

AWS Glue supports additional SAP entities as zero-ETL integration sources

AWS Glue now supports full snapshot and incremental load ingestion for new SAP entities using zero-ETL integrations
Read more
https://aws.amazon.com/about-aws/whats-new/2026/04/aws-iot-israel-tel-aviv-europe-milan/

AWS IoT is now available in Israel (Tel Aviv) and Europe (Milan) AWS Regions

AWS IoT Core and AWS IoT Device Management services are now available in the Israel (Tel Aviv) and Europe (Milan) AWS Regions
Read more
View all