AWS Glue Interactive Sessions now support Spark Connect for interactive workloads

AWS Glue Interactive Sessions Update

AWS Glue Interactive Sessions now support Apache Spark Connect, enabling development and execution of Apache Spark applications from preferred environments such as managed notebooks in Amazon SageMaker Unified Studio, Jupyter, Visual Studio Code, and other IDEs. This is achieved without managing clusters, leveraging AWS Glue's serverless infrastructure.

With Spark Connect, Spark jobs are submitted to AWS Glue Interactive Sessions using a thin client architecture, decoupling the client application from the Spark execution environment. This allows for ad hoc data exploration, iterative debugging, and incremental PySpark job development before production deployment, all within familiar tools.

Spark Connect also enhances stability and simplifies upgrades by isolating client dependencies from the server-side Spark runtime. Observability is improved with real-time session monitoring via the Spark UI, history tracking through the Spark History Server, and session management using the AWS Glue API, CLI, or SDK.

What to do

• Connect to Glue Interactive Sessions using Spark Connect from notebooks in Amazon SageMaker Unified Studio, your favorite IDE, or the AWS API, SDK, and CLI.
• Explore the new capabilities and workflows enabled by Spark Connect.
• Visit the AWS Glue Interactive Sessions documentation for more information.

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.