Amazon EMR introduces Long Term Support with Apache Spark 4.1

Amazon EMR Long Term Support (LTS) Releases
Amazon EMR introduces Long Term Support (LTS) releases, starting with emr-spark-8.1.0 and Apache Spark 4.1. LTS provides 36 months of support for designated versions of the AWS runtime for Apache Spark, including critical and high severity security, bug, and data-corruption fixes.
New Features
- Apache Iceberg v3 Support: Full support for Apache Iceberg v3, including geospatial, high-precision timestamp, and schema-evolution capabilities.
- Cross-Account Catalogs: Spark SQL queries can reference catalogs by name, including cross-account and Amazon S3 Tables catalogs, automatically detecting Apache Iceberg, Delta Lake, and Apache Hudi table formats.
- Fine-Grained Access Control: Expanded fine-grained access control for more Apache Iceberg operations and Delta Lake VACUUM operation, allowing column-level and row-level permissions for a wider set of jobs.
- Spark Connect Endpoints: Amazon EMR on EKS clusters now support Spark Connect endpoints with token-based authentication.
What to do
- Create an EMR cluster or application with emr-spark-8.1.0 from the AWS Management Console.
- Use the Apache Spark Upgrade Agent for Amazon EMR to move existing applications to the release.
Source: AWS release notes
If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.



