Amazon EMR Serverless eliminates local storage provisioning for Apache Spark workloads

Amazon EMR Serverless Serverless Storage

Amazon EMR Serverless now offers serverless storage that eliminates local storage provisioning for Apache Spark workloads, reducing data processing costs by up to 20% and preventing job failures from disk capacity constraints.

You no longer need to configure local disk type and size for each application. EMR Serverless automatically handles intermediate data operations such as shuffle with no local storage charges. You pay only for compute and memory resources your job consumes.

EMR Serverless offloads intermediate data operations to a fully managed, auto-scaling serverless storage that encrypts data in transit and at rest with job-level isolation. Serverless storage decouples storage from compute, allowing Spark to release workers immediately when idle rather than keeping workers active to preserve temporary data. It eliminates job failures from insufficient disk capacity and reduces costs by avoiding idle worker charges.

This feature is particularly valuable for jobs using dynamic resource allocation, such as recommendation engines processing millions of customer interactions, where initial stages process large datasets with high parallelism then narrow as data aggregates.

What to do

• Visit the serverless storage for EMR Serverless documentation to get started.
• Check Supported AWS Regions for availability.

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.