Amazon EMR now supports native data quality checks in Apache Spark jobs

Amazon EMR Data Quality

Amazon EMR now validates data quality natively within Apache Spark jobs. Data engineering teams can catch quality issues inline before bad data reaches downstream systems, without building or maintaining separate validation infrastructure.

Amazon EMR Data Quality, powered by AWS Glue, is available at no additional cost and uses the same declarative rules language (Data Quality Definition Language) and APIs as AWS Glue for a consistent experience across both engines, and works with Apache Iceberg, Apache Hudi, and Delta Lake across EC2, EKS, and Serverless deployment modes.

You can author validation rules in your Spark scripts or reference rules stored in the AWS Glue Data Catalog, and then apply them at any stage of your pipeline. With 30+ rule types, including dynamic rules that compare against historical runs and ML-based anomaly detection, Amazon EMR Data Quality provides the flexibility of choosing the validation framework that fits your needs.

You can take actions like failing a pipeline, filtering invalid records, or routing them to quarantine based on the quality evaluation results. Results are produced at the dataset, rule, and row level, so that you can easily identify which records or rules are driving the quality failures.

What to do

• Visit Amazon EMR Data Quality documentation to get started.
• Add DQDL rules for data quality validation to the Spark scripts of your EMR workloads.

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.