AWS Glue Data Quality now supports pre-processing queries

AWS Glue Data Quality Preprocessing Queries

AWS announces the general availability of preprocessing queries for AWS Glue Data Quality, allowing you to transform data before running data quality checks via AWS Glue Data Catalog APIs. This feature enables creating derived columns, filtering data, performing calculations, and validating relationships between columns within your data quality evaluation process.

Preprocessing queries enhance flexibility for complex data quality scenarios requiring data transformation before validation. You can create derived metrics, limit the number of columns for data quality recommendations, or filter datasets to focus quality checks on specific data subsets. This capability streamlines your data quality workflows by eliminating the need for separate data pre-processing steps.

What to do

• Use AWS Glue Data Catalog APIs to implement preprocessing queries.
• Explore the Glue Data Quality documentation for more details.

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.