AWS Glue announces catalog federation for remote Apache Iceberg catalogs

AWS Glue Catalog Federation for Remote Iceberg Catalogs

AWS Glue has introduced the general availability of catalog federation for remote Iceberg catalogs, enabling direct and secure access to Iceberg tables stored in Amazon S3 and cataloged in remote catalogs using AWS analytics engines.

With catalog federation, you can query remote Iceberg tables using your preferred AWS analytics engines without moving or copying tables. It synchronizes metadata in real-time across AWS Glue Data Catalog and remote catalogs, ensuring query results are always up-to-date. This feature supports a wide variety of analytics engines, including Amazon Redshift, Amazon EMR, Amazon Athena, AWS Glue, Apache Spark, and Amazon SageMaker with serverless notebooks.

Catalog federation uses AWS Lake Formation for access controls, allowing fine-grained access controls, cross-account sharing, and trusted identity propagation. It integrates with catalog implementations that support the Iceberg REST specifications.

This feature is available in the Lake Formation console and via AWS Glue and Lake Formation SDKs and APIs. It is generally available in all AWS commercial regions where AWS Glue and Lake Formation are available. You can federate to remote catalogs, discover databases and tables, grant permissions, and query remote Iceberg tables with just a few clicks in the console.

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.