Glue Data Catalog and Crawlers

What it is

A data catalog is a central index of what data exists and what it looks like. It records databases, tables, columns, data types, locations and partitions, but it does not store the data itself.

The AWS Glue Data Catalog is the managed catalog for AWS analytics. Athena, Redshift Spectrum, EMR, Glue jobs and Lake Formation all read the same definitions, so you describe a dataset once.

How it works

What the catalog holds

The catalog is organized as databases that contain tables. A table has a schema, a location such as an S3 prefix, the data format, the serialization library (SerDe) that reads it and a list of partitions. The catalog is compatible with the Apache Hive metastore, so EMR and other Hive aware tools can use it as their metastore.

Tables can point at S3, JDBC sources, DynamoDB, Kinesis and Kafka. The catalog stores metadata only, so dropping a table does not delete the files in S3 for external tables.

Crawlers

A crawler connects to a data store, samples the data, infers the schema and creates or updates tables in the catalog. Classifiers identify formats such as CSV, JSON, Parquet and Avro, and you can add custom classifiers when the built in ones guess wrong.

Crawlers also detect Hive style partitions in S3 paths and add them to the table. A crawler can run on S3 event notifications so it only looks at new data, which is cheaper than scanning a whole prefix every time.

Other ways to build the catalog

Crawlers are not the only way. You can write definitions yourself in DDL (data definition language), through the API, CloudFormation or Glue ETL jobs that update the catalog as they write. In Athena you create a table with CREATE EXTERNAL TABLE.

CREATE EXTERNAL TABLE orders (
  order_id bigint, amount double)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://my-bucket/curated/orders/';

MSCK REPAIR TABLE orders;

Keeping partitions current

New partitions in S3 are invisible to queries until the catalog knows about them. You have several choices.

• Run a crawler on a schedule or on new data.
• Run MSCK REPAIR TABLE in Athena to discover Hive style partitions.
• Use ALTER TABLE ADD PARTITION or a Glue API call from your ETL job.
• Use partition projection in Athena, which calculates partitions from rules in table properties, with no catalog entries to maintain.

Partition projection suits tables with predictable partitions such as dates, and it speeds up planning on tables with many partitions.

Schema evolution and the registry

Source data changes over time. Crawlers can update the table when columns are added, according to the schema change policy. For streaming data, the AWS Glue Schema Registry stores schemas for Avro, JSON Schema and Protocol Buffers and checks that new versions follow a compatibility rule, such as backward or forward compatible, before producers use them.

Connections, security and governance

A Glue connection stores how to reach a data source, such as a JDBC URL, network settings and the secret that holds the credentials. Catalog access is controlled with IAM and with Lake Formation, which can grant permissions down to table, column and row level. The catalog can be shared across accounts with Lake Formation or with AWS Resource Access Manager.

Data lineage and discovery are easier when tables carry descriptions and tags, and Lake Formation tags (LF tags) can then drive permissions.

Catalog statistics

Column statistics stored in the catalog can help engines plan queries. They are optional, and they matter most for large tables with many joins.