Data Operations and Support · Topic 6 of 6
Ensuring Data Quality
Define and enforce data quality on AWS with Glue Data Quality DQDL rules, DataBrew profiling, validation checks, quarantine and duplicate handling.
- 5 min
- Glue Data Quality uses DQDL rules and can run on catalog tables or inside ETL jobs.
- Rules can recommend from data, and results go to CloudWatch and EventBridge.
- DataBrew profile jobs and rulesets check completeness, duplicates and outliers.
What it is
Data quality describes whether data is fit for its use. Bad data leads to wrong reports and wrong decisions, so pipelines should check data automatically and act on the result.
The common dimensions are completeness, consistency, accuracy, validity, uniqueness and timeliness. Each can be turned into a rule that a job can test.
How it works
Quality dimensions
| Dimension | Question | Example rule |
|---|---|---|
| Completeness | Are required values present? | customer_id is never null |
| Uniqueness | Are keys free of duplicates? | order_id is unique |
| Validity | Do values match allowed formats or ranges? | amount is above zero |
| Consistency | Do related fields and sources agree? | Totals match between systems |
| Timeliness | Did the data arrive on time? | Latest record is under a day old |
| Accuracy | Do values match reality? | Postcode exists for the stated city |
AWS Glue Data Quality
AWS Glue Data Quality lets you write rules in Data Quality Definition Language (DQDL) and run them against Data Catalog tables or inside Glue ETL jobs. It can recommend rules from your data, and it produces a score and a result per rule. Rules cover things such as row counts, completeness, uniqueness and column value checks.
Rules = [
RowCount > 0,
IsComplete "order_id",
IsUnique "order_id",
ColumnValues "amount" > 0,
Completeness "customer_id" > 0.98
]Inside a Glue ETL job, a quality evaluation step can stop the job on failure, or separate rows that pass from rows that fail, so bad records go to a quarantine location. Results can be published to CloudWatch and EventBridge, which lets you alarm and start follow up work.
AWS Glue DataBrew
DataBrew profile jobs analyze a dataset and report missing values, distinct counts, outliers and value distributions. You can define data quality rulesets in DataBrew and validate a dataset against them. It suits analysts who want a visual way to explore quality before building a pipeline.
Other validation techniques
Validation can sit in many places.
- At ingestion: schema checks in the Glue Schema Registry, and Lambda validation before Firehose delivery.
- During transformation: Spark filters, type casting and deduplication in Glue or EMR.
- In the warehouse: constraints are informational in Redshift, so check uniqueness and nulls with queries instead.
- In streaming: Managed Service for Apache Flink can filter or route invalid events.
Deequ is an open source library from AWS Labs, built on Spark, for defining and measuring quality checks. Glue Data Quality builds on it, and you can also run Deequ yourself on EMR.
Handling bad data
When a record fails a rule you have four choices.
| Choice | Use when |
|---|---|
| Reject and stop the pipeline | The error is severe and everything downstream depends on it |
| Quarantine the record | Keep the good data flowing and review bad rows later |
| Correct automatically | The fix is safe, such as trimming spaces or applying defaults |
| Flag and continue | The rule is advisory, and consumers should see a quality flag |
Send quarantined rows to a separate S3 prefix with the reason, so engineers can fix and replay them.
Duplicates, nulls and schema drift
Handle duplicates by deduplicating on a business key, with the latest record winning. Treat nulls deliberately: reject them, fill with defaults, or keep them and document the meaning. Schema drift, where a source adds or changes a field, should be caught by schema checks before it breaks downstream tables.
Sampling and skew
Profiling a sample of a large dataset gives quick insight, but a small random sample can miss rare bad values. For skew, check the distribution of keys, because a handful of values with most rows can cause both quality and performance problems.
Quality as a process
Run checks at each zone boundary, store results over time, alert on trends and assign an owner for each dataset. Quality scores that fall slowly often show a source problem earlier than a hard failure does.
What to remember for the exam
- Glue Data Quality uses DQDL rules and can run on catalog tables or inside ETL jobs.
- Rules can recommend from data, and results go to CloudWatch and EventBridge.
- DataBrew profile jobs and rulesets check completeness, duplicates and outliers.
- Common dimensions are completeness, uniqueness, validity, consistency, accuracy and timeliness.
- Quarantine bad records to a separate S3 location instead of dropping them silently.
- Redshift primary key constraints are informational, so enforce uniqueness yourself.
- Deduplicate on a business key, and decide a null policy.
- Catch schema drift early with a schema registry or catalog checks.
When the question wants bad rows kept for review while good rows continue, the answer is quarantine, not stop or delete.
Common mistakes
Trusting Redshift constraints. They help the optimizer but do not block duplicates.
Dropping bad rows silently. You lose evidence and cannot replay the fix.
Checking quality only at the end. Errors found late are expensive to trace.
Using a tiny sample as proof. Rare errors hide in samples, so run full rules on critical datasets.
A quality rule that fails the whole job on every minor issue trains teams to disable the check, so set severity to match impact.
Check yourself
What language does Glue Data Quality use for rules?
Data Quality Definition Language (DQDL).
Where should failed records go if good data must keep flowing?
To a quarantine location, such as a separate S3 prefix, with the reason recorded.
Does a Redshift primary key stop duplicate rows?
No, the constraint is informational, so you must check uniqueness yourself.
Topic 6 of 6 in Data Operations and Support
Test yourself on Data Operations and Support
The full AWS DEA-C01 question bank has 845 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.