Programming Concepts and IaC

What it is

A data engineer writes code, but also packages, tests and deploys it in a repeatable way. This topic covers the programming habits that the exam expects, not deep software design.

The key ideas are treating infrastructure and pipeline definitions as code, storing them in version control, and releasing them through an automated pipeline.

How it works

Infrastructure as code

Infrastructure as code (IaC) describes resources in files, so an environment can be created, reviewed and rebuilt the same way each time.

A short SAM style template for a Lambda function looks like this.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  TransformFn:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: app.handler
      Runtime: python3.12
      MemorySize: 1024
      Timeout: 300

Stacks support change sets, which preview changes before applying them, and rollback on failure. Parameters and separate stacks per environment keep development and production apart.

CI/CD for data pipelines

Continuous integration (CI) builds and tests every change. Continuous delivery (CD) deploys tested changes through stages. On AWS, AWS CodePipeline orchestrates the stages, AWS CodeBuild runs builds and tests, and AWS CodeDeploy handles some deployment targets, including Lambda.

Store pipeline code, SQL and templates in a Git repository. Common Git actions are git clone, git branch, git commit, git push, git merge and pull requests for review. Teams use branches per feature and merge after review and passing tests.

Lambda for data work

Lambda runs code in response to events without servers. Useful settings for data jobs are memory (which also scales CPU), timeout, and concurrency controls. A Lambda function can run up to 15 minutes.

Lambda layers share libraries between functions, and container images allow larger dependencies. Versions and aliases let you release a new version gradually and roll back. Environment variables hold configuration, while secrets belong in Secrets Manager.

Storage and data APIs in Lambda

Each function has a /tmp directory that is ephemeral and configurable up to 10 GB. For shared or persistent files, mount an Amazon EFS (Elastic File System) file system.

The function must be attached to a VPC, with a mount target reachable from its subnets and a security group that allows NFS. Mounting goes through an EFS access point, and the local mount path must sit under /mnt.

aws lambda update-function-configuration --function-name transform-fn \
  --file-system-configs Arn=arn:aws:elasticfilesystem:us-east-1:123456789012:access-point/fsap-0abc1234,LocalMountPath=/mnt/data

To serve data to other systems, put Amazon API Gateway in front of a Lambda function (or, for simple DynamoDB reads and writes, a direct service integration). The function can run a DynamoDB query, start an Athena query, or call the Redshift Data API. Athena and Redshift answers are not instant, so return a query ID that the client polls, and use pagination tokens for large results.

SQL skills

The exam expects practical SQL: joins, aggregations with GROUP BY, window functions and common table expressions (CTEs). Performance habits matter as much as correctness.

SELECT customer_id,
       order_date,
       SUM(amount) OVER (PARTITION BY customer_id ORDER BY order_date) AS running_total
FROM orders
WHERE order_date >= DATE '2026-01-01';

Select only the columns you need, filter on partition columns, and avoid joining huge tables without a filter. In Redshift, EXPLAIN shows the plan, and sort and distribution keys affect the plan.

Distributed computing and data structures

Spark splits data into partitions and processes them in parallel on many workers. Wide operations such as joins and group by cause a shuffle, where data moves between workers, which is expensive. Reduce shuffles by filtering early and broadcasting small tables.

On data structures, know the idea behind arrays, lists, hash maps, queues, stacks, trees and graphs. A hash map gives fast lookup by key, a queue gives first in first out order, and a graph models relationships such as lineage or networks.

Testing and packaging

Unit test transformation logic with sample data before deploying. Use the SAM command line tool to test Lambda locally. Keep configuration separate from code so the same artifact moves from test to production.

sam build
sam deploy --guided