Basics of Deep Learning and Foundation Models · Topic 3 of 3

Transformer and Foundation Model Infrastructure

Learn self-attention, multi-head attention, positional encoding, BERT and GPT, pre-training, fine tuning and LoRA, and compute behind foundation models.

6 min
Includes a diagram

In a nutshell

Attention = softmax(QK^T / sqrt(d_k)) V; self-attention cost grows with the square of sequence length.
Multi-head attention runs several heads in parallel; positional encoding supplies word order.
BERT is encoder-only, GPT is decoder-only, T5 is encoder-decoder.

What it is

The Transformer is a neural network architecture, introduced in the 2017 paper "Attention Is All You Need", that processes a whole sequence at once using self-attention instead of recurrence. It trains in parallel and captures long-range dependencies well.

Foundation models are very large Transformers pre-trained on huge data and then adapted to many tasks. Their infrastructure is the compute, storage, networking and distributed training methods needed to build and serve them.

How it works

Self-attention

Each token's vector is projected into a query (Q), a key (K) and a value (V). A token's query is compared with every token's key to decide how much attention to pay to each one.

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

Dividing by sqrt(d_k) keeps scores from growing too large, which would push softmax into tiny gradients. With d_k = 64 the scores are divided by 8.

Every token attends to every other token, so cost grows with the square of the sequence length. This is why long context windows are expensive.

Multi-head attention

Multi-head attention runs several attention operations in parallel, each with its own Q, K and V projections, then concatenates them. Different heads can learn different relations, such as syntax or reference.

The original Transformer used a model width of 512 with 8 heads, so each head works in 512 / 8 = 64 dimensions.

from torch import nn

attn = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
out, weights = attn(x, x, x)             # self-attention: query, key, value all x

Positional encoding

Attention by itself ignores word order. Positional encoding adds position information to each token embedding; the original paper used fixed sine and cosine functions, and many later models use learned or rotary position methods.

Transformer block

Transformer encoder and decoder: each encoder block has self-attention and a feed-forward network, each with add and normalize; each decoder block has masked self-attention, cross-attention to the encoder output and a feed-forward network, then linear and softmax predict the next token.

Part	Role
Multi-head self-attention	Mixes information between tokens
Feed-forward network	Processes each token independently
Residual connection plus layer normalization	Around each sublayer; keeps deep stacks trainable
Masked self-attention (decoder)	Each token sees only earlier tokens, for generation
Cross-attention (decoder)	Decoder attends to encoder outputs

Three model families

Family	Example	Typical use
Encoder-only	BERT	Understanding: classification, entity recognition; trained by masked language modeling
Decoder-only	GPT series, most chat LLMs	Generation; trained to predict the next token
Encoder-decoder	T5, the original Transformer	Sequence to sequence, such as translation and summarization

From pre-training to use

Pre-training is self-supervised: the labels come from the text itself, such as the next token. Fine tuning adapts the model to tasks with much less data.

Fine tuning approach	What changes
Full fine tuning	All weights updated; highest cost
Parameter-efficient fine tuning (PEFT)	Only a small set of added or chosen parameters updated
LoRA	Freezes W and learns a low-rank update B x A, with rank r much smaller than the layer width

Large models are often reported to show emergent abilities, and they support in-context learning: they follow examples given in the prompt without any weight change.

Infrastructure for foundation models

Layer	What it provides
Compute	Large clusters of AI accelerators, such as GPUs or Huawei Ascend NPUs
Interconnect	High-bandwidth links between accelerators and servers, because parallel training exchanges huge amounts of data
Storage	Fast, large storage for training data and checkpoints
Software	Frameworks (PyTorch, MindSpore), distributed training libraries, inference engines
Data	Collected, cleaned and deduplicated corpora

A model too large for one device needs distributed training.

Parallelism	What is split
Data parallelism	Each device holds a full model copy and processes different batches; gradients are averaged
Tensor (model) parallelism	A single layer's matrices are split across devices
Pipeline parallelism	Different layers run on different devices, with micro-batches flowing through
Expert parallelism	Experts of an MoE model placed on different devices

Mixed precision (FP16, BF16, even FP8) cuts memory and speeds training. For inference, quantization, distillation and the KV cache reduce cost and latency.

A worked example

A 7 billion parameter model stored in FP16 uses 2 bytes per parameter: 7 x 10^9 x 2 = 14 GB for the weights alone. Training needs far more, for gradients, optimizer states and activations, so it is spread across many devices.

With LoRA on a 4096 x 4096 weight matrix, full tuning would update 16,777,216 values. LoRA with r = 8 learns A (8 x 4096) and B (4096 x 8), which is 65,536 values, about 0.39%.

How the exam asks about this

"Why does the Transformer train faster than an RNN?" → Self-attention processes all tokens in parallel instead of one step at a time.
"Which model family is BERT?" → Encoder-only, pre-trained with masked language modeling.
"In which parallelism does each device hold a full copy of the model?" → Data parallelism.

What to remember for the exam

Attention = softmax(QK^T / sqrt(d_k)) V; self-attention cost grows with the square of sequence length.
Multi-head attention runs several heads in parallel; positional encoding supplies word order.
BERT is encoder-only, GPT is decoder-only, T5 is encoder-decoder.
Pre-training is self-supervised on large unlabeled data; fine tuning adapts it; LoRA trains a small low-rank update.
Infrastructure: accelerator clusters, fast interconnect and storage; data, tensor, pipeline and expert parallelism.

Watch out

The Transformer has no built-in sense of order. Without positional encoding, "dog bites man" and "man bites dog" look the same to self-attention.

Common mistakes

Thinking pre-training needs labeled data. Pre-training is self-supervised. The text itself provides the targets, such as the next or masked token.

Confusing data and model parallelism. Data parallelism copies the whole model and splits the batches. Model (tensor or pipeline) parallelism splits the model itself.

Thinking LoRA changes the original weights during training. The base weights stay frozen. Only the small A and B matrices are trained, and they can be merged afterward.

Check yourself

Why is the dot product divided by sqrt(d_k) in attention?
Large dot products push softmax toward extreme values with tiny gradients. Scaling by sqrt(d_k) keeps the scores in a stable range.
What does masked self-attention do in a decoder?
It stops each token from attending to later tokens, so the model can only use earlier context when predicting the next token.
How many bytes do the weights of a 7 billion parameter model take in FP16?
About 14 GB, since each parameter uses 2 bytes in FP16.

Topic 3 of 3 in Basics of Deep Learning and Foundation Models

Test yourself on Basics of Deep Learning and Foundation Models

The full Huawei HCIA-AI H13-311 question bank has 600 questions, with timed practice exams and an explanation for every answer.

Get Full Access Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Huawei. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.