How Generative AI Models Work

What it is

A generative AI model creates new content, such as text, code or images, in response to a prompt. A large language model, or LLM, does this for text. It does not look answers up. It predicts what should come next, one small piece at a time, based on patterns learned from huge amounts of training data.

Knowing the parts of that process explains the settings you meet later: tokens, the context window and sampling parameters.

How it works

Tokens

Models do not read words. They read tokens, which are chunks of text. A token can be a whole short word, part of a longer word, a space or a punctuation mark. As a rough guide, one token is about three quarters of an English word, and other languages and code usually use more tokens for the same meaning.

Tokens matter because they are what you pay for, what the model's limits are measured in, and what the model generates one at a time.

Embeddings

An embedding is a list of numbers, called a vector, that captures the meaning of a piece of text, an image or other content. Texts with similar meaning have embeddings that are close together. Embedding models are used for searching by meaning, grouping similar items and finding relevant passages to give to a language model. They do not write text, and a vector cannot be turned back into the original words.

Transformers and attention

Modern language models use the transformer architecture. Its key feature is attention, which lets the model weigh how much each earlier token matters when it predicts the next one. That is how it keeps track of what a pronoun refers to, or which part of the prompt is the question.

Generating a response

1. The prompt is split into tokens.
2. The model scores every possible next token.
3. One token is chosen, using settings such as temperature and top p.
4. The new token is added to the sequence, and the process repeats until the answer ends, a stop sequence appears or the token limit is reached.

Because a token is chosen from a spread of likely options, the same prompt can give different answers. The choice is probabilistic.

The context window

The context window is the maximum number of tokens a model can handle in one request, counting the input and the output together. A model with a 4,096 token window cannot accept a 3,500 token prompt and also produce 1,000 tokens, because 4,500 is above the limit. If a conversation grows past the window, older content has to be dropped or summarised.

How models are built and improved

A model does not learn from your prompts. Each request starts from the same trained model, so you supply any new information in the prompt.

Types of model

• Chat or completion models generate and transform text.
• Reasoning models spend extra effort on multi step problems.
• Embedding models turn content into vectors.
• Multimodal models accept more than one kind of input, such as text and images, or text and audio.
• Image generation models create pictures from a text description.
• Small language models, such as Microsoft's Phi family, give good results at lower cost and size.

Limits to keep in mind

• Hallucination. The model can produce fluent statements that are false.
• Knowledge cutoff. It knows only what was in its training data, so it may not know recent events.
• Bias. It can reflect biases in its data.
• Inconsistency. Answers vary between runs.

Grounding reduces these problems by giving the model trusted data inside the prompt, for example documents retrieved from a search index. This approach is called retrieval augmented generation, or RAG.

A worked scenario

A support team wants a chatbot that answers questions about its own product manuals. A base model does not know those manuals, and it cannot learn them from a conversation. The team converts each manual section into an embedding, stores the vectors in a search index, and for every question retrieves the closest sections. Those sections are placed in the prompt together with an instruction to answer only from them. The model then produces a grounded answer, and the team tracks the token count so the prompt and the reply stay within the context window.