Fine-Tuning Methods, Synthetic Data and Production Deployment

What it is

Fine-tuning continues training a model on your examples so it learns a style, format or behaviour. It is one of three levers, and the most expensive to maintain, so use the cheaper ones first. Synthetic data is training data generated by a stronger model (and then filtered) when real examples are scarce. A tuned model is a versioned asset that moves from development to production like any other.

How it works

Prompting, RAG or fine-tuning

Fine-tuning changes how a model behaves, not reliably what it knows.

Advanced methods

Data formats are JSONL. SFT for chat models:

{"messages": [{"role": "system", "content": "You are a support agent. Answer in two sentences."}, {"role": "user", "content": "How do I reset my token?"}, {"role": "assistant", "content": "Open the Security portal and choose Reset token. A new token is sent within a minute."}]}

DPO:

{"input": {"messages": [{"role": "user", "content": "Summarize this ticket."}]}, "preferred_output": [{"role": "assistant", "content": "Customer cannot sign in after a password reset."}], "non_preferred_output": [{"role": "assistant", "content": "The ticket is about a thing."}]}

A training job takes a training file, an optional validation file, and hyperparameters such as n_epochs, batch_size and learning_rate_multiplier:

training = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune")
validation = client.files.create(file=open("val.jsonl", "rb"), purpose="fine-tune")

job = client.fine_tuning.jobs.create(
    model=base_model,
    training_file=training.id,
    validation_file=validation.id,
    method={"type": "supervised",
            "supervised": {"hyperparameters": {"n_epochs": 3}}},
    extra_body={"trainingType": "GlobalStandard"},
)

Here client comes from AIProjectClient(...).get_openai_client(), and base_model supports fine-tuning for the method you choose. For DPO use method={"type": "dpo"}. The training type is Standard (region), Data zone, Global (cheaper, no residency guarantee) or Developer (idle capacity, can be preempted). Foundry User is enough to train; deploying the tuned model also needs Foundry Owner or a role that can write deployments. Quality beats quantity.

Synthetic data

1. Write seed examples by hand that show exactly the target behaviour.
2. Use a stronger teacher model to generate many variations (new inputs and ideal responses), prompting for diversity across topics, lengths and difficulty.
3. Filter: validate the format, remove duplicates, drop low quality items with a judge model or rules, check for personal data and for accidental copies of test cases.
4. Mix with real examples when you have them, and keep the evaluation set real and untouched.
5. Version and document the data: generator model and prompt, date, filters and counts, stored in Git or as a data asset.

Check the teacher model's terms of use before training on its outputs. Synthetic data can repeat the teacher's mistakes and be too uniform, so sample and read it.

Monitor and optimize the tuned model

Training produces metrics: training loss, validation loss and token accuracy. Validation loss rising while training loss falls means overfitting. Remedies: fewer epochs, more or more varied data, or a lower learning rate multiplier. Then:

• Evaluate the tuned model against the base model (with the best prompt) on the same held out set, using task, quality and safety evaluators.
• Check for regression on general abilities and safety, since tuning can weaken them.
• Compare cost and latency: a tuned small model with a short prompt may beat a large model.
• Do error analysis on failures, add targeted examples, and retrain; this loop is normal.
• In production, run continuous evaluation, track input drift, and re-tune when behaviour or data changes.

Development to production

Tuned models carry hosting charges (the Developer type has no hourly hosting fee), so delete deployments you do not use. Cancelling a training job does not delete a deployment or stop its charges. Automate retraining and deployment in a pipeline (GitHub Actions) with the same evaluation gates as for prompts.

A worked example

A claims team needs summaries in a strict template. They hand write 40 seed examples, generate 2,000 more with a teacher model, filter to 1,400, and add 300 real examples. After SFT with three epochs, validation loss flattens, template compliance rises from 71 to 97 percent, and a smaller model replaces the larger one with a shorter prompt. The tuned deployment gets 10 percent of traffic first, with the old one kept for rollback.

How the exam asks about this

• "Teach consistent output structure; knowledge changes weekly." Fine-tune for structure and use RAG for knowledge.
• "Model outputs must follow reviewer preferences between two answers." DPO with preferred and non preferred outputs.
• "Validation loss rises while training loss falls." Overfitting; reduce epochs or add data.