Collaborating within and across teams to manage data and models · Topic 4 of 4
Tracking Experiments, Evaluating Models and Lineage
Track runs with Experiments on Agent Platform, evaluate predictive and gen AI models with metrics and LLM-as-a-judge, and record lineage with ML Metadata.
- 5 min
- Experiments logs parameters and metrics for any framework; TensorBoard shows training curves.
- Agent Platform Pipelines runs Kubeflow Pipelines and TFX code serverlessly and tracks artifacts.
- Pick metrics by task and error cost; check slices, not just the overall score.
What it is
An experiment is a set of training runs that try different data, features, code or hyperparameters. Tracking records each run's parameters, metrics and artifacts, so you can compare runs and reproduce the winner.
Evaluation decides which run is best, using metrics that match the task. Lineage records which data, code and run produced each model, which teams need for audits and debugging.
How it works
Choosing the environment for experimentation
| Option | Best for |
|---|---|
| Experiments on Gemini Enterprise Agent Platform | Logging parameters and metrics from notebooks or training jobs and comparing runs side by side; works with any framework |
| Managed TensorBoard on the platform | Time-series training curves (loss per step) for TensorFlow, PyTorch and other frameworks |
| Gemini Enterprise Agent Platform Pipelines | Repeatable multi-step workflows written with the Kubeflow Pipelines SDK or TFX; runs are tracked and their artifacts recorded automatically |
| Kubeflow Pipelines on your own GKE cluster | Teams that must run open-source Kubeflow themselves, for portability or custom cluster control |
Choose by framework and maturity: ad-hoc scikit-learn or PyTorch runs from a notebook go to Experiments; TFX or Kubeflow Pipelines code runs on Agent Platform Pipelines; self-managed Kubeflow only when a managed service is ruled out.
from google.cloud import aiplatform
aiplatform.init(project="my-project", location="us-central1",
experiment="churn-exp")
aiplatform.start_run("xgb-depth6")
aiplatform.log_params({"max_depth": 6, "learning_rate": 0.1})
# ... train and evaluate the model ...
aiplatform.log_metrics({"roc_auc": 0.91, "log_loss": 0.28})
aiplatform.end_run()
runs = aiplatform.get_experiment_df("churn-exp") # one row per runEvaluating predictive models
| Task | Metrics | Use when |
|---|---|---|
| Classification | Precision, recall, F1, ROC AUC, PR AUC, log loss, confusion matrix | PR AUC and recall for rare positives; precision when false alarms are costly |
| Regression | RMSE, MAE, R squared | MAE is less sensitive to outliers than RMSE |
| Forecasting | MAE, RMSE, MAPE | MAPE is hard to read when actual values are near zero |
| Ranking and recommendation | Precision at k, NDCG | Order of results matters |
Evaluate on a test set the model never saw, and also on slices such as region or customer group, because a good overall score can hide a group that fails.
Evaluating gen AI solutions
Free text has no single correct answer, so gen AI evaluation combines methods.
| Method | How it works | Good for |
|---|---|---|
| Computation-based metrics | Compare output to a reference: exact match, BLEU, ROUGE | Translation, extraction, summaries with a reference answer |
| LLM-as-a-judge (model-based metrics) | A judge model scores each response against criteria or a rubric: fluency, groundedness, safety, instruction following | Open-ended answers without one reference |
| Pointwise | Score one response on a scale | Track a single model over time |
| Pairwise | Judge which of two responses is better | Compare a candidate model or prompt against a baseline |
| Human evaluation | Experts rate a sample | Calibrate the judge and high-risk use cases |
The platform's gen AI evaluation service runs these metrics over an evaluation dataset of prompts (and references where available) and can log results to Experiments, so prompt and model versions are compared like any other run.
Tracking artifacts, versions and lineage
Gemini Enterprise Agent Platform ML Metadata stores a graph of:
| Element | Example |
|---|---|
| Artifact | A dataset, a trained model, an evaluation result |
| Execution | A preprocessing step or training run |
| Context | A group such as an experiment or a pipeline run |
Pipeline runs record lineage automatically. From a model in Model Registry you can trace back to the run, its parameters and the exact data it used.
A worked example
Three data scientists tune a churn model in notebooks and paste results into a spreadsheet. Nobody can reproduce the best score. The engineer adds aiplatform.init(experiment=...) with start_run, log_params and log_metrics to the shared training code, so every run is recorded with its parameters and metrics. The best configuration then moves into a pipeline, which records lineage from dataset to model.
How the exam asks about this
- "Compare several model versions trained with different hyperparameters in notebooks, with the least effort." Log runs to Experiments on Agent Platform and compare them.
- "Auditors ask which dataset version trained the production model." Use the lineage recorded in ML Metadata by the pipeline run.
- "Compare two prompt versions of a summarizer that has no reference answers." Use a pairwise model-based metric (LLM-as-a-judge) in the gen AI evaluation service.
What to remember for the exam
- Experiments logs parameters and metrics for any framework; TensorBoard shows training curves.
- Agent Platform Pipelines runs Kubeflow Pipelines and TFX code serverlessly and tracks artifacts.
- Pick metrics by task and error cost; check slices, not just the overall score.
- Gen AI: computation-based metrics need references; LLM-as-a-judge handles open-ended output.
- ML Metadata records artifacts, executions and contexts for lineage.
Accuracy on an imbalanced dataset is misleading. A fraud model that never predicts fraud can be 99% accurate; use recall, precision and PR AUC.
Common mistakes
Tracking results by hand. Spreadsheets lose parameters and code versions; log runs to Experiments.
Using BLEU or ROUGE for open-ended answers. Overlap scores punish correct answers worded differently; use a model-based judge with clear criteria.
Self-managing Kubeflow without a reason. Agent Platform Pipelines runs the same SDK with no cluster to maintain.
Check yourself
What does pairwise evaluation answer?
Which of two responses, for example from a candidate and a baseline model, is better for the same prompt, as judged by a model or a human.
Which service records that a model was trained on a specific dataset version?
Gemini Enterprise Agent Platform ML Metadata, which pipeline runs populate with artifacts, executions and lineage.
Why evaluate on data slices?
A strong overall metric can hide poor performance for one group, such as a region or customer segment.
Topic 4 of 4 in Collaborating within and across teams to manage data and models
Test yourself on Collaborating within and across teams to manage data and models
The full Google Cloud Professional Machine Learning Engineer question bank has 749 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.