Training and Scoring with Feature Tables

What it is

Once features live in a Unity Catalog feature table, you do not join them by hand. You describe which features to look up for each row, the Feature Engineering client builds the training DataFrame, and the logged model remembers those lookups.

At scoring time you pass only the keys. The model fetches the same features from the feature table, so training and scoring use identical feature logic. Real-time serving reads the same features from an online store instead.

How it works

Building a training set

Start from a DataFrame with the lookup key and the label. Each FeatureLookup names a feature table, the key that joins it, and the features to take.

from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup
fe = FeatureEngineeringClient()
lookups = [FeatureLookup(table_name="ml.retail.customer_features",
    feature_names=["orders_30d", "avg_basket"], lookup_key="customer_id")]
training_set = fe.create_training_set(df=labels_df,
    feature_lookups=lookups, label="churned", exclude_columns=["customer_id"])
train_df = training_set.load_df()

load_df returns the label DataFrame joined with the looked up features. exclude_columns drops columns you do not want as model inputs, such as the key itself.

Other useful FeatureLookup options are feature_names=None (take all features except the keys), output_name to rename a feature, default_values for keys with no match, and timestamp_lookup_key for point in time lookups on time series feature tables.

Training and logging the model

Train on train_df as usual, then log with the Feature Engineering client rather than plain MLflow.

import mlflow
fe.log_model(model=model, artifact_path="model",
    flavor=mlflow.sklearn, training_set=training_set,
    registered_model_name="ml.retail.churn_model")

Passing training_set packages the feature metadata with the model and records lineage from the feature tables to the model in Unity Catalog. That metadata is what makes automatic lookup possible later.

Scoring with feature lookups

score_batch takes a model URI and a DataFrame that only needs the lookup keys.

preds = fe.score_batch(
    model_uri="models:/ml.retail.churn_model@champion",
    df=customer_ids_df)

The client looks up the features from the feature table, runs the model, and returns the input DataFrame plus the feature columns and a prediction column.

Online versus offline feature tables

You publish an offline table to an online store with fe.publish_table, which keeps it synced in a triggered, continuous or snapshot mode. A served model logged with fe.log_model then finds its features in the online store automatically when a request arrives with just the keys.

Worked example

A model uses 12 features from two feature tables. A nightly job scores 5 million customers: it calls score_batch against the offline tables. A mobile app needs a prediction for one customer in under 100 ms: the request hits a serving endpoint, which reads the 12 features for that one key from the online store.

How the exam asks about this

• "Batch scoring must use exactly the features used in training, with only customer_id supplied." Log with fe.log_model and the training set, then call fe.score_batch.
• "A real-time endpoint needs feature values in milliseconds." Publish the feature table to an online store; the offline Delta table is too slow for per request lookups.
• "Which call returns a Spark DataFrame of labels joined with features?" training_set.load_df() after fe.create_training_set.