Imputing Missing Values

What it is

Imputation replaces missing values with an estimate so that rows are not thrown away and models that cannot accept nulls can still train. The simplest estimates are one value per column: the mean, the median or the mode (the most frequent value).

Which one is right depends on the column. You choose it after looking at the column's type and distribution, not by a fixed rule.

How it works

Mean, median and mode compared

A small worked example shows the difference. For incomes of 10, 12, 14, 15 and 200 (in thousands), the mean is 50.2 while the median is 14.

Filling the gaps with 50.2 gives every missing row an income higher than four of the five real values. The median, 14, is much closer to a typical row, which is why skewed data calls for the median.

What every single value imputation does

All three strategies put the same value into every missing row. That shrinks the column's variance and weakens its correlation with other columns, more so as the share of missing values grows.

A common companion step is an indicator column, such as income_missing = 1 where the value was null. The model can then learn whether missingness itself carries a signal, for example customers who refused to state an income.

When a large share of a column is missing, a single filled value carries little information. Consider dropping the column or using a model based imputation instead.

Imputing with Spark ML's Imputer

pyspark.ml.feature.Imputer is an estimator. fit() computes the mean, median or mode of each input column and returns an ImputerModel, a transformer that fills the gaps.

from pyspark.ml.feature import Imputer
imputer = Imputer(strategy="median",
                  inputCols=["age", "income"],
                  outputCols=["age_imp", "income_imp"])
model = imputer.fit(train_df)
train_out = model.transform(train_df)
test_out = model.transform(test_df)

Because Imputer is an estimator, it fits naturally as a stage in a Pipeline. Fit it on the training data and reuse the fitted model for validation, test and scoring data, so the filled value never comes from rows the model is evaluated on.

Imputing a string column with the mode

Imputer rejects string columns, so for a categorical text column compute the mode yourself and fill with fillna().

from pyspark.sql import functions as F
mode = (train_df.filter(F.col("city").isNotNull())
        .groupBy("city").count().orderBy(F.desc("count")).first()["city"])
train_df = train_df.fillna({"city": mode})

The isNotNull() filter matters. Without it, null itself could be the most frequent "value" and the fill would do nothing.

Picking by the data, not by habit

Questions on this objective make the point directly. To get correct results with little effort, examine the distribution and then pick the strategy; no imputer chooses for you, and "always use the mean" is not correct for every continuous column.

How the exam asks about this

• "House sizes are right skewed with a few mansions; fill the missing sizes." Use the median, which the extreme values do not pull upward.
• "Fill missing values in a string column called payment_method." Use the mode, computed and applied with fillna(), since Imputer accepts only numeric columns.
• "A pipeline fills missing values, and the test score looks too good." The imputer was fit on all the data; fit it on the training split only and reuse the fitted model.