Data Processing · Topic 3 of 3
One-Hot Encoding and Log Transformation
Databricks ML Associate: one-hot encode categorical features in Spark ML, know which models and data sets it does not suit, and when to apply a log scale.
- 5 min
- Spark's OneHotEncoder needs index columns; run StringIndexer first on strings.
- dropLast defaults to True, so k levels become a vector of length k - 1; the output is a sparse vector.
- One-hot suits linear models and nominal data with few levels; tree models can use indexed categories directly.
What it is
One-hot encoding turns a categorical feature into a set of 0 or 1 indicators, one per level, so that a model which expects numbers can use it without inventing an order. A color column with levels red, green and blue becomes indicators such as is_red and is_green.
A log transformation replaces a positive numeric value x with log(x). It compresses large values, so a right skewed column that spans several orders of magnitude becomes far more symmetric.
How it works
One-hot encoding in Spark ML
Spark's OneHotEncoder takes a column of category indices, not strings. For string data you run StringIndexer first, which maps each level to an index (by default the most frequent level gets 0).
from pyspark.ml import Pipeline
from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler
idx = StringIndexer(inputCols=["color"], outputCols=["color_idx"], handleInvalid="keep")
ohe = OneHotEncoder(inputCols=["color_idx"], outputCols=["color_vec"])
asm = VectorAssembler(inputCols=["color_vec", "sqft"], outputCol="features")
model = Pipeline(stages=[idx, ohe, asm]).fit(train_df)| Setting | Meaning |
|---|---|
| inputCols / outputCols | Encode several index columns in one stage, one output vector each |
| dropLast (default True) | Drops the last category, so it is encoded as all zeros |
| handleInvalid | "error" throws on an unseen index; "keep" assigns it an extra index |
| Output | A sparse vector column per input column |
With dropLast=True, a feature with 4 levels becomes a vector of length 3. This avoids redundant columns in linear models with an intercept.
Both StringIndexer and OneHotEncoder are estimators, so fit them on training data inside a Pipeline. StringIndexer's handleInvalid defaults to "error", so a level that appears only in new data fails the transform unless you set "keep" or "skip".
Where one-hot encoding fits and where it does not
| Situation | One-hot encoding? | Why |
|---|---|---|
| Linear or logistic regression | Yes | They need numbers and would read an index as an order |
| Neural networks, k-means, SVM | Usually yes | Distance and weights assume numeric meaning |
| Nominal data with few levels (plan, color) | Yes | A handful of new columns, no false order |
| Tree based models (decision tree, random forest, gradient boosted trees) | Often not needed | Trees can split on indexed categories directly |
| High cardinality (zip code, user ID, product SKU) | No | Thousands of sparse columns, slow training, overfitting |
| Ordinal data (low, medium, high) | Often not | An ordered index keeps the order that one-hot discards |
Spark ML tree models can use StringIndexer output as categorical features, so a tree pipeline can skip OneHotEncoder. One-hot encoding splits a feature into many weak indicators that rarely win a split.
For high cardinality columns, group rare levels into "other" or use feature hashing, trading some detail for far fewer columns.
When a log scale is appropriate
| Use a log scale when | Avoid it when |
|---|---|
| The feature or label is right skewed (long tail of large values) | The data is already roughly normal and centered near zero |
| Values span several orders of magnitude (10 to 10,000,000) | The column has zeros or negatives and you have no plan for them |
| Effects are multiplicative or relative (a 10% change matters) | The column is categorical |
| Large values dominate a linear model or a distance metric | The only goal is to rescale a feature for a tree based model |
A worked example: house prices of 100,000 and 10,000,000 differ by a factor of 100. Their base 10 logs are 5 and 7, so the gap shrinks from 9.9 million to 2.
The log of zero is undefined and negatives have no log. For counts that can be zero, use log(1 + x), which Spark offers as F.log1p.
from pyspark.sql import functions as F
df = df.withColumn("log_price", F.log1p("price"))
pred_df = pred_df.withColumn("price_pred", F.expm1("log_pred"))A model trained on a log-transformed label predicts in log units. Exponentiate the predictions (expm1 here, because log1p was used) before computing RMSE or MAE in the original units.
Tree based models are largely unaffected by a log of a feature, because a monotonic transformation keeps the same order of values and the same possible splits.
How the exam asks about this
- "Linear regression must use a 'region' column with five values." StringIndexer then OneHotEncoder, assembled into the feature vector.
- "A random forest uses a zip code column with 40,000 distinct values." Do not one-hot encode; index it, group rare levels or hash it.
- "Income and target are right skewed and span several orders of magnitude." Apply a log transformation to compress the large values.
What to remember for the exam
- Spark's OneHotEncoder needs index columns; run StringIndexer first on strings.
- dropLast defaults to True, so k levels become a vector of length k - 1; the output is a sparse vector.
- One-hot suits linear models and nominal data with few levels; tree models can use indexed categories directly.
- High cardinality features make one-hot encoding a poor choice: many sparse columns and overfitting.
- A log scale suits right skewed, positive data spanning orders of magnitude; use log1p when zeros occur.
- Predictions from a log-transformed label must be exponentiated before evaluation in original units.
A log transformation does not fix every skew. It cannot take zeros or negative values directly, and it is not a way to reduce the cardinality of a categorical feature.
Common mistakes
Feeding strings straight into OneHotEncoder. Spark's OneHotEncoder encodes numeric category indices. Put a StringIndexer stage before it.
One-hot encoding everything. A user ID or zip code column explodes into thousands of columns. Tree models handle indexed categories, and high cardinality features need grouping, hashing or another encoding.
Reporting RMSE in log units. A model trained on log(price) gives errors in log units. Exponentiate the predictions first, or the metric cannot be compared with models trained on raw prices.
Check yourself
Why must StringIndexer run before OneHotEncoder on a string column?
Spark's OneHotEncoder takes numeric category indices, so StringIndexer first maps each string level to an index.
When is one-hot encoding a poor choice?
For high cardinality features such as zip codes or IDs, and often for tree based models, which can split on indexed categories without the extra sparse columns.
Which column suits a log transformation?
A positive, right skewed column that spans several orders of magnitude, such as income or price; use log1p when the column contains zeros.
Topic 3 of 3 in Data Processing
Test yourself on Data Processing
The full Databricks Machine Learning Associate question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.