Exploring Data: Summary Statistics, Outliers and Charts

What it is

Exploratory data analysis is the first look at a DataFrame before any feature work. You measure each column's center, spread and missing values, find and remove extreme values, and draw charts that show the distribution.

On Databricks the main tools are the Spark DataFrame method summary(), the notebook utility dbutils.data.summarize(), simple filters for outliers and the notebook's built-in charts.

How it works

Summary statistics

The percentiles from summary() are approximate, which keeps the call fast on large data. Both summary() and describe() return a new DataFrame, so you can show it or keep working with it.

dbutils.data.summarize() displays the result in the notebook instead of returning a DataFrame. By default (precise=False) some figures are approximations to cut run time; precise=True makes all statistics exact except the histograms and percentiles of numeric columns. It reads the complete DataFrame, so it can be slow on very large inputs.

df.summary().show()
df.select("price", "sqft").summary("mean", "50%").show()
dbutils.data.summarize(df, precise=True)

Removing outliers

Two rules are common. The standard deviation rule keeps rows within k standard deviations of the mean, often k = 3. The IQR rule keeps rows between Q1 - 1.5 x IQR and Q3 + 1.5 x IQR, where IQR = Q3 - Q1.

The IQR rule uses quartiles, so the extreme values it is hunting barely move its bounds. The standard deviation rule assumes a roughly normal shape, and a few huge values inflate the mean and the standard deviation, which widens the bounds and lets outliers survive.

from pyspark.sql import functions as F
q1, q3 = df.approxQuantile("price", [0.25, 0.75], 0.01)
iqr = q3 - q1
clean = df.filter(F.col("price").between(q1 - 1.5 * iqr, q3 + 1.5 * iqr))

approxQuantile(column, probabilities, relativeError) returns a list of values; a relativeError of 0 computes exact quantiles at a higher cost. Note that the filter also drops rows where price is null, because a comparison with null is never true.

s = df.select(F.mean("price").alias("m"), F.stddev("price").alias("sd")).first()
clean = df.filter(F.abs(F.col("price") - s["m"]) <= 3 * s["sd"])

Charts for one feature

Calling display(df) in a Databricks notebook shows a table with options to add a visualization or a data profile. The right chart depends on the type of the feature.

Comparing two features

df.stat.corr("a", "b") returns the Pearson correlation of two numeric columns. For a correlation matrix, or for Spearman, use pyspark.ml.stat.Correlation on a vector column built with VectorAssembler.

from pyspark.ml.feature import VectorAssembler
from pyspark.ml.stat import Correlation
vec = VectorAssembler(inputCols=["sqft", "price"], outputCol="v").transform(df.dropna())
matrix = Correlation.corr(vec, "v", "spearman").head()[0]

For two categorical columns, df.stat.crosstab("plan", "region") counts each pair of levels. ChiSquareTest in pyspark.ml.stat tests whether categorical features are independent of a categorical label; a small p-value suggests they are related.

Correlation only measures association between numbers. A correlation coefficient on two category codes is meaningless, because the codes have no real order or distance.

How the exam asks about this

• "Get the mean, standard deviation and quartiles of every numeric column in a Spark DataFrame." Use df.summary(); describe() has no quartiles.
• "A column has a few extreme values and a skewed distribution; remove outliers robustly." Use the IQR rule with approxQuantile.
• "Check whether subscription plan and region are related." Both are categorical, so use a crosstab and a chi-square test, not a correlation.