Difficulty: Hard | Concepts: Spark Partitioning, Data Skew, Z-Ordering, OPTIMIZE, VACUUM, Caching

"A notebook that used to take 5 minutes now takes 45 — how do you debug and fix it?" is a favorite Fabric Notebook performance interview question because it's open-ended and reveals whether a candidate actually understands how Spark executes work, not just how to write PySpark syntax.

What Is Data Skew and Why Does It Matter?

Spark splits a Dataframe into partitions and processes them in parallel across executors. Data skew happens when the data isn't evenly distributed across those partitions — for example, if you partition sales data by Country and 80% of rows belong to "USA", one executor ends up doing almost all the work while the others sit idle. The job's total time is bound by its slowest task, so a skewed partition can make a job run many times slower than it should, even though total data volume hasn't changed. Recognizing and fixing skew (via repartitioning, salting keys, or broadcast joins) is one of the highest-leverage Spark performance skills.

Scenario

A nightly notebook that aggregates fact_sales (500M rows) by StoreID has degraded from 5 minutes to 45 minutes over the past month, with no change to the notebook code. How do you investigate and fix it?

Diagnosis Checklist

Check	What it reveals
Spark UI → Stages tab	One task taking far longer than others = skew
Number of small files in Delta table	"Small file problem" from frequent appends without compaction
Explain plan (`df.explain()`)	Unnecessary shuffles or missing partition pruning
Table growth over time	Natural volume growth vs. a real regression

Fixes

# 1. Repartition to spread skewed keys more evenly before a groupBy
df = df.repartition(200, "StoreID")

# 2. Compact small files and improve data layout with OPTIMIZE + Z-ORDER
spark.sql("OPTIMIZE fact_sales ZORDER BY (StoreID, SaleDate)")

# 3. Clean up old file versions no longer needed (after OPTIMIZE)
spark.sql("VACUUM fact_sales RETAIN 168 HOURS")

# 4. Cache a reused intermediate Dataframe within the same notebook run
lookup_df = spark.read.table("dim_store").cache()

Visual Representation

Before: fact_sales stored as 40,000 tiny Parquet files (small-file problem) | OPTIMIZE fact_sales ZORDER BY (StoreID, SaleDate) | After: compacted into ~400 well-sized files, co-located by StoreID/SaleDate | groupBy(StoreID) now reads fewer files with better data locality | Runtime drops from 45 min back toward 5 min

Common Mistake to Avoid

Candidates often jump straight to "add more nodes / increase capacity" without first checking the Spark UI for skew or the small-file problem. Throwing more compute at a skew problem rarely helps, since the bottleneck is one slow task, not overall cluster capacity — interviewers want to see diagnosis before scaling.

Conclusion

Strong answers walk through diagnosis first (Spark UI, file counts, explain plans) before jumping to fixes, and know the three go-to tools for Delta table health: repartition for skew, OPTIMIZE + ZORDER for file layout, and VACUUM for cleanup.