The Spark job ran for 53 minutes. It was a chain of joins on a table that was big, but not that big — the sort of job I expected to finish while I made coffee.
The Spark UI showed 250 tasks finishing in seconds. One task then sat there for the next 50-plus minutes. One partition had inherited nearly all the data, so one executor kept working while the rest of the cluster waited.
That is skewed data: one or a few keys have far more rows than the others. Spark partitions rows into chunks and gives each chunk to one task on one machine. An even split keeps the cluster busy. An uneven split makes the whole job run at the speed of the overloaded task.
The usual damage is easy to recognize after you have seen it once. Most tasks finish quickly while a few drag. The hot partition can run out of executor memory, especially when intermediate results are cached. CPU and memory on the other machines sit unused. Joins and aggregations are worse because a shuffle — moving data across the network — can push almost all of one key onto one machine. In the worst case the task hits a memory limit or timeout and the job fails.
What I tried first
There are good tools for skew. Salting adds a random value to a key before the shuffle, spreads that key across partitions, and removes the value afterward. It adds some overhead but prevents any single key from dominating a partition. Co-partitioning puts both datasets in the same partitioning scheme on the join key so Spark can join them locally without reshuffling either side across the network. Spark can also detect skewed joins and automatically split a hot key across multiple tasks, although the result depends on the Spark version and configuration. More partitions, executor memory, or CPU can help too. Sometimes brute force works. Usually it does not, and it can leave a cluster 90% idle while burning money.
None of those fixed this 53-minute job.
The part I missed
The job was part of a high-traffic batch pipeline with large tables and multiple consecutive joins. Every run hit the same single-partition bottleneck. I first blamed the compressed GZip input. GZip is not splittable, so Spark cannot hand different pieces of one file to different tasks; by default it creates one partition per file and loses parallelism at the read stage.
I installed Niels Basjes’s SplittableGZipCodec. Rahul Singha has a good walkthrough for Databricks. The codec lets multiple Spark tasks read the same gzip file. Each task seeks to a byte offset and starts decompressing there, which cuts wall-clock time at the cost of slightly higher total core-hours.
In a Databricks notebook, the path is cluster details → Libraries → Install new → Maven → search for splittablegzip → install. Figures 3 and 4 in the Spark UI walkthrough show the screens.
It helped, but only at the read layer. Parallelism went up (Figure 5), while writes stayed slow and Stage 251 still had a long-tail task (Figure 6). The UI showed the thing I should have checked first: one task was handling dramatically more data than the others (Figure 7).
The real problem was the DAG. Spark represents each transformation — filter, map, join, or group-by — as a node in a Directed Acyclic Graph, the lineage of how each piece of data was produced. I had chained 18 joins without materializing an intermediate result. By the 18th join, the graph was so deep that I zoomed the browser to 30% to capture it in Figure 8.
The lineage is how Spark recovers from lost partitions: it can recompute a partition from any ancestor node. But carrying 18 ancestors through every task adds tracking overhead, and the memory pressure compounds at each step.
I fixed it with checkpointing. After each join, I materialized the intermediate DataFrame. Checkpointing writes that result to disk and severs the DAG; the next join starts clean instead of attaching to the previous 17 joins. .persist() with the DISK_ONLY storage level gives a similar result without the full checkpoint overhead.
Figure 9 shows the PySpark change to perform_joins. Figure 10 shows the shorter DAG, with each join separated from the tangle.
The runtime fell from 53 minutes to 11 seconds.
The joins were cheap. Carrying 18 generations of lineage through every task was expensive. That was the load-bearing mistake.
What I’d Do Differently Next Time
The codec was a reasonable instinct, but it was the second problem. I should have opened the Spark UI’s DAG view before changing the file format. Stage 251 had already shown me the long-tail task; I just did not read it.
The broader lesson isn’t about Spark specifically. It’s that distributed systems make you pay for complexity in surprising places. The joins themselves were small. The lineage tracking was enormous. The cost wasn’t in the compute — it was in the bookkeeping.
The standard advice — salt your keys, tune your partitions, bump your memory — would have been useless here. No amount of salting fixes a DAG that is 18 joins deep. The UI tells you the answer if you’re willing to read it. I wasn’t, at first. Next time I will be.