Spark Job History Server Outofmemoryerror

One of phData’s customers hit an issue where the Spark Job History was running out of memory every few hours. The heap size was set to 4GB and the customer was not a heavy user of Spark, submitting no more than a couple jobs a day. We noticed that they had many long running spark-shell jobs in the 2hr-60hr range. 

Collecting Heap Dumps

We collected a Java heap dump of the Spark Job History Server and used Eclipse Memory Analyzer (MAT) to analyze it. Our favorite tool for analyzing Java heap dumps is IBM HeapAnalyzer, which can open heap dumps from both Sun and IBM JVMs. However, HeapAnalyzer has trouble opening large heaps which are common in a Hadoop environment. The heap dump showed a couple of leak suspects.

In the heap dump, we can see that there are many instances of JobProgressListener using 300mb – 400mb of heap space. Each one of these instances is for a Spark application shown in the history server. If we drill down into the JobProgressListener, we see that it’s actually 1000s of instances task ui data that’s using all the space in the heap.

Looking at our long running spark-shell jobs in the Spark job history ui, we found that they had iterated through several stages with 10,000+ tasks. (Note: the screenshot shows only 100 tasks, but is illustrative of where you’d see the number of tasks for each stage.)

Looking at the JobProgressListener source code, we found that it will prune the number of jobs and stages retained on the heap but has no limit on the task data stored on heap due to the no pruning functionality.

Since there is no way to limit the number of tasks stored on the heap, we recommend first analyzing your jobs to see if they have too many tasks. In our case, this was caused by a small file problem but also the iterative nature of someone working in spark-shell. The additional complexity with the Spark History Server is that it loads many applications into memory, as configured by spark.history.retainedApplications, unlike the application specific UI available during running applications.

The default 2gb Heap Size is probably too small. We recommend increasing heap size until you find a number that works for you up to 40gb or what you have available. If you’ve increased the heap and are still seeing problems, you should lower the number of retained applications in the spark history server. This is configured by the spark.history.retainedApplications setting discussed earlier which defaults to 50. If you still have problems, you may need to delete the application history files for a specific  problematic Spark applications. These are stored in spark.history.fs.logDirectory, which is configured for /user/spark/applicationHistory by default on the Cloudera distribution of Hadoop.

To conclude:

1. Investigate the source data to ensure they weren’t suffering from a small file problem.
2. Increase the heap on the Spark Job History Server.
3. Reduce spark.history.retainedApplications.
4. If all else fails, delete application history for problem jobs.