About Sparklens

Sparklens is a profiling and performance-tuning tool for Apache Spark applications from Qubole. Its main goal is to make the scalability limits of a Spark application easy to understand, showing how efficiently it uses the compute resources it was given.

From a single run of an application, a built-in Spark scheduler simulator estimates completion time and cluster utilization for different numbers of executors, which helps judge whether adding executors will pay off. It also narrows problems down to the stages, the driver, skew or lack of tasks that limit scaling, and shows a job and stage timeline of how parallel stages ran.

A reporting service at sparklens.qubole.com lets users upload the Sparklens JSON output and get a shareable HTML report with charts. The tool is written in Scala and licensed under Apache-2.0, and it aims to turn Spark tuning into a defined process instead of trial and error.