Sparklens

Open source

Sparklens is a Spark profiling tool with a built-in scheduler simulator that shows how far an application can scale with more executors.

GitHub stars
593
Last 30 days
+0
Last commit
2 yr ago
Repository age
8 years
Licence
Apache-2.0
Self-hosted
Yes

About Sparklens

Sparklens is a profiling tool for Apache Spark that includes a Spark scheduler simulator. Its stated aim is to make the scalability limits of a Spark application easy to understand, and to show how efficiently the application uses the compute resources it was given.

From a single run of an application, it estimates completion time and cluster utilisation for different executor counts, so you can judge whether adding executors is worth it. It also points to the stages, the driver, skew or a lack of tasks that hold the job back, and gives context on what may be wrong with them.

The report includes a job and stage timeline, per-stage metrics such as input, output and shuffle volumes, and OneCoreComputeHours available and used per stage. Internally it is built around analyzers that emit events, and the project invites contributions of new ones. It is written in Scala under the Apache License 2.0, and a companion web service renders uploaded Sparklens JSON files as shareable HTML reports.

Key features

  • Estimates runtime for different executor counts
  • Job and stage timeline showing parallel stage scheduling
  • Per-stage input, output and shuffle metrics
  • Compute hours available and used per stage
  • Pinpoints stages, driver or skew limiting scale-out
  • Shareable HTML reports from uploaded JSON files
  • Runs inside the app or later from saved data

Good fit for

  • Deciding whether more executors will speed up a Spark job
  • Finding stages limited by skew or too few tasks
  • Sharing a Spark tuning report with colleagues
Built with
Scala tools
Tags
spark
profiling
performance
tuning
scala
scheduler
simulation

Sparklens: questions and answers

What is Sparklens used for?
Sparklens is a Spark profiling tool with a built-in scheduler simulator that shows how far an application can scale with more executors. It is a good fit for deciding whether more executors will speed up a Spark job, finding stages limited by skew or too few tasks and sharing a Spark tuning report with colleagues.
Is Sparklens open source?
Yes. Sparklens is open source under the Apache-2.0 licence. Its source code is on GitHub at qubole/sparklens and is written mainly in Scala.
Is Sparklens free?
Yes. Sparklens is open source, so the software itself is free to use.
What are some alternatives to Sparklens?
Similar open-source tools in the Monitoring & Observability category include Pyroscope, Sysstat and Sentry. SaaS products in the same category include Blackfire, PHP Profiler (by Tideways) and Cortex for Mac.
Is Sparklens actively maintained?
The most recent commit to Sparklens was on 26 June 2024. The project has 593 stars on GitHub.

Open-source alternatives to Sparklens

See all

SaaS alternatives to Sparklens

See all