About Sparklens
Sparklens is a profiling tool for Apache Spark that includes a Spark scheduler simulator. Its stated aim is to make the scalability limits of a Spark application easy to understand, and to show how efficiently the application uses the compute resources it was given.
From a single run of an application, it estimates completion time and cluster utilisation for different executor counts, so you can judge whether adding executors is worth it. It also points to the stages, the driver, skew or a lack of tasks that hold the job back, and gives context on what may be wrong with them.
The report includes a job and stage timeline, per-stage metrics such as input, output and shuffle volumes, and OneCoreComputeHours available and used per stage. Internally it is built around analyzers that emit events, and the project invites contributions of new ones. It is written in Scala under the Apache License 2.0, and a companion web service renders uploaded Sparklens JSON files as shareable HTML reports.
Key features
- Estimates runtime for different executor counts
- Job and stage timeline showing parallel stage scheduling
- Per-stage input, output and shuffle metrics
- Compute hours available and used per stage
- Pinpoints stages, driver or skew limiting scale-out
- Shareable HTML reports from uploaded JSON files
- Runs inside the app or later from saved data
Good fit for
- Deciding whether more executors will speed up a Spark job
- Finding stages limited by skew or too few tasks
- Sharing a Spark tuning report with colleagues
Sparklens: questions and answers
- What is Sparklens used for?
- Sparklens is a Spark profiling tool with a built-in scheduler simulator that shows how far an application can scale with more executors. It is a good fit for deciding whether more executors will speed up a Spark job, finding stages limited by skew or too few tasks and sharing a Spark tuning report with colleagues.
- Is Sparklens open source?
- Yes. Sparklens is open source under the Apache-2.0 licence. Its source code is on GitHub at qubole/sparklens and is written mainly in Scala.
- Is Sparklens free?
- Yes. Sparklens is open source, so the software itself is free to use.
- What are some alternatives to Sparklens?
- Similar open-source tools in the Monitoring & Observability category include Pyroscope, Sysstat and Sentry. SaaS products in the same category include Blackfire, PHP Profiler (by Tideways) and Cortex for Mac.
- Is Sparklens actively maintained?
- The most recent commit to Sparklens was on 26 June 2024. The project has 593 stars on GitHub.
Open-source alternatives to Sparklens
See all
Pyroscope
Monitoring & Observability
Continuous Profiling Platform. Debug performance issues down to a single line of code
AGPL-3.0vs Datadog★ 12kSysstat
Monitoring & Observability
Performance monitoring tools for Linux
GPL-2.0★ 3.4k
Sentry
Monitoring & Observability
Developer-first error tracking and performance monitoring
OSSvs Rollbar★ 46k
Pinpoint
Monitoring & Observability
APM, (Application Performance Management) tool for large-scale distributed systems.
Apache-2.0vs Dynatrace★ 14k
Coroot
Monitoring & Observability
Coroot is an open-source observability and APM tool with AI-powered Root Cause Analysis. I
Apache-2.0vs Datadog★ 8k
Pixie
Monitoring & Observability
Instant Kubernetes-Native Application Observability
Apache-2.0vs Datadog★ 6.6k
SaaS alternatives to Sparklens
See all
Blackfire
Monitoring & Observability
PHP and Python application profiler and performance monitoring service
SaaS
PHP Profiler (by Tideways)
Monitoring & Observability
Shows where PHP performance breaks down in production in real time
SaaS
Cortex for Mac
Monitoring & Observability
Activity Monitor alternative for macOS
SaaS
Lightspeed.run
Monitoring & Observability
Website performance monitoring and optimization tools for conversions and SEO ranking
SaaS
Opskitty
Monitoring & Observability
Website monitoring tool
SaaS
Ultimate WP Audit
Monitoring & Observability
WordPress audit tool running 258+ automated checks across 18 categories on health, security and performance
SaaS
