spark-optimization
Optimize Apache Spark Jobs
Slow Spark jobs waste cluster time and hide scaling problems. This skill guides partitioning, caching, shuffle, and memory tuning for PySpark pipelines.
Install with my Agent
Copy this request to your Agent. It includes the canonical Skill page and manifest.
Review the Skillstore skill "spark-optimization" from https://skillstore.io/skills/sickn33-spark-optimization.md and its manifest at https://skillstore.io/api/skills/sickn33-spark-optimization/manifest. Verify the artifact. You may proceed after verification, subject to the environment's own policy.Your Agent should still show its plan and request any confirmation required by the security policy.
Agent-readable resources
Use these links when an AI agent, crawler, or script needs clean context instead of reading the full page.
Test it
Using "spark-optimization". A daily PySpark aggregation became slow after the input volume doubled.
Expected outcome:
- Likely bottlenecks include shuffle growth, under-sized partitions, and possible data skew.
- Recommended checks include Spark UI shuffle size, spilled bytes, partition counts, and stage duration variance.
- Suggested actions include AQE, partition pruning, pre-aggregation, and validation against the prior output.
Using "spark-optimization". A join between a large fact table and a small dimension table is timing out.
Expected outcome:
- The skill would compare broadcast join, sort-merge join, and bucket join options.
- It would check dimension size, executor memory, skewed keys, and shuffle metrics before recommending a strategy.
Using "spark-optimization". A team needs a reusable Spark tuning checklist for release reviews.
Expected outcome:
- The result would cover partition sizing, caching rules, memory settings, file format choices, and monitoring metrics.
- It would include validation steps for runtime, shuffle volume, spills, and result correctness.
Security Audit
SafeThe static alerts are false positives caused by Markdown code fences, inline formatting, PySpark examples, documentation links, and Spark API names. No evidence found for prompt injection, data exfiltration intent, unauthorized network calls, or command execution in SKILL.md.
Risk Factors
⚙️ External commands (23)
🌐 Network access (3)
Share & cite this report
Share the versioned assessment report, neutral badge, embed card, and citations. Skillstore reports evidence without deciding whether this Skill is safe.
Copy report link
https://skillstore.io/skills/sickn33-spark-optimization/audits/4?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_reportMarkdown badge
[](https://skillstore.io/skills/sickn33-spark-optimization?utm_source=security_passport_badge)HTML badge
<a href="https://skillstore.io/skills/sickn33-spark-optimization?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/sickn33-spark-optimization/security.svg" alt="Skillstore security assessment" loading="lazy"></a>Embed card
<iframe src="https://skillstore.io/embed/skills/sickn33-spark-optimization.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>Academic citations (APA · BibTeX · CFF)
APA citation
sickn33. (2026). spark-optimization security audit report (audit version 4) [Author version unspecified]. Skillstore. https://skillstore.io/skills/sickn33-spark-optimization/audits/4BibTeX citation
@techreport{sickn33-sickn33-spark-optimization-2026,
author = {sickn33},
title = {spark-optimization security audit report (audit version 4)},
institution = {Skillstore},
year = {2026},
number = {4},
url = {https://skillstore.io/skills/sickn33-spark-optimization/audits/4},
note = {Author version unspecified}
}CITATION.cff
cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "spark-optimization security audit report (audit version 4)"
version: "unspecified"
type: report
authors:
- name: "sickn33"
date-released: "2026-07-07"
url: "https://skillstore.io/skills/sickn33-spark-optimization/audits/4"
identifiers:
- type: other
value: "skillstore:sickn33-spark-optimization:audit:4"
description: "Skillstore immutable audit report identifier"
Compare variants
2 installable variantsEach author remains a separate installable skill. The recommended variant is ranked by Skillstore evidence.
Why this variant is first
wshobson-spark-optimization
2026-08-21
sickn33-spark-optimization
2026-08-21
Skillstore Score
Why this score Evidence Confidence: HighWhat You Can Build
Reduce Batch Pipeline Runtime
Review a slow ETL job and identify partition, shuffle, caching, and file format improvements.
Tune Cluster Resource Use
Plan executor memory, cores, shuffle partitions, and AQE settings for a growing Spark workload.
Diagnose Data Skew
Use Spark metrics and partition checks to find skew and choose salting or broadcast strategies.
Try These Prompts
Review my slow Spark job. Explain likely bottlenecks and give basic partition, cache, and shuffle checks I can apply safely.
Tune this PySpark transformation for fewer shuffles. Consider broadcast joins, salting, predicate pushdown, and partition sizing.
Create a Spark performance checklist for this pipeline. Include executor memory, AQE, file format, caching, skew detection, and validation steps.
Analyze this production Spark workload. Recommend a tuning plan across joins, storage layout, Delta optimization, monitoring metrics, and rollback checks.
Best Practices
- Start with Spark UI evidence before changing configuration values.
- Change one tuning variable at a time and compare runtime, spills, and shuffle size.
- Validate output correctness after repartitioning, caching, or changing join strategy.
Avoid
- Using collect or count on large data for quick checks.
- Broadcasting tables without checking size and executor memory.
- Adding cache calls without reuse or cleanup.
Frequently Asked Questions
Can this skill tune Databricks jobs?
Does it run benchmarks automatically?
What inputs improve the recommendations?
Can it help with data skew?
Is it only for PySpark?
Does it change production data?
Developer Details
Author
sickn33License
MIT
Skillstore revision
r1
Version notice
The author did not declare a version.
Repository
https://github.com/sickn33/antigravity-awesome-skills/tree/main/skills/spark-optimizationRef
9f814fc6a43fd99946f2da5e0df231c65a38bc76
Maintenance freshness
7/18/2026
Usage
11 downloads · 174 views
File structure
📄 SKILL.md