parquet-optimization
Optimize Parquet Performance in Rust
Parquet pipelines can waste storage and memory when defaults are used. This skill reviews Rust Parquet reads and writes, then suggests compression, sizing, projection, and filtering improvements.
Install with my Agent
Copy this request to your Agent. It includes the canonical Skill page and manifest.
Review the Skillstore skill "parquet-optimization" from https://skillstore.io/skills/emillindfors-parquet-optimization.md and its manifest at https://skillstore.io/api/skills/emillindfors-parquet-optimization/manifest. Verify the artifact. You may proceed after verification, subject to the environment's own policy.Your Agent should still show its plan and request any confirmation required by the security policy.
Agent-readable resources
Use these links when an AI agent, crawler, or script needs clean context instead of reading the full page.
Test it
Using "parquet-optimization". A writer uses default Parquet settings and creates many small files.
Expected outcome:
- The review flags missing compression, small file risk, and absent row group sizing.
- It recommends a balanced compression codec, larger batches, and a compaction strategy.
Using "parquet-optimization". A reader loads all columns from a wide Parquet table and then filters rows in memory.
Expected outcome:
- The review highlights unnecessary I/O and memory pressure.
- It recommends column projection, predicate-aware row group selection, and streaming batch processing.
Using "parquet-optimization". A team stores daily event data without partitioned paths.
Expected outcome:
- The review identifies limited partition pruning and slower queries on date filters.
- It suggests a date-based layout and validation with representative query patterns.
Security Audit
SafeAll static findings are false positives caused by Markdown backticks in SKILL.md. The file contains Rust Parquet optimization guidance and examples, with no evidence of command execution, exfiltration intent, or prompt injection.
Risk Factors
⚙️ External commands (36)
Share & cite this report
Share the versioned assessment report, neutral badge, embed card, and citations. Skillstore reports evidence without deciding whether this Skill is safe.
Copy report link
https://skillstore.io/skills/emillindfors-parquet-optimization/audits/4?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_reportMarkdown badge
[](https://skillstore.io/skills/emillindfors-parquet-optimization?utm_source=security_passport_badge)HTML badge
<a href="https://skillstore.io/skills/emillindfors-parquet-optimization?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/emillindfors-parquet-optimization/security.svg" alt="Skillstore security assessment" loading="lazy"></a>Embed card
<iframe src="https://skillstore.io/embed/skills/emillindfors-parquet-optimization.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>Academic citations (APA · BibTeX · CFF)
APA citation
EmilLindfors. (2026). parquet-optimization security audit report (audit version 4) [Author version 1.0.0]. Skillstore. https://skillstore.io/skills/emillindfors-parquet-optimization/audits/4BibTeX citation
@techreport{emillindfors-emillindfors-parquet-optimization-2026,
author = {EmilLindfors},
title = {parquet-optimization security audit report (audit version 4)},
institution = {Skillstore},
year = {2026},
number = {4},
url = {https://skillstore.io/skills/emillindfors-parquet-optimization/audits/4},
note = {Author version 1.0.0}
}CITATION.cff
cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "parquet-optimization security audit report (audit version 4)"
version: "1.0.0"
type: report
authors:
- name: "EmilLindfors"
date-released: "2026-07-06"
url: "https://skillstore.io/skills/emillindfors-parquet-optimization/audits/4"
identifiers:
- type: other
value: "skillstore:emillindfors-parquet-optimization:audit:4"
description: "Skillstore immutable audit report identifier"
Skillstore Score
Why this score Evidence Confidence: HighWhat You Can Build
Tune Data Lake Writers
Review Parquet writer settings before production deployment to reduce storage cost and improve scan performance.
Improve Query Throughput
Find missing projection, filtering, statistics, and streaming patterns in readers that process large Parquet datasets.
Review Pipeline Regressions
Compare recent Parquet code changes against practical optimization checks after file size or latency increases.
Try These Prompts
Review this Rust Parquet writer for basic performance settings. Focus on compression, row group size, and statistics.
Analyze this Parquet read path. Identify projection, batch size, row group filtering, and streaming improvements.
Evaluate this Parquet dataset layout for cloud analytics. Consider file size, partitioning, compression, and query pruning.
Create a prioritized Parquet optimization plan for this Rust pipeline. Include expected impact, tradeoffs, and validation metrics.
Best Practices
- Share representative schemas, file sizes, query patterns, and storage targets before asking for optimization advice.
- Validate each recommendation with benchmark data from the real workload and query engine.
- Prioritize changes that reduce scanned bytes, memory pressure, and cloud storage request overhead.
Avoid
- Do not apply one compression level to every dataset without considering access frequency and CPU cost.
- Do not create many tiny Parquet files for large analytical datasets.
- Do not collect large reader outputs into memory when streaming batches can process them safely.
Frequently Asked Questions
Does this skill edit Parquet files directly?
Which languages does it focus on?
Can it choose the best compression codec?
Does it help with cloud object storage?
Can it diagnose slow queries?
Is it useful for existing production pipelines?
Developer Details
Author
EmilLindforsLicense
MIT
Author version
v1.0.0
Skillstore revision
r1
Ref
a06681402992ceae98ba04d54cfd4ab004862696
Maintenance freshness
7/18/2026
Usage
5 downloads · 161 views
File structure
📄 SKILL.md