# Optimize Parquet Performance in Rust

Parquet pipelines can waste storage and memory when defaults are used. This skill reviews Rust Parquet reads and writes, then suggests compression, sizing, projection, and filtering improvements.

## Install

```bash
npx skillstore add emillindfors/parquet-optimization
```

## Metadata

- Status: approved
- Slug: emillindfors-parquet-optimization
- Version: 1.0.0
- Author version: 1.0.0
- Skillstore revision: r1
- Version status: valid
- Tree hash: f04bfd002a3c42d119b554027798d3aff1aad75c98021e7d0042c8efc3f6cb99
- Author: EmilLindfors
- GitHub username: EmilLindfors
- License: MIT
- Repository: https://github.com/EmilLindfors/claude-marketplace/tree/main/plugins/rust-data-engineering/skills/parquet-optimization
- Ref: a06681402992ceae98ba04d54cfd4ab004862696
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands
- Quality score: 80
- Quality tier: silver
- Public page: https://skillstore.pages.dev/skills/emillindfors-parquet-optimization
- Manifest: https://skillstore.pages.dev/api/skills/emillindfors-parquet-optimization/manifest

## Capabilities

- Detects Rust Parquet writer and reader patterns in code or technical discussion.
- Suggests compression choices such as ZSTD and Snappy for different access patterns.
- Reviews row group size, file size, data page size, and statistics settings.
- Checks read paths for projection, batch size control, row group filtering, and streaming.
- Explains tradeoffs for hot, warm, and cold data storage strategies.
- Highlights common issues such as small files, missing partitioning, and weak retry handling.

## Use Cases

- Tune Data Lake Writers: Review Parquet writer settings before production deployment to reduce storage cost and improve scan performance.
- Improve Query Throughput: Find missing projection, filtering, statistics, and streaming patterns in readers that process large Parquet datasets.
- Review Pipeline Regressions: Compare recent Parquet code changes against practical optimization checks after file size or latency increases.

## Prompt Templates

### Basic Writer Review

```
Review this Rust Parquet writer for basic performance settings. Focus on compression, row group size, and statistics.
```

### Reader Efficiency Check

```
Analyze this Parquet read path. Identify projection, batch size, row group filtering, and streaming improvements.
```

### Data Lake Layout Advice

```
Evaluate this Parquet dataset layout for cloud analytics. Consider file size, partitioning, compression, and query pruning.
```

### Production Optimization Plan

```
Create a prioritized Parquet optimization plan for this Rust pipeline. Include expected impact, tradeoffs, and validation metrics.
```

## Limitations

- Guidance is focused on Rust Parquet, Apache Arrow, and related data engineering APIs.
- It cannot benchmark workloads unless users provide measurements, metadata, or representative examples.
- Recommendations may need tuning for specific cloud storage, query engines, and schema layouts.
- It does not modify files or run optimization tools by itself.

## Best Practices

- Share representative schemas, file sizes, query patterns, and storage targets before asking for optimization advice.
- Validate each recommendation with benchmark data from the real workload and query engine.
- Prioritize changes that reduce scanned bytes, memory pressure, and cloud storage request overhead.

## Anti Patterns

- Do not apply one compression level to every dataset without considering access frequency and CPU cost.
- Do not create many tiny Parquet files for large analytical datasets.
- Do not collect large reader outputs into memory when streaming batches can process them safely.

## Security Audit

- Audited at: 2026-07-06T15:13:47.637\+00:00
- Summary: All static findings are false positives caused by Markdown backticks in SKILL.md. The file contains Rust Parquet optimization guidance and examples, with no evidence of command execution, exfiltration intent, or prompt injection.

## Stats

- Views: 161
- Downloads: 8
- Favorites: 0
- Popularity score: 0