# Optimize DataFusion Queries

Slow DataFusion queries can waste compute and hide simple plan issues. This skill reviews SQL and DataFrame patterns, then suggests concrete performance improvements.

## Install

```bash
npx skillstore add emillindfors/datafusion-query-advisor
```

## Metadata

- Status: approved
- Slug: emillindfors-datafusion-query-advisor
- Version: 1.0.0
- Author version: 1.0.0
- Skillstore revision: r1
- Version status: valid
- Tree hash: eaf2c4d828f49986aae8200577156c5c4ad9d82c95a33c7b795029ea5087ac47
- Author: EmilLindfors
- GitHub username: EmilLindfors
- License: MIT
- Repository: https://github.com/EmilLindfors/claude-marketplace/tree/main/plugins/rust-data-engineering/skills/datafusion-query-advisor
- Ref: a06681402992ceae98ba04d54cfd4ab004862696
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands
- Quality score: 80
- Quality tier: silver
- Public page: https://skillstore.pages.dev/skills/emillindfors-datafusion-query-advisor
- Manifest: https://skillstore.pages.dev/api/skills/emillindfors-datafusion-query-advisor/manifest

## Capabilities

- Reviews DataFusion SQL for predicate pushdown and partition pruning opportunities.
- Identifies SELECT star usage and recommends narrower column projection.
- Analyzes joins, aggregations, and window functions for common performance issues.
- Suggests DataFusion configuration changes for parallelism, memory, and batch size.
- Guides users to validate changes with EXPLAIN, logical plans, and physical plans.
- Recommends streaming results instead of collecting large datasets into memory.

## Use Cases

- Review a slow analytics query: Find missing filters, wide projections, expensive aggregations, and plan checks before running large DataFusion workloads.
- Improve a Rust DataFrame pipeline: Check DataFrame API usage for late filtering, large collects, and avoidable memory pressure.
- Tune a query service: Review parallelism, memory limits, batch size, and streaming choices for DataFusion-backed services.

## Prompt Templates

### Check One Query

```
Review this DataFusion SQL query for performance issues. Focus on predicate pushdown, partition pruning, projection, and simple rewrites.
```

### Review DataFrame Usage

```
Review this DataFusion DataFrame workflow. Identify late filters, large collect calls, missing projection, and safer streaming options.
```

### Analyze an EXPLAIN Plan

```
Analyze this DataFusion EXPLAIN output. Explain whether filters, projections, partition pruning, and joins are planned efficiently.
```

### Optimize a Workload

```
Review this DataFusion workload and runtime configuration. Prioritize changes for memory, parallelism, batch size, joins, and validation steps.
```

## Limitations

- It does not execute queries or measure runtime performance.
- It needs the user to provide queries, schemas, plans, or workload context.
- Recommendations can vary by DataFusion version, storage format, and table statistics.
- It cannot replace production profiling or benchmark testing.

## Best Practices

- Share the query, table layout, partition columns, and expected result size.
- Include EXPLAIN output when asking for plan-level performance advice.
- Validate each recommendation with benchmarks or production-like samples.

## Anti Patterns

- Do not ask for tuning advice without the query or DataFrame workflow.
- Do not treat suggestions as proof without EXPLAIN or runtime validation.
- Do not expose sensitive production data when sample schemas and plans are enough.

## Security Audit

- Audited at: 2026-07-06T14:40:43.103\+00:00
- Summary: All static findings were adjudicated as false positives. Detections came from Markdown code fences, inline SQL/Rust examples, and sample SQL column names. No evidence found for prompt injection, data exfiltration, command execution, or host reconnaissance intent.

## Stats

- Views: 200
- Downloads: 8
- Favorites: 0
- Popularity score: 0