# Design Reliable Data Pipelines

Data teams need architectures that meet throughput, quality, reliability, and cost targets. This skill provides structured guidance for batch, streaming, orchestration, storage, and monitoring decisions.

## Install

```bash
npx skillstore add sickn33/data-engineering-data-pipeline
```

## Metadata

- Status: approved
- Slug: sickn33-data-engineering-data-pipeline
- Skillstore revision: r2
- Version status: missing
- Tree hash: 50fa097aca69919c1698f55fbac540668bdec7d5dc676beb7c08478ee9f55fa7
- Author: sickn33
- GitHub username: sickn33
- License: MIT
- Repository: https://github.com/sickn33/antigravity-awesome-skills/tree/main/skills/data-engineering-data-pipeline
- Ref: f9e2c34b4f19c7f3e6b0a1e93227b5f77cc12526
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands
- Quality score: 78
- Quality tier: bronze
- Public page: https://skillstore.pages.dev/skills/sickn33-data-engineering-data-pipeline
- Manifest: https://skillstore.pages.dev/api/skills/sickn33-data-engineering-data-pipeline/manifest

## Capabilities

- Compares ETL, ELT, Lambda, Kappa, and Lakehouse architecture patterns.
- Plans batch and streaming ingestion with retries, validation, and replay handling.
- Guides Airflow, Prefect, dbt, Spark, Delta Lake, and Iceberg design choices.
- Defines data quality checks using Great Expectations and dbt tests.
- Outlines monitoring, alerting, storage optimization, and recovery practices.
- Produces architecture, implementation, configuration, observability, and operations deliverables.

## Use Cases

- Plan a new analytics pipeline: Select an ingestion, transformation, storage, and orchestration pattern from defined business requirements.
- Improve pipeline reliability: Add retries, data quality checks, dead letter handling, freshness monitoring, and recovery procedures.
- Review platform costs: Evaluate partitioning, file sizing, lifecycle policies, compute selection, and query optimization opportunities.

## Prompt Templates

### Choose a pipeline pattern

```
Compare ETL, ELT, and Lakehouse patterns for [source], [volume], [latency], and [target]. Recommend one pattern and explain the tradeoffs.
```

### Design a batch pipeline

```
Design an incremental batch pipeline from [source] to [destination]. Include watermarking, validation, retries, orchestration, storage, and monitoring.
```

### Design a streaming pipeline

```
Design a streaming pipeline for [events] at [throughput]. Address delivery semantics, offsets, windowing, schema changes, replay, storage, and alerts.
```

### Review an existing platform

```
Review this pipeline architecture: [details]. Identify reliability, quality, scalability, observability, recovery, and cost risks. Prioritize improvements with measurable acceptance criteria.
```

## Limitations

- The skill provides guidance and generated examples, but it does not deploy infrastructure.
- Recommendations require validation against actual schemas, volumes, service limits, and security policies.
- The included Python pipeline is illustrative and depends on modules not supplied by the skill.
- Cost and performance outcomes depend on workload measurements and provider pricing.

## Best Practices

- Provide source, volume, latency, target, quality, and retention requirements before selecting an architecture.
- Make pipelines idempotent and define replay, rollback, and schema evolution behavior.
- Validate recommendations with workload tests, quality thresholds, service limits, and cost measurements.

## Anti Patterns

- Do not choose tools before documenting workload requirements and operational constraints.
- Do not treat exactly-once processing as automatic across every system boundary.
- Do not optimize partitioning or file sizes without measuring representative queries.

## Security Audit

- Audited at: 2026-07-23T22:13:25.707\+00:00
- Summary: All four static alerts are false positives caused by ordinary data engineering prose and a fenced Python example. No shell execution, reconnaissance behavior, prompt injection, or malicious intent was found.

## Stats

- Views: 109
- Downloads: 9
- Favorites: 0
- Popularity score: 0