# Build Reliable Data Pipelines and Warehouses

Data teams need dependable pipelines across fragmented platforms. This skill designs implementation-ready batch, streaming, modeling, orchestration, and quality workflows.

## Install

```bash
npx skillstore add zl2023github/data-engineer
```

## Metadata

- Status: approved
- Slug: zl2023github-data-engineer
- Skillstore revision: r2
- Version status: missing
- Tree hash: 83caa09a6e980e93d04f3e6fd5ac3555ca6cf26b9eafb3940ea57f96261f9be7
- Author: zl2023github
- GitHub username: zl2023github
- License: MIT
- Repository: https://github.com/zl2023github/software-engineer-skills/tree/main/software-engineering/data-engineer
- Ref: 88a8e9a07f4c54ab105c1c41b6267c287146b07b
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands
- Quality score: 69
- Public page: https://skillstore.pages.dev/skills/zl2023github-data-engineer
- Manifest: https://skillstore.pages.dev/api/skills/zl2023github-data-engineer/manifest

## Capabilities

- Designs batch and streaming pipelines for databases, APIs, files, queues, and object storage.
- Creates warehouse and lakehouse models with layered schemas, partitions, and storage formats.
- Drafts Spark, Flink, SQL, dbt, and Python implementation templates.
- Defines Airflow, Dagster, and Prefect orchestration workflows with dependencies, retries, and alerts.
- Builds data quality checks, reconciliation plans, metadata practices, and lineage workflows.
- Diagnoses performance issues involving SQL plans, Spark shuffles, skew, partitions, and file sizes.

## Use Cases

- Build an Incremental Warehouse Pipeline: Design source extraction, transformation, loading, scheduling, reconciliation, and recovery for a daily database pipeline.
- Launch a Streaming Analytics Platform: Define Kafka topics, Flink processing, checkpoints, warehouse sinks, monitoring, and failure recovery.
- Resolve Pipeline Reliability Problems: Analyze delays, skew, failed checks, storage pressure, and inefficient queries using supplied operational evidence.

## Prompt Templates

### Design a Batch ETL Pipeline

```
Design a daily pipeline from [source] to [target]. Ask for missing schemas, volume, service levels, security constraints, and incremental keys.
```

### Create a Warehouse Model

```
Model [business domain] using the provided entities and metrics. Define grain, dimensions, facts, partitions, tests, lineage, and naming conventions.
```

### Implement a Streaming Pipeline

```
Design Kafka to Flink to [sink] for [event type]. Cover schemas, partitions, watermarks, state, checkpoints, delivery guarantees, monitoring, and recovery.
```

### Diagnose a Production Data Incident

```
Analyze these logs, metrics, plans, and samples: [evidence]. Rank root causes, propose reversible tests, define safeguards, and provide a recovery sequence.
```

## Limitations

- Requires access details, schemas, volumes, service levels, and platform constraints from the user.
- Cannot verify production behavior without logs, metrics, execution plans, and representative data.
- Generated configurations need review for installed versions, connectors, authentication, and network policies.
- Operational commands may change or delete data and require confirmation, backups, and controlled testing.

## Best Practices

- Provide schemas, data volume, latency targets, retention rules, and failure evidence before implementation.
- Test pipelines with representative data, reconciliation checks, retries, and restart scenarios.
- Review credentials, permissions, destructive operations, and production deployment steps before execution.

## Anti Patterns

- Do not deploy generated examples directly into production without environment-specific review.
- Do not use full refreshes or overwrite modes when an incremental and recoverable process is available.
- Do not optimize from assumptions when execution plans, metrics, logs, and data distributions are available.

## Security Audit

- Audited at: 2026-07-23T23:28:01.511\+00:00
- Summary: All 42 backtick detections are Markdown fences or inline code, not shell execution. Eight reconnaissance matches are schema identifiers, and multilingual diagrams explain the entropy alert. Destructive delete and overwrite examples still lack confirmation, backup, and environment safeguards.

## Stats

- Views: 0
- Downloads: 5
- Favorites: 0
- Popularity score: 0