# Design Reliable Data Platforms

Data teams need reliable architectures that meet scale, latency, governance, and cost requirements. This skill turns those requirements into pipeline designs, implementation plans, quality controls, and operating guidance.

## Install

```bash
npx skillstore add sickn33/data-engineer
```

## Metadata

- Status: approved
- Slug: sickn33-data-engineer
- Skillstore revision: r2
- Version status: missing
- Tree hash: 09e27930c01829b78d25501b73e863913e0f403920d85aa0e4400294741f7427
- Author: sickn33
- GitHub username: sickn33
- License: MIT
- Repository: https://github.com/sickn33/antigravity-awesome-skills/tree/main/skills/data-engineer
- Ref: f9e2c34b4f19c7f3e6b0a1e93227b5f77cc12526
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Quality score: 79
- Quality tier: bronze
- Public page: https://skillstore.pages.dev/skills/sickn33-data-engineer
- Manifest: https://skillstore.pages.dev/api/skills/sickn33-data-engineer/manifest

## Capabilities

- Designs batch and streaming pipeline architectures around volume, latency, consistency, and reliability requirements.
- Recommends storage, processing, orchestration, and cloud services for warehouse and lakehouse workloads.
- Plans dimensional models, incremental loading, change data capture, partitioning, and query optimization.
- Defines data quality checks, lineage, access controls, privacy measures, monitoring, and failure recovery.
- Evaluates performance, scalability, operational complexity, and cloud cost tradeoffs.

## Use Cases

- Build an Analytics Warehouse: Plan ingestion, dbt transformations, dimensional models, tests, lineage, and orchestration for a governed analytics warehouse.
- Design a Streaming Platform: Select messaging, processing, storage, schema evolution, monitoring, and recovery patterns for low-latency event workloads.
- Modernize Data Governance: Define quality controls, catalogs, access policies, privacy safeguards, retention rules, and audit processes across data products.

## Prompt Templates

### Plan a Basic Batch Pipeline

```
Recommend a batch pipeline from [source] to [warehouse]. Use [daily volume], [refresh target], and [cloud]. Explain components, checks, and operations.
```

### Design a Warehouse Model

```
Design a dimensional model for [business process]. Include grain, facts, dimensions, history handling, incremental loads, tests, lineage, and key assumptions.
```

### Architect a Streaming Pipeline

```
Architect a streaming pipeline for [event source] at [throughput] and [latency]. Address schemas, ordering, duplicates, late data, recovery, monitoring, and cost.
```

### Evaluate an Enterprise Data Platform

```
Compare [options] for [workload portfolio]. Score scalability, reliability, governance, security, migration effort, operating complexity, and cost. Recommend a phased architecture.
```

## Limitations

- It provides guidance and generated artifacts but does not deploy infrastructure or access data systems by itself.
- Recommendations require accurate source schemas, workloads, service levels, compliance needs, and platform constraints.
- Generated designs and implementation details need environment-specific testing, security review, and cost validation.
- It is not intended for standalone exploratory analysis or model development without data pipeline requirements.

## Best Practices

- Provide data volumes, latency targets, retention, consistency, recovery objectives, and expected growth.
- Specify current tools, cloud constraints, source schemas, sink requirements, team skills, and budget limits.
- Request quality, security, governance, observability, cost, and failure recovery requirements in every production design.

## Anti Patterns

- Do not request a tool choice without describing workloads, service levels, constraints, and ownership.
- Do not deploy generated configurations before testing permissions, data correctness, failure behavior, and rollback procedures.
- Do not optimize only for throughput while ignoring quality, privacy, operational complexity, and total cost.

## Security Audit

- Audited at: 2026-07-23T22:11:05.454\+00:00
- Summary: The sole static finding is a false positive caused by a list of real-time analytics products in SKILL.md. No system reconnaissance instructions, prompt injection, or other malicious intent were found.

## Stats

- Views: 98
- Downloads: 10
- Favorites: 1
- Popularity score: 0