Build Reliable Data Pipelines and Warehouses
Data teams need dependable pipelines across fragmented platforms. This skill designs implementation-ready batch, streaming, modeling, orchestration, and quality workflows.
Copy this request to your Agent. It includes the canonical Skill page and manifest.
Review the Skillstore skill "data-engineer" from https://skillstore.io/skills/zl2023github-data-engineer.md and its manifest at https://skillstore.io/api/skills/zl2023github-data-engineer/manifest. Verify the artifact. You may proceed after verification, subject to the environment's own policy.Your Agent should still show its plan and request any confirmation required by the security policy.
Use these links when an AI agent, crawler, or script needs clean context instead of reading the full page.
Using "data-engineer". Design a daily incremental pipeline from MySQL orders to ClickHouse.
Expected outcome:
Using "data-engineer". A Spark join has one slow task and heavy shuffle spill.
Expected outcome:
Using "data-engineer". Plan a Kafka and Flink pipeline for user activity metrics.
Expected outcome:
The plan defines event schemas, partition keys, watermarks, state retention, checkpoints, ClickHouse writes, lag monitoring, and replay procedures.
All 42 backtick detections are Markdown fences or inline code, not shell execution. Eight reconnaissance matches are schema identifiers, and multilingual diagrams explain the entropy alert. Destructive delete and overwrite examples still lack confirmation, backup, and environment safeguards.
Share the versioned assessment report, neutral badge, embed card, and citations. Skillstore reports evidence without deciding whether this Skill is safe.
https://skillstore.io/skills/zl2023github-data-engineer/audits/4?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_report[](https://skillstore.io/skills/zl2023github-data-engineer?utm_source=security_passport_badge)<a href="https://skillstore.io/skills/zl2023github-data-engineer?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/zl2023github-data-engineer/security.svg" alt="Skillstore security assessment" loading="lazy"></a><iframe src="https://skillstore.io/embed/skills/zl2023github-data-engineer.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>zl2023github. (2026). data-engineer security audit report (audit version 4) [Author version unspecified]. Skillstore. https://skillstore.io/skills/zl2023github-data-engineer/audits/4@techreport{zl2023github-zl2023github-data-engineer-2026,
author = {zl2023github},
title = {data-engineer security audit report (audit version 4)},
institution = {Skillstore},
year = {2026},
number = {4},
url = {https://skillstore.io/skills/zl2023github-data-engineer/audits/4},
note = {Author version unspecified}
}cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "data-engineer security audit report (audit version 4)"
version: "unspecified"
type: report
authors:
- name: "zl2023github"
date-released: "2026-07-23"
url: "https://skillstore.io/skills/zl2023github-data-engineer/audits/4"
identifiers:
- type: other
value: "skillstore:zl2023github-data-engineer:audit:4"
description: "Skillstore immutable audit report identifier"
Each author remains a separate installable skill. The recommended variant is ranked by Skillstore evidence.
Why this variant is first
sickn33-data-engineer
2026-08-21
zl2023github-data-engineer
2026-08-21
Design source extraction, transformation, loading, scheduling, reconciliation, and recovery for a daily database pipeline.
Define Kafka topics, Flink processing, checkpoints, warehouse sinks, monitoring, and failure recovery.
Analyze delays, skew, failed checks, storage pressure, and inefficient queries using supplied operational evidence.
Design a daily pipeline from [source] to [target]. Ask for missing schemas, volume, service levels, security constraints, and incremental keys.
Model [business domain] using the provided entities and metrics. Define grain, dimensions, facts, partitions, tests, lineage, and naming conventions.
Design Kafka to Flink to [sink] for [event type]. Cover schemas, partitions, watermarks, state, checkpoints, delivery guarantees, monitoring, and recovery.
Analyze these logs, metrics, plans, and samples: [evidence]. Rank root causes, propose reversible tests, define safeguards, and provide a recovery sequence.
Author
zl2023githubLicense
MIT
Skillstore revision
r2
Version notice
The author did not declare a version.
Ref
88a8e9a07f4c54ab105c1c41b6267c287146b07b
Maintenance freshness
7/26/2026
Usage
1 downloads · 0 views
File structure
📄 SKILL.md
Diagnose and Improve Full-Stack Performance
Full-stack teams often lack a repeatable way to measure performance and locate bottlenecks. This skill provides testing workflows, diagnostic commands, optimization guidance, and report structures.
Build and Optimize Search Systems
Search teams need reliable guidance across indexing, ranking, semantic retrieval, and evaluation. This skill provides practical workflows and examples for building and improving search systems.
Design Reliable Payment Systems
Payment teams need consistent designs for transactions, channels, reconciliation, security, and failure handling. This skill provides structured guidance, examples, and review checklists.
Build Reliable Business Intelligence Analysis
Business teams often lack consistent metrics and repeatable analysis workflows. This skill guides SQL, Python, dashboards, experiments, and automated reporting from requirements to delivery.
Evaluate Technology Choices with Clear Tradeoffs
Technology choices often hide tradeoffs across performance, cost, reliability, operations, and team readiness. This skill compares candidates, recommends an option, and defines proof-of-concept checks.
Generate Clear Software Architecture Documents
Architecture decisions often remain scattered across requirements, discussions, and diagrams. This skill organizes them into consistent Markdown documents for implementation and review.
Design Reliable Data Pipelines
by sickn33
Data teams need architectures that meet throughput, quality, reliability, and cost targets. This skill provides structured guidance for batch, streaming, orchestration, storage, and monitoring decisions.
Build Production Data Pipelines
by alirezarezvani
Production data teams need reliable patterns for pipelines, quality checks, and orchestration. This skill guides architecture, scaffolding, and DataOps practices for modern data stacks.
Coordinate Data Pipelines and Quality Checks
by Brownbull
Data pipeline work can become inconsistent when ingestion, features, quality, and reporting are planned separately. This skill organizes data tasks around pipeline context, quality gates, lineage, and handoffs.
Optimize Parquet Performance in Rust
by EmilLindfors
Parquet pipelines can waste storage and memory when defaults are used. This skill reviews Rust Parquet reads and writes, then suggests compression, sizing, projection, and filtering improvements.
Optimize DataFusion Queries
by EmilLindfors
Slow DataFusion queries can waste compute and hide simple plan issues. This skill reviews SQL and DataFrame patterns, then suggests concrete performance improvements.
Optimize Apache Spark Jobs Faster
by wshobson
Slow Spark jobs waste cluster budget and delay data delivery. This skill guides Claude, Codex, and Claude Code through partitioning, shuffle, memory, join, caching, and monitoring improvements.