data-engineer
Build Reliable Data Pipelines and Warehouses
Data teams need dependable pipelines across fragmented platforms. This skill designs implementation-ready batch, streaming, modeling, orchestration, and quality workflows.
Mit meinem Agent installieren
Kopieren Sie diese Anfrage in Ihren Agent. Sie enthält die maßgebliche Skill-Seite und das Manifest.
Review the Skillstore skill "data-engineer" from https://skillstore.io/skills/zl2023github-data-engineer.md and its manifest at https://skillstore.io/api/skills/zl2023github-data-engineer/manifest. Verify the artifact. You may proceed after verification, subject to the environment's own policy.Ihr Agent sollte weiterhin seinen Plan anzeigen und alle von der Sicherheitsrichtlinie verlangten Bestätigungen anfordern.
Agent-lesbare Ressourcen
Verwenden Sie diese Links, wenn ein KI-Agent, Crawler oder Skript sauberen Kontext benötigt, statt die vollständige Seite zu lesen.
Testen
„data-engineer“ wird verwendet. Design a daily incremental pipeline from MySQL orders to ClickHouse.
Erwartetes Ergebnis:
- Recommended flow: Airflow schedules extraction, transformation, loading, reconciliation, and alerting.
- Use update time plus primary key as the incremental cursor, with overlap and deduplication.
- Validate row counts, key uniqueness, totals, freshness, and restart behavior before release.
„data-engineer“ wird verwendet. A Spark join has one slow task and heavy shuffle spill.
Erwartetes Ergebnis:
- Likely cause: skewed join keys concentrate records in one partition.
- Confirm with task duration, shuffle size, key frequency, and execution-plan evidence.
- Test adaptive skew handling, salting, filtered null keys, or a broadcast join.
„data-engineer“ wird verwendet. Plan a Kafka and Flink pipeline for user activity metrics.
Erwartetes Ergebnis:
The plan defines event schemas, partition keys, watermarks, state retention, checkpoints, ClickHouse writes, lag monitoring, and replay procedures.
Sicherheitsaudit
Mittleres RisikoAll 42 backtick detections are Markdown fences or inline code, not shell execution. Eight reconnaissance matches are schema identifiers, and multilingual diagrams explain the entropy alert. Destructive delete and overwrite examples still lack confirmation, backup, and environment safeguards.
Bestätigte Sicherheitsbedenken (1)
Risikofaktoren
⚙️ Externe Befehle (42)
Diesen Bericht teilen & zitieren
Teile den versionierten Bewertungsbericht, das neutrale Badge, die Einbettungskarte und Zitate. Skillstore berichtet Nachweise, ohne zu entscheiden, ob dieser Skill sicher ist.
Berichtslink kopieren
https://skillstore.io/skills/zl2023github-data-engineer/audits/4?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_reportMarkdown-Badge
[](https://skillstore.io/skills/zl2023github-data-engineer?utm_source=security_passport_badge)HTML-Badge
<a href="https://skillstore.io/skills/zl2023github-data-engineer?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/zl2023github-data-engineer/security.svg" alt="Skillstore security assessment" loading="lazy"></a>Einbettungskarte
<iframe src="https://skillstore.io/embed/skills/zl2023github-data-engineer.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>Wissenschaftliche Zitate (APA · BibTeX · CFF)
APA-Zitat
zl2023github. (2026). data-engineer security audit report (audit version 4) [Author version unspecified]. Skillstore. https://skillstore.io/skills/zl2023github-data-engineer/audits/4BibTeX-Zitat
@techreport{zl2023github-zl2023github-data-engineer-2026,
author = {zl2023github},
title = {data-engineer security audit report (audit version 4)},
institution = {Skillstore},
year = {2026},
number = {4},
url = {https://skillstore.io/skills/zl2023github-data-engineer/audits/4},
note = {Author version unspecified}
}CITATION.cff
cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "data-engineer security audit report (audit version 4)"
version: "unspecified"
type: report
authors:
- name: "zl2023github"
date-released: "2026-07-23"
url: "https://skillstore.io/skills/zl2023github-data-engineer/audits/4"
identifiers:
- type: other
value: "skillstore:zl2023github-data-engineer:audit:4"
description: "Skillstore immutable audit report identifier"
Varianten vergleichen
2 installierbare VariantenJeder Autor bleibt ein eigener installierbarer Skill. Die empfohlene Variante wird nach Skillstore-Nachweisen bewertet.
Warum diese Variante zuerst steht
sickn33-data-engineer
2026-09-09
zl2023github-data-engineer
2026-09-09
Skillstore-Score
Warum dieser Score Evidenzvertrauen: MittelWas Sie erstellen können
Build an Incremental Warehouse Pipeline
Design source extraction, transformation, loading, scheduling, reconciliation, and recovery for a daily database pipeline.
Launch a Streaming Analytics Platform
Define Kafka topics, Flink processing, checkpoints, warehouse sinks, monitoring, and failure recovery.
Resolve Pipeline Reliability Problems
Analyze delays, skew, failed checks, storage pressure, and inefficient queries using supplied operational evidence.
Diese Prompts ausprobieren
Design a daily pipeline from [source] to [target]. Ask for missing schemas, volume, service levels, security constraints, and incremental keys.
Model [business domain] using the provided entities and metrics. Define grain, dimensions, facts, partitions, tests, lineage, and naming conventions.
Design Kafka to Flink to [sink] for [event type]. Cover schemas, partitions, watermarks, state, checkpoints, delivery guarantees, monitoring, and recovery.
Analyze these logs, metrics, plans, and samples: [evidence]. Rank root causes, propose reversible tests, define safeguards, and provide a recovery sequence.
Bewährte Praktiken
- Provide schemas, data volume, latency targets, retention rules, and failure evidence before implementation.
- Test pipelines with representative data, reconciliation checks, retries, and restart scenarios.
- Review credentials, permissions, destructive operations, and production deployment steps before execution.
Vermeiden
- Do not deploy generated examples directly into production without environment-specific review.
- Do not use full refreshes or overwrite modes when an incremental and recoverable process is available.
- Do not optimize from assumptions when execution plans, metrics, logs, and data distributions are available.
Häufig gestellte Fragen
Which data platforms does this skill cover?
Can it generate implementation code?
Can it deploy pipelines automatically?
What information should I provide?
Can it diagnose slow Spark or SQL workloads?
How does it address data quality?
Entwicklerdetails
Autor
zl2023githubLizenz
MIT
Skillstore-Revision
r2
Versionshinweis
Der Autor hat keine Version angegeben.
Ref.
88a8e9a07f4c54ab105c1c41b6267c287146b07b
Aktualität der Wartung
29.7.2026
Nutzung
3 Downloads · 0 Aufrufe
Dateistruktur
📄 SKILL.md