data-engineer
Build Reliable Data Pipelines and Warehouses
Data teams need dependable pipelines across fragmented platforms. This skill designs implementation-ready batch, streaming, modeling, orchestration, and quality workflows.
Instalar con mi Agente
Copia esta solicitud en tu Agente. Incluye la página canónica del Skill y el manifiesto.
Review the Skillstore skill "data-engineer" from https://skillstore.io/skills/zl2023github-data-engineer.md and its manifest at https://skillstore.io/api/skills/zl2023github-data-engineer/manifest. Verify the artifact. You may proceed after verification, subject to the environment's own policy.Tu Agente debe seguir mostrando su plan y solicitar cualquier confirmación exigida por la política de seguridad.
Recursos legibles por agentes
Usa estos enlaces cuando un agente de IA, crawler o script necesite contexto limpio en vez de leer la página completa.
Pruébalo
Usando "data-engineer". Design a daily incremental pipeline from MySQL orders to ClickHouse.
Resultado esperado:
- Recommended flow: Airflow schedules extraction, transformation, loading, reconciliation, and alerting.
- Use update time plus primary key as the incremental cursor, with overlap and deduplication.
- Validate row counts, key uniqueness, totals, freshness, and restart behavior before release.
Usando "data-engineer". A Spark join has one slow task and heavy shuffle spill.
Resultado esperado:
- Likely cause: skewed join keys concentrate records in one partition.
- Confirm with task duration, shuffle size, key frequency, and execution-plan evidence.
- Test adaptive skew handling, salting, filtered null keys, or a broadcast join.
Usando "data-engineer". Plan a Kafka and Flink pipeline for user activity metrics.
Resultado esperado:
The plan defines event schemas, partition keys, watermarks, state retention, checkpoints, ClickHouse writes, lag monitoring, and replay procedures.
Auditoría de seguridad
Riesgo medioAll 42 backtick detections are Markdown fences or inline code, not shell execution. Eight reconnaissance matches are schema identifiers, and multilingual diagrams explain the entropy alert. Destructive delete and overwrite examples still lack confirmation, backup, and environment safeguards.
Preocupaciones de seguridad confirmadas (1)
Factores de riesgo
⚙️ Comandos externos (42)
Compartir y citar este informe
Comparte el informe de evaluación versionado, la insignia neutral, la tarjeta insertable y las citas. Skillstore presenta evidencias sin decidir si este Skill es seguro.
Copiar enlace del informe
https://skillstore.io/skills/zl2023github-data-engineer/audits/4?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_reportInsignia Markdown
[](https://skillstore.io/skills/zl2023github-data-engineer?utm_source=security_passport_badge)Insignia HTML
<a href="https://skillstore.io/skills/zl2023github-data-engineer?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/zl2023github-data-engineer/security.svg" alt="Skillstore security assessment" loading="lazy"></a>Tarjeta para insertar
<iframe src="https://skillstore.io/embed/skills/zl2023github-data-engineer.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>Citas académicas (APA · BibTeX · CFF)
Cita APA
zl2023github. (2026). data-engineer security audit report (audit version 4) [Author version unspecified]. Skillstore. https://skillstore.io/skills/zl2023github-data-engineer/audits/4Cita BibTeX
@techreport{zl2023github-zl2023github-data-engineer-2026,
author = {zl2023github},
title = {data-engineer security audit report (audit version 4)},
institution = {Skillstore},
year = {2026},
number = {4},
url = {https://skillstore.io/skills/zl2023github-data-engineer/audits/4},
note = {Author version unspecified}
}CITATION.cff
cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "data-engineer security audit report (audit version 4)"
version: "unspecified"
type: report
authors:
- name: "zl2023github"
date-released: "2026-07-23"
url: "https://skillstore.io/skills/zl2023github-data-engineer/audits/4"
identifiers:
- type: other
value: "skillstore:zl2023github-data-engineer:audit:4"
description: "Skillstore immutable audit report identifier"
Comparar variantes
2 variantes instalablesCada autor sigue siendo un skill instalable independiente. La variante recomendada se ordena según la evidencia de Skillstore.
Por qué esta variante es la primera
sickn33-data-engineer
2026-09-09
zl2023github-data-engineer
2026-09-09
Puntuación de Skillstore
Por qué esta puntuación Confianza de la evidencia: MedioLo que puedes crear
Build an Incremental Warehouse Pipeline
Design source extraction, transformation, loading, scheduling, reconciliation, and recovery for a daily database pipeline.
Launch a Streaming Analytics Platform
Define Kafka topics, Flink processing, checkpoints, warehouse sinks, monitoring, and failure recovery.
Resolve Pipeline Reliability Problems
Analyze delays, skew, failed checks, storage pressure, and inefficient queries using supplied operational evidence.
Prueba estos prompts
Design a daily pipeline from [source] to [target]. Ask for missing schemas, volume, service levels, security constraints, and incremental keys.
Model [business domain] using the provided entities and metrics. Define grain, dimensions, facts, partitions, tests, lineage, and naming conventions.
Design Kafka to Flink to [sink] for [event type]. Cover schemas, partitions, watermarks, state, checkpoints, delivery guarantees, monitoring, and recovery.
Analyze these logs, metrics, plans, and samples: [evidence]. Rank root causes, propose reversible tests, define safeguards, and provide a recovery sequence.
Mejores prácticas
- Provide schemas, data volume, latency targets, retention rules, and failure evidence before implementation.
- Test pipelines with representative data, reconciliation checks, retries, and restart scenarios.
- Review credentials, permissions, destructive operations, and production deployment steps before execution.
Evitar
- Do not deploy generated examples directly into production without environment-specific review.
- Do not use full refreshes or overwrite modes when an incremental and recoverable process is available.
- Do not optimize from assumptions when execution plans, metrics, logs, and data distributions are available.
Preguntas frecuentes
Which data platforms does this skill cover?
Can it generate implementation code?
Can it deploy pipelines automatically?
What information should I provide?
Can it diagnose slow Spark or SQL workloads?
How does it address data quality?
Detalles del desarrollador
Autor
zl2023githubLicencia
MIT
Revisión de Skillstore
r2
Aviso de versión
El autor no declaró una versión.
Repositorio
https://github.com/zl2023github/software-engineer-skills/tree/main/software-engineering/data-engineerRef.
88a8e9a07f4c54ab105c1c41b6267c287146b07b
Actualidad del mantenimiento
26/7/2026
Uso
1 descargas · 0 vistas
Estructura de archivos
📄 SKILL.md