스킬 data-engineer

📦

data-engineer

Name: data-engineer
Author: zl2023github

콘텐츠 리비전 r2 중간 위험 ⚙️ 외부 명령어

Build Reliable Data Pipelines and Warehouses

Data teams need dependable pipelines across fragmented platforms. This skill designs implementation-ready batch, streaming, modeling, orchestration, and quality workflows.

지원: Claude Codex Code(CC)

📊 69 적절함

Maintain this Skill? Claim it 📊 69 적절함

내 에이전트로 설치

이 요청을 에이전트에 복사하세요. 공식 Skill 페이지와 매니페스트가 포함되어 있습니다.

에이전트 요청

Review the Skillstore skill "data-engineer" from https://skillstore.io/skills/zl2023github-data-engineer.md and its manifest at https://skillstore.io/api/skills/zl2023github-data-engineer/manifest. Verify the artifact. You may proceed after verification, subject to the environment's own policy.

에이전트는 계속해서 계획을 표시하고 보안 정책에서 요구하는 확인을 요청해야 합니다.

에이전트가 읽기 쉬운 리소스

AI 에이전트, 크롤러, 스크립트가 전체 페이지 대신 정리된 컨텍스트만 필요할 때 이 링크를 사용하세요.

Markdown 상세 GET /skills/zl2023github-data-engineer.md 서명된 manifest GET /api/skills/zl2023github-data-engineer/manifest 서명된 lockfile GET /api/skills/zl2023github-data-engineer/lockfile

테스트해 보기

"data-engineer" 사용 중입니다. Design a daily incremental pipeline from MySQL orders to ClickHouse.

예상 결과:

Recommended flow: Airflow schedules extraction, transformation, loading, reconciliation, and alerting.
Use update time plus primary key as the incremental cursor, with overlap and deduplication.
Validate row counts, key uniqueness, totals, freshness, and restart behavior before release.

"data-engineer" 사용 중입니다. A Spark join has one slow task and heavy shuffle spill.

예상 결과:

Likely cause: skewed join keys concentrate records in one partition.
Confirm with task duration, shuffle size, key frequency, and execution-plan evidence.
Test adaptive skew handling, salting, filtered null keys, or a broadcast join.

"data-engineer" 사용 중입니다. Plan a Kafka and Flink pipeline for user activity metrics.

예상 결과:

The plan defines event schemas, partition keys, watermarks, state retention, checkpoints, ClickHouse writes, lag monitoring, and replay procedures.

보안 감사

중간 위험

v4 • 2026. 7. 23. 버전별 보고서 열기

All 42 backtick detections are Markdown fences or inline code, not shell execution. Eight reconnaissance matches are schema identifiers, and multilingual diagrams explain the entropy alert. Destructive delete and overwrite examples still lack confirmation, backup, and environment safeguards.

스캔된 파일

1,058

분석된 줄 수

검토 항목

오탐 무시됨

확인된 보안 우려 사항 (1)

중간

SKILL.md:698 SKILL.md:819-825

Destructive Data Operations Lack Guardrails

The command reference shows an ALTER TABLE DELETE, and the PySpark template uses overwrite mode. Neither example requires confirmation, backup verification, or a dry run.

Both destructive operations appear directly in reusable examples, with no adjacent safety gate. The placeholder targets reduce immediate impact but not reuse risk.

감사자: codex 감사 이력 보기 →

이 보고서 공유 및 인용

버전이 지정된 평가 보고서, 중립 배지, 임베드 카드 및 인용을 공유하세요. Skillstore는 근거를 보고하지만 이 Skill의 안전 여부를 판단하지 않습니다.

버전별 보고서 열기

보고서 링크 복사

https://skillstore.io/skills/zl2023github-data-engineer/audits/4?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_report

Markdown 배지

[![Skillstore security assessment](https://skillstore.io/badges/skills/zl2023github-data-engineer/security.svg)](https://skillstore.io/skills/zl2023github-data-engineer?utm_source=security_passport_badge)

HTML 배지

<a href="https://skillstore.io/skills/zl2023github-data-engineer?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/zl2023github-data-engineer/security.svg" alt="Skillstore security assessment" loading="lazy"></a>

임베드 카드

<iframe src="https://skillstore.io/embed/skills/zl2023github-data-engineer.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>

학술 인용 (APA · BibTeX · CFF)

APA 인용

zl2023github. (2026). data-engineer security audit report (audit version 4) [Author version unspecified]. Skillstore. https://skillstore.io/skills/zl2023github-data-engineer/audits/4

BibTeX 인용

@techreport{zl2023github-zl2023github-data-engineer-2026,
  author = {zl2023github},
  title = {data-engineer security audit report (audit version 4)},
  institution = {Skillstore},
  year = {2026},
  number = {4},
  url = {https://skillstore.io/skills/zl2023github-data-engineer/audits/4},
  note = {Author version unspecified}
}

CITATION.cff

cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "data-engineer security audit report (audit version 4)"
version: "unspecified"
type: report
authors:
  - name: "zl2023github"
date-released: "2026-07-23"
url: "https://skillstore.io/skills/zl2023github-data-engineer/audits/4"
identifiers:
  - type: other
    value: "skillstore:zl2023github-data-engineer:audit:4"
    description: "Skillstore immutable audit report identifier"

변형 비교

설치 가능한 변형 2개

각 작성자의 스킬은 별도의 설치 항목으로 유지됩니다. 추천 변형은 Skillstore 근거로 순위가 정해집니다.

이 변형이 1위인 이유

Skillstore 점수가 가장 높음

작성자 / 소스 Skillstore 점수 증거 신뢰도 Skillstore 사용량 업데이트됨

sickn33 권장

sickn33-data-engineer

Skillstore 점수 79

증거 신뢰도 높음

Skillstore 사용량 15

업데이트됨

2026-09-09

zl2023github 현재

zl2023github-data-engineer

Skillstore 점수 69

증거 신뢰도 중간

Skillstore 사용량 12

업데이트됨

2026-09-09

Skillstore 점수

이 점수의 이유 증거 신뢰도: 중간

아키텍처

유지보수성

콘텐츠

커뮤니티

사양 준수

만들 수 있는 것

Build an Incremental Warehouse Pipeline

Design source extraction, transformation, loading, scheduling, reconciliation, and recovery for a daily database pipeline.

Launch a Streaming Analytics Platform

Define Kafka topics, Flink processing, checkpoints, warehouse sinks, monitoring, and failure recovery.

Resolve Pipeline Reliability Problems

Analyze delays, skew, failed checks, storage pressure, and inefficient queries using supplied operational evidence.

이 프롬프트를 사용해 보세요

Design a Batch ETL Pipeline

Design a daily pipeline from [source] to [target]. Ask for missing schemas, volume, service levels, security constraints, and incremental keys.

Create a Warehouse Model

Model [business domain] using the provided entities and metrics. Define grain, dimensions, facts, partitions, tests, lineage, and naming conventions.

Implement a Streaming Pipeline

Design Kafka to Flink to [sink] for [event type]. Cover schemas, partitions, watermarks, state, checkpoints, delivery guarantees, monitoring, and recovery.

Diagnose a Production Data Incident

Analyze these logs, metrics, plans, and samples: [evidence]. Rank root causes, propose reversible tests, define safeguards, and provide a recovery sequence.

모범 사례

Provide schemas, data volume, latency targets, retention rules, and failure evidence before implementation.
Test pipelines with representative data, reconciliation checks, retries, and restart scenarios.
Review credentials, permissions, destructive operations, and production deployment steps before execution.

피하기

Do not deploy generated examples directly into production without environment-specific review.
Do not use full refreshes or overwrite modes when an incremental and recoverable process is available.
Do not optimize from assumptions when execution plans, metrics, logs, and data distributions are available.

자주 묻는 질문

Which data platforms does this skill cover?

It covers common Spark, Flink, Kafka, Airflow, dbt, warehouse, lakehouse, and data quality workflows.

Can it generate implementation code?

Yes. It can draft Python, SQL, Spark, Flink, dbt, and orchestration templates for review.

Can it deploy pipelines automatically?

Deployment depends on available tools and permissions. Review all commands, credentials, targets, and rollback steps before execution.

What information should I provide?

Provide source and target schemas, volumes, latency goals, incremental keys, security constraints, and current platform details.

Can it diagnose slow Spark or SQL workloads?

Yes. Supply execution plans, task metrics, logs, configurations, and representative data distributions for evidence-based diagnosis.

How does it address data quality?

It defines completeness, uniqueness, validity, consistency, freshness, reconciliation, lineage, and alerting checks.

개발자 세부 정보

작성자

zl2023github

라이선스

MIT

Skillstore 리비전

버전 안내

작성자가 버전을 선언하지 않았습니다.

리포지토리

https://github.com/zl2023github/software-engineer-skills/tree/main/software-engineering/data-engineer

참조

88a8e9a07f4c54ab105c1c41b6267c287146b07b

유지보수 최신성

2026. 7. 26.

사용량

1 다운로드 · 0 조회

파일 구조

📄 SKILL.md

data-engineer

내 에이전트로 설치

에이전트가 읽기 쉬운 리소스

테스트해 보기

보안 감사

확인된 보안 우려 사항 (1)

위험 요인

보고서 링크 복사

Markdown 배지

HTML 배지

임베드 카드

APA 인용

BibTeX 인용

CITATION.cff

변형 비교

Skillstore 점수

만들 수 있는 것

Build an Incremental Warehouse Pipeline

Launch a Streaming Analytics Platform

Resolve Pipeline Reliability Problems

이 프롬프트를 사용해 보세요

모범 사례

피하기

자주 묻는 질문

개발자 세부 정보