# Build Reliable Web Data Collectors

Collecting reliable data from static pages, dynamic sites, and APIs requires several tools and careful engineering. This skill provides implementation templates for extraction, cleaning, storage, monitoring, and scaling.

## Install

```bash
npx skillstore add zl2023github/web-scraper
```

## Metadata

- Status: approved
- Slug: zl2023github-web-scraper
- Skillstore revision: r2
- Version status: missing
- Tree hash: 2c59395ca48057f175214df9d47e6d3c50175e8d9c72816975bafb10a89e8e30
- Author: zl2023github
- GitHub username: zl2023github
- License: MIT
- Repository: https://github.com/zl2023github/software-engineer-skills/tree/main/software-engineering/web-scraper
- Ref: 88a8e9a07f4c54ab105c1c41b6267c287146b07b
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: confirmation\_required
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands, network, filesystem, env\_access
- Quality score: 38
- Quality tier: warning
- Public page: https://skillstore.pages.dev/skills/zl2023github-web-scraper
- Manifest: https://skillstore.pages.dev/api/skills/zl2023github-web-scraper/manifest

## Capabilities

- Creates request and parsing patterns for static pages with Requests, Beautiful Soup, lxml, and Parsel.
- Builds Playwright workflows for dynamic rendering, selector waits, scrolling, and browser-based extraction.
- Provides Scrapy templates with pagination, retries, delays, pipelines, and distributed Redis scheduling.
- Cleans, deduplicates, validates, and stores collected records with pandas and MongoDB examples.
- Demonstrates authorized application traffic inspection and response capture with mitmproxy.
- Adds health checks, record-count monitoring, failure detection, and data quality reports.

## Use Cases

- Collect Public Research Data: Build a rate-limited collector for permitted public pages, then normalize and validate the extracted records.
- Automate Dynamic Catalog Extraction: Use Playwright to load dynamic product pages, handle scrolling, and extract changing catalog fields.
- Operate a Scalable Collection Pipeline: Design Scrapy workers, Redis scheduling, storage, monitoring, retries, and data quality controls for authorized sources.

## Prompt Templates

### Create a Basic Scraper

```
Create a polite scraper for [URL] that extracts [fields]. Respect robots.txt, use a one-second delay, and return a table.
```

### Handle Dynamic Content

```
Build a Playwright scraper for [URL]. Wait for [selector], handle scrolling or pagination, extract [fields], and include retries and structured logs.
```

### Build a Data Pipeline

```
Design a Scrapy pipeline for [domain] with deduplication, incremental updates, validation, and [database] storage. Keep robots.txt enforcement enabled.
```

### Architect Distributed Collection

```
Architect an authorized collector for [sources] at [volume]. Include rate limits, queueing, observability, recovery, data lineage, privacy controls, and capacity estimates.
```

## Limitations

- Examples require target-specific selectors, schemas, authentication, and error handling before production use.
- The skill does not grant permission to collect data or guarantee legal and contractual compliance.
- Browser automation, databases, proxies, and distributed queues require separate installation and infrastructure.
- Anti-bot techniques may violate site policies and should not be used without explicit authorization.

## Best Practices

- Confirm authorization, service terms, robots.txt rules, privacy limits, and retention requirements before collecting data.
- Start with small samples, conservative concurrency, explicit timeouts, retries, and source-specific rate limits.
- Preserve data lineage, validate required fields, monitor structural changes, and redact sensitive logs.

## Anti Patterns

- Do not bypass CAPTCHA, WAF, authentication, or browser controls without explicit written authorization.
- Do not route credentials or sensitive records through public proxies or unverified infrastructure.
- Do not store raw intercepted traffic without redaction, encryption, restrictive permissions, and retention limits.

## Security Audit

- Audited at: 2026-07-24T00:18:15.057\+00:00
- Summary: The command-execution alerts are false positives caused by Markdown code fences, and the credential alerts are ordinary parameters or a localhost Redis example. The skill does perform network requests and writes intercepted responses to disk. High-risk semantic concerns include anti-bot circumvention guidance and capture of potentially sensitive application traffic.

## Stats

- Views: 2
- Downloads: 6
- Favorites: 0
- Popularity score: 0