Build Reliable Web Data Collectors
Collecting reliable data from static pages, dynamic sites, and APIs requires several tools and careful engineering. This skill provides implementation templates for extraction, cleaning, storage, monitoring, and scaling.
Stop for confirmation before installing.
Review the plan and obtain explicit user consent before changing files.
Copy this request to your Agent. It includes the canonical Skill page and manifest.
Review the Skillstore skill "web-scraper" from https://skillstore.io/skills/zl2023github-web-scraper.md and its manifest at https://skillstore.io/api/skills/zl2023github-web-scraper/manifest. Verify the artifact. Stop and obtain explicit user consent before installing or changing files.Your Agent should still show its plan and request any confirmation required by the security policy.
Use these links when an AI agent, crawler, or script needs clean context instead of reading the full page.
Using "web-scraper". Extract article titles, publication dates, and links from an approved news archive.
Expected outcome:
A normalized table with title, publication date, source link, validation status, and collection timestamp.
Using "web-scraper". Design a recurring collector for a permitted dynamic catalog with daily updates.
Expected outcome:
Using "web-scraper". Assess a proposed Scrapy deployment for several authorized sources.
Expected outcome:
A deployment plan covering workers, Redis scheduling, storage, monitoring, failure recovery, data validation, and source-specific limits.
The command-execution alerts are false positives caused by Markdown code fences, and the credential alerts are ordinary parameters or a localhost Redis example. The skill does perform network requests and writes intercepted responses to disk. High-risk semantic concerns include anti-bot circumvention guidance and capture of potentially sensitive application traffic.
These are real local capabilities that may be expected for this skill, so they require review but are not counted as confirmed malicious behavior.
Share the versioned assessment report, neutral badge, embed card, and citations. Skillstore reports evidence without deciding whether this Skill is safe.
https://skillstore.io/skills/zl2023github-web-scraper/audits/4?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_report[](https://skillstore.io/skills/zl2023github-web-scraper?utm_source=security_passport_badge)<a href="https://skillstore.io/skills/zl2023github-web-scraper?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/zl2023github-web-scraper/security.svg" alt="Skillstore security assessment" loading="lazy"></a><iframe src="https://skillstore.io/embed/skills/zl2023github-web-scraper.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>zl2023github. (2026). web-scraper security audit report (audit version 4) [Author version unspecified]. Skillstore. https://skillstore.io/skills/zl2023github-web-scraper/audits/4@techreport{zl2023github-zl2023github-web-scraper-2026,
author = {zl2023github},
title = {web-scraper security audit report (audit version 4)},
institution = {Skillstore},
year = {2026},
number = {4},
url = {https://skillstore.io/skills/zl2023github-web-scraper/audits/4},
note = {Author version unspecified}
}cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "web-scraper security audit report (audit version 4)"
version: "unspecified"
type: report
authors:
- name: "zl2023github"
date-released: "2026-07-24"
url: "https://skillstore.io/skills/zl2023github-web-scraper/audits/4"
identifiers:
- type: other
value: "skillstore:zl2023github-web-scraper:audit:4"
description: "Skillstore immutable audit report identifier"
Build a rate-limited collector for permitted public pages, then normalize and validate the extracted records.
Use Playwright to load dynamic product pages, handle scrolling, and extract changing catalog fields.
Design Scrapy workers, Redis scheduling, storage, monitoring, retries, and data quality controls for authorized sources.
Create a polite scraper for [URL] that extracts [fields]. Respect robots.txt, use a one-second delay, and return a table.
Build a Playwright scraper for [URL]. Wait for [selector], handle scrolling or pagination, extract [fields], and include retries and structured logs.
Design a Scrapy pipeline for [domain] with deduplication, incremental updates, validation, and [database] storage. Keep robots.txt enforcement enabled.
Architect an authorized collector for [sources] at [volume]. Include rate limits, queueing, observability, recovery, data lineage, privacy controls, and capacity estimates.
Author
zl2023githubLicense
MIT
Skillstore revision
r2
Version notice
The author did not declare a version.
Repository
https://github.com/zl2023github/software-engineer-skills/tree/main/software-engineering/web-scraperRef
88a8e9a07f4c54ab105c1c41b6267c287146b07b
Maintenance freshness
7/26/2026
Usage
2 downloads · 1 views
File structure
📄 SKILL.md
Diagnose and Improve Full-Stack Performance
Full-stack teams often lack a repeatable way to measure performance and locate bottlenecks. This skill provides testing workflows, diagnostic commands, optimization guidance, and report structures.
Build and Optimize Search Systems
Search teams need reliable guidance across indexing, ranking, semantic retrieval, and evaluation. This skill provides practical workflows and examples for building and improving search systems.
Design Reliable Payment Systems
Payment teams need consistent designs for transactions, channels, reconciliation, security, and failure handling. This skill provides structured guidance, examples, and review checklists.
Build Reliable Business Intelligence Analysis
Business teams often lack consistent metrics and repeatable analysis workflows. This skill guides SQL, Python, dashboards, experiments, and automated reporting from requirements to delivery.
Evaluate Technology Choices with Clear Tradeoffs
Technology choices often hide tradeoffs across performance, cost, reliability, operations, and team readiness. This skill compares candidates, recommends an option, and defines proof-of-concept checks.
Generate Clear Software Architecture Documents
Architecture decisions often remain scattered across requirements, discussions, and diagrams. This skill organizes them into consistent Markdown documents for implementation and review.
Extract Web Data with Firecrawl
by CK991357
Teams often need web content in clean formats without writing custom scrapers. This skill guides Claude, Codex, and Claude Code through Firecrawl modes for scraping, search, maps, crawling, extraction, and status checks.
Build Reliable Playwright Tests
by C0ntr0lledCha0s
Playwright test suites can become flaky, slow, and difficult to maintain. This skill provides practical patterns for locators, fixtures, page objects, debugging, mocking, and configuration.
Create Playwright Web App Tests
by AutumnsGrove
Manual browser testing is slow and often misses regressions across forms, devices, and network states. This skill helps Claude, Codex, and Claude Code create practical Playwright tests for real web workflows.
Build Firecrawl Browser Interactions
by firecrawl
Dynamic pages often hide data behind clicks, forms, and login flows. This skill helps teams choose and scope Firecrawl /interact integrations.
Improve Playwright Test Reliability
by 0xBigBoss
Playwright tests often become flaky, slow, and noisy in agent workflows. This skill provides concise patterns for locators, assertions, fixtures, auth reuse, network mocks, and compact reporters.
Extract Clean Web Content
by 21pounder
Web pages often include navigation, ads, and dynamic content that obscure useful information. This skill guides Claude, Codex, or Claude Code to capture pages and return clean content.