技能 web-scraper

📦

web-scraper

Name: web-scraper
Author: zl2023github

内容修订 r2 高风险 ⚙️ 外部命令🌐 网络访问📁 文件系统访问🔑 环境变量

Build Reliable Web Data Collectors

Collecting reliable data from static pages, dynamic sites, and APIs requires several tools and careful engineering. This skill provides implementation templates for extraction, cleaning, storage, monitoring, and scaling.

支持: Claude Codex Code(CC)

⚠️ 38 差

安装前请停下并请求确认。

请审查安装计划，并在更改文件前获得用户明确同意。

使用我的智能体安装

将此请求复制给你的智能体，其中包含权威 Skill 页面和清单。

智能体请求

Review the Skillstore skill "web-scraper" from https://skillstore.io/skills/zl2023github-web-scraper.md and its manifest at https://skillstore.io/api/skills/zl2023github-web-scraper/manifest. Verify the artifact. Stop and obtain explicit user consent before installing or changing files.

你的智能体仍应展示其计划，并按安全策略请求必要确认。

Agent 可读资源

当 AI Agent、爬虫或脚本需要干净上下文，而不是读取完整页面时，请使用这些链接。

Markdown 详情 GET /skills/zl2023github-web-scraper.md 已签名 manifest GET /api/skills/zl2023github-web-scraper/manifest 已签名 lockfile GET /api/skills/zl2023github-web-scraper/lockfile

测试它

正在使用“web-scraper”。 Extract article titles, publication dates, and links from an approved news archive.

预期结果:

A normalized table with title, publication date, source link, validation status, and collection timestamp.

正在使用“web-scraper”。 Design a recurring collector for a permitted dynamic catalog with daily updates.

预期结果:

Playwright extraction flow with stable selector guidance
Incremental update and deduplication strategy
Rate limits, retries, logs, and health checks
Validated records prepared for database storage

正在使用“web-scraper”。 Assess a proposed Scrapy deployment for several authorized sources.

预期结果:

A deployment plan covering workers, Redis scheduling, storage, monitoring, failure recovery, data validation, and source-specific limits.

安全审计

高风险

v4 • 2026/7/24 打开版本化报告

The command-execution alerts are false positives caused by Markdown code fences, and the credential alerts are ordinary parameters or a localhost Redis example. The skill does perform network requests and writes intercepted responses to disk. High-risk semantic concerns include anti-bot circumvention guidance and capture of potentially sensitive application traffic.

已扫描文件

749

分析行数

复核项

已忽略误报

已确认安全问题 (3)

高

SKILL.md:282-296 SKILL.md:381-458 SKILL.md:581-590

Anti-Bot Control Circumvention Guidance

The skill teaches proxy rotation, CAPTCHA solving, browser fingerprint concealment, WAF avoidance, and disables Scrapy robots.txt enforcement. These techniques can bypass access controls and site policies.

The prose and code explicitly recommend defeating CAPTCHA, WebDriver detection, WAF controls, and robots.txt enforcement.

高

SKILL.md:515-533 SKILL.md:625-644

Sensitive Application Traffic Capture

The mitmproxy examples print intercepted request headers and bodies and save complete responses. Authorization tokens, personal data, and confidential application content may be exposed.

The examples directly print headers and body content and persist raw responses without redaction or access controls.

中

SKILL.md:351-375

Untrusted Public Proxy Ingestion

The proxy pool downloads a mutable public list and routes requests through arbitrary entries without integrity checks. Malicious proxies can observe metadata or manipulate traffic.

The code fetches a remote public proxy list and immediately constructs routing configuration from its unverified entries.

能力复核项 (12)

这些是真实的本地能力，对此技能可能是预期行为，因此需要复核，但不会按已确认恶意行为计分。

高

SKILL.md:515

HTTP client library

def request(flow: http.HTTPFlow):

This mitmproxy callback inspects live application requests, including URLs, headers, and bodies. It creates network-data exposure risk when traffic contains secrets or personal data.

高

SKILL.md:625

HTTP client library

def request(flow: http.HTTPFlow):

This mitmproxy callback handles intercepted network requests and prints selected request bodies. Sensitive application data could be exposed through logs.

高

SKILL.md:643

Python file write/append

with open(fname, 'w') as f:

The callback writes complete intercepted response text to disk. This can persist credentials, personal data, or other sensitive application content.

中

SKILL.md:641

Python os file operations

os.makedirs(OUTPUT_DIR, exist_ok=True)

The interception callback creates a local output directory before saving captured responses. This enables persistent storage of potentially sensitive network data.

低

SKILL.md:97

Python HTTP libraries

resp = requests.get(url, headers=headers, timeout=10)

The example performs an HTTP GET to a supplied URL. Network access is intentional and can contact arbitrary user-selected hosts.

低

SKILL.md:262

Python HTTP libraries

resp = requests.get(base_url, params=params, headers=headers, timeout=15)

The API collector performs repeated HTTP GET requests to a supplied base URL. It may send a bearer token to that selected endpoint.

低

SKILL.md:359

Python HTTP libraries

resp = requests.get(source, timeout=5)

The proxy loader downloads a mutable public proxy list from GitHub. This is real external network access to untrusted operational data.

低

SKILL.md:375

Python HTTP libraries

resp = requests.get('http://httpbin.org/ip', proxies=proxy, timeout=5)

The code sends a test request to httpbin through a selected public proxy. Both the external request and untrusted routing are intentional.

低

SKILL.md:354

Hardcoded URL

'https://raw.githubusercontent.com/proxifly/free-proxy-list/main/proxies/proxies.json',

The URL points to a mutable GitHub-hosted public proxy list that the example downloads. Compromise or manipulation could redirect scraper traffic through hostile proxies.

低

SKILL.md:363

Hardcoded URL

'http': f'http://{p["ip"]}:{p["port"]}',

The code constructs an HTTP proxy endpoint from remotely supplied host and port values. Requests routed through such untrusted proxies may be observed or altered.

低

SKILL.md:364

Hardcoded URL

'https': f'http://{p["ip"]}:{p["port"]}',

The code configures HTTPS traffic to use an HTTP proxy built from remotely supplied values. An untrusted proxy can observe destinations and disrupt traffic.

低

SKILL.md:375

Hardcoded URL

resp = requests.get('http://httpbin.org/ip', proxies=proxy, timeout=5)

The hardcoded httpbin endpoint is used for a real external proxy test. It discloses connection metadata to the endpoint and selected proxy.

风险因素

⚙️ 外部命令 (35)

🌐 网络访问 (11)

SKILL.md:515 SKILL.md:625 SKILL.md:97 SKILL.md:262 SKILL.md:359 SKILL.md:375 SKILL.md:159 SKILL.md:354 SKILL.md:363 SKILL.md:364 SKILL.md:375

📁 文件系统访问 (2)

SKILL.md:643 SKILL.md:641

🔑 环境变量 (4)

SKILL.md:502 SKILL.md:249 SKILL.md:255 SKILL.md:256

审计者: codex 查看审计历史 →

分享与引用此报告

分享版本化评估报告、中性徽章、嵌入卡片和引用。Skillstore 只报告证据，不判断该 Skill 是否安全。

打开版本化报告

复制报告链接

https://skillstore.io/skills/zl2023github-web-scraper/audits/4?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_report

Markdown 徽章

[![Skillstore security assessment](https://skillstore.io/badges/skills/zl2023github-web-scraper/security.svg)](https://skillstore.io/skills/zl2023github-web-scraper?utm_source=security_passport_badge)

HTML 徽章

<a href="https://skillstore.io/skills/zl2023github-web-scraper?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/zl2023github-web-scraper/security.svg" alt="Skillstore security assessment" loading="lazy"></a>

嵌入卡片

<iframe src="https://skillstore.io/embed/skills/zl2023github-web-scraper.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>

学术引用 (APA · BibTeX · CFF)

APA 引用

zl2023github. (2026). web-scraper security audit report (audit version 4) [Author version unspecified]. Skillstore. https://skillstore.io/skills/zl2023github-web-scraper/audits/4

BibTeX 引用

@techreport{zl2023github-zl2023github-web-scraper-2026,
  author = {zl2023github},
  title = {web-scraper security audit report (audit version 4)},
  institution = {Skillstore},
  year = {2026},
  number = {4},
  url = {https://skillstore.io/skills/zl2023github-web-scraper/audits/4},
  note = {Author version unspecified}
}

CITATION.cff

cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "web-scraper security audit report (audit version 4)"
version: "unspecified"
type: report
authors:
  - name: "zl2023github"
date-released: "2026-07-24"
url: "https://skillstore.io/skills/zl2023github-web-scraper/audits/4"
identifiers:
  - type: other
    value: "skillstore:zl2023github-web-scraper:audit:4"
    description: "Skillstore immutable audit report identifier"

Skillstore 评分

评分依据证据置信度: 中

架构

可维护性

内容

社区

规范符合性

你能构建什么

Collect Public Research Data

Build a rate-limited collector for permitted public pages, then normalize and validate the extracted records.

Automate Dynamic Catalog Extraction

Use Playwright to load dynamic product pages, handle scrolling, and extract changing catalog fields.

Operate a Scalable Collection Pipeline

Design Scrapy workers, Redis scheduling, storage, monitoring, retries, and data quality controls for authorized sources.

试试这些提示

Create a Basic Scraper

Create a polite scraper for [URL] that extracts [fields]. Respect robots.txt, use a one-second delay, and return a table.

Handle Dynamic Content

Build a Playwright scraper for [URL]. Wait for [selector], handle scrolling or pagination, extract [fields], and include retries and structured logs.

Build a Data Pipeline

Design a Scrapy pipeline for [domain] with deduplication, incremental updates, validation, and [database] storage. Keep robots.txt enforcement enabled.

Architect Distributed Collection

Architect an authorized collector for [sources] at [volume]. Include rate limits, queueing, observability, recovery, data lineage, privacy controls, and capacity estimates.

最佳实践

Confirm authorization, service terms, robots.txt rules, privacy limits, and retention requirements before collecting data.
Start with small samples, conservative concurrency, explicit timeouts, retries, and source-specific rate limits.
Preserve data lineage, validate required fields, monitor structural changes, and redact sensitive logs.

避免

Do not bypass CAPTCHA, WAF, authentication, or browser controls without explicit written authorization.
Do not route credentials or sensitive records through public proxies or unverified infrastructure.
Do not store raw intercepted traffic without redaction, encryption, restrictive permissions, and retention limits.

常见问题

Can this skill scrape JavaScript-rendered pages?

Yes. It provides Playwright patterns for loading pages, waiting for selectors, scrolling, and extracting rendered content.

Does it support API collection?

Yes. It includes paginated request patterns, optional bearer authentication, timeouts, response validation, and record aggregation.

Can it build distributed crawlers?

It provides Scrapy and Redis scheduling examples, but production infrastructure, capacity planning, and operations remain your responsibility.

Does the skill ensure scraping is legal?

No. You must verify authorization, service terms, robots.txt rules, privacy obligations, copyright, and local laws.

Can it inspect application traffic?

It includes mitmproxy examples for authorized testing. Captured headers, bodies, and responses may contain sensitive data and require protection.

What must be customized before use?

Configure domains, selectors, schemas, authentication, storage, rate limits, retries, monitoring, privacy controls, and retention policies.

开发者详情

作者

zl2023github

许可证

MIT

Skillstore 修订

版本提示

作者未声明版本。

仓库

https://github.com/zl2023github/software-engineer-skills/tree/main/software-engineering/web-scraper

引用

88a8e9a07f4c54ab105c1c41b6267c287146b07b

维护新鲜度

2026/7/26

使用量

2 次下载 · 1 次浏览

文件结构

📄 SKILL.md