# Extract Web Data with Crawl4AI

Web pages are hard to convert into clean, reusable data. This skill helps Claude, Codex, and Claude Code use Crawl4AI for crawling, markdown, and extraction workflows.

## Install

```bash
npx skillstore add smallnest/crawl4ai
```

## Metadata

- Status: approved
- Slug: smallnest-crawl4ai
- Skillstore revision: r1
- Version status: missing
- Tree hash: d9b77aa5549f10dbdcea25111f0a20d01704f1301c5541cba4222ad7e4dc682e
- Author: smallnest
- GitHub username: smallnest
- License: MIT
- Repository: https://github.com/smallnest/crawl4ai-skill/tree/master/
- Ref: 3e4b6c31a74a3bd1a291c98cf585d720cb9fbc88
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: confirmation\_required
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: network, filesystem, env\_access, scripts, external\_commands
- Quality score: 38
- Quality tier: warning
- Public page: https://skillstore.pages.dev/skills/smallnest-crawl4ai
- Manifest: https://skillstore.pages.dev/api/skills/smallnest-crawl4ai/manifest

## Capabilities

- Runs Crawl4AI CLI examples for single-page crawling and markdown output.
- Shows Python SDK patterns for AsyncWebCrawler and CrawlerRunConfig.
- Supports CSS schema extraction for repeatable structured data tasks.
- Provides batch crawling examples for multiple URLs with concurrency settings.
- Explains JavaScript waits, sessions, screenshots, proxies, and content filters.
- Includes helper scripts for basic crawling, batch crawling, extraction, and Google result parsing.

## Use Cases

- Convert Documentation Sites: Turn public documentation pages into clean markdown for analysis, migration, or knowledge base preparation.
- Monitor Product Listings: Use CSS extraction schemas to collect product names, prices, links, and descriptions from permitted sources.
- Prototype Crawling Pipelines: Test Crawl4AI CLI and SDK settings before building a larger automated data pipeline.

## Prompt Templates

### Crawl One Page

```
Use Crawl4AI to crawl this URL and return clean markdown. Confirm the URL before running any command: <url>
```

### Extract Repeating Items

```
Create a Crawl4AI CSS extraction plan for this page. Extract titles, links, dates, and summaries from repeated items: <url>
```

### Batch Crawl Sources

```
Prepare a Crawl4AI batch crawl for these approved URLs. Save markdown outputs and summarize failed pages: <urls>
```

### Build Extraction Pipeline

```
Design a Crawl4AI pipeline that generates a schema once, reuses it without LLM calls, and validates extracted fields for: <task>
```

## Limitations

- Requires Crawl4AI, Playwright browsers, and network access in the local environment.
- Some LLM extraction examples require a configured provider API key.
- Website selectors can break when target pages change their HTML structure.
- Users must confirm they have permission to crawl each target site.

## Best Practices

- Confirm permission, scope, and rate limits before crawling any site.
- Prefer CSS extraction schemas for repeatable pages before using LLM extraction.
- Write outputs into a dedicated working directory and review files before reuse.

## Anti Patterns

- Do not crawl authenticated or protected pages without explicit authorization.
- Do not use proxy or anti-detection settings to bypass site rules.
- Do not store API tokens in shared files or commit generated output.

## Security Audit

- Audited at: 2026-07-09T07:55:13.922\+00:00
- Summary: Most static findings are false positives from Markdown documentation, placeholder API-key examples, and code-fence backticks. Confirmed risks are expected crawler side effects: local result writes, Google Search requests, unrestricted user-supplied URLs, and dual-use anti-detection guidance. Static review was capped at 400/959 representative findings; omitted static matches are unconfirmed, so automatic publishing stays disabled until manual review.

## Stats

- Views: 252
- Downloads: 243
- Favorites: 8
- Popularity score: 0