web-scrape
Extract clean content from any webpage
๋ํ ๋ค์์์ ์ฌ์ฉํ ์ ์์ต๋๋ค: 21pounder
Web scraping is time-consuming and error-prone when done manually. This skill uses intelligent content extraction to pull clean, structured content from any URL in seconds. It handles dynamic pages, removes noise like ads and navigation, and outputs in markdown, JSON, or plain text.
์คํฌ ZIP ๋ค์ด๋ก๋
Claude์์ ์ ๋ก๋
์ค์ โ ๊ธฐ๋ฅ โ ์คํฌ โ ์คํฌ ์ ๋ก๋๋ก ์ด๋
ํ ๊ธ์ ์ผ๊ณ ์ฌ์ฉ ์์
ํ ์คํธํด ๋ณด๊ธฐ
"web-scrape" ์ฌ์ฉ ์ค์ ๋๋ค. Scrape https://example.com/blog/post-title as markdown
์์ ๊ฒฐ๊ณผ:
- # How to Build a REST API
- **Source:** https://example.com/blog/post-title
- **Date:** January 10, 2025
- **Author:** Jane Developer
- ---
- REST APIs are the backbone of modern web applications...
- ## Getting Started
- First, install your preferred HTTP client...
๋ณด์ ๊ฐ์ฌ
์์ This skill is a prompt-based wrapper that uses MCP Playwright tools for browser automation. The supporting Node.js script (html_clean.js) performs safe HTML-to-markdown conversion using standard libraries (cheerio, turndown) with stdin/stdout I/O only. No network calls, file writes, command execution, or sensitive data access. Security guidelines explicitly prohibit dangerous behaviors like executing page JavaScript or handling authentication.
ํ์ง ์ ์
๋ง๋ค ์ ์๋ ๊ฒ
Research data gathering
Extract article content, documentation, and research papers from multiple sources into structured notes
API documentation capture
Save API docs and technical content for offline reference or integration work
Content aggregation
Collect and curate content from multiple web sources for analysis or inspiration
์ด ํ๋กฌํํธ๋ฅผ ์ฌ์ฉํด ๋ณด์ธ์
Scrape https://example.com/article and return the content as markdown
Extract product information from https://shop.example.com/product as JSON with title, price, and description
Scrape the documentation at https://docs.example.com/getting-started. Check if there are multiple pages and ask if you should continue
Navigate to https://example.com and take a full-page screenshot saved as example_page.png
๋ชจ๋ฒ ์ฌ๋ก
- Start with the simplest scrape command and add options like --scroll or --screenshot only when needed
- Review the extracted content for accuracy, especially for complex pages with dynamic elements
- Respect website terms of service and robots.txt when scraping content
ํผํ๊ธฐ
- Do not use this skill to scrape login-protected or subscription-only content without authorization
- Do not attempt to bypass CAPTCHAs or access restrictionsโthis will fail and waste resources
- Do not scrape high-frequency or real-time data without appropriate rate limiting
์์ฃผ ๋ฌป๋ ์ง๋ฌธ
What platforms is this skill compatible with?
What are the rate limits?
Can I integrate this with other tools?
Is my scraping activity tracked?
Why did my scrape fail?
How is this different from curl or wget?
๊ฐ๋ฐ์ ์ธ๋ถ ์ ๋ณด
์์ฑ์
21pounder๋ผ์ด์ ์ค
MIT
๋ฆฌํฌ์งํ ๋ฆฌ
https://github.com/21pounder/terminalAgent/tree/main/deepresearch/.claude/skills/web-scrape์ฐธ์กฐ
main
ํ์ผ ๊ตฌ์กฐ