AXe Skills HubSearch /

← All skills

Crawler

CH ClawHub 

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations

Definition

The catalogue holds this skill’s description, not a full SKILL.md — no upstream address was recorded at ingestion, so the body cannot be fetched.

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations

Metadata

Tier
community
Version
1.0.0

Use with an agent

Fetch this skill’s definition over the open API — no key required.

curl -s /v1/skills/community__axehub:ClawHub__ClawHub__Crawler