针对AI爬虫的robots.txt配置模板，覆盖GPTBot、Google-Extended、PerplexityBot等。
允许所有AI爬虫访问的配置。
# GEO-friendly robots.txt # Allow all AI crawlers User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml # AI-specific crawlers User-agent: GPTBot Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Bytespider Allow: / User-agent: MoonshotBot Allow: / User-agent: DeepSeekBot Allow: /
仅允许特定AI爬虫，屏蔽其他。
# Selective AI crawler access User-agent: * Allow: / Disallow: /admin/ Disallow: /internal/ # Allow major AI crawlers User-agent: GPTBot Allow: / Disallow: /admin/ User-agent: Google-Extended Allow: / Disallow: /admin/ # Block other AI crawlers User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / # Block unauthorized scrapers User-agent: AhrefsBot Disallow: /
| 爬虫名 | 所属平台 | User-Agent | 建议 |
|---|---|---|---|
| GPTBot | OpenAI | GPTBot | 允许 |
| Google-Extended | Google-Extended | 允许 | |
| PerplexityBot | Perplexity | PerplexityBot | 允许 |
| ClaudeBot | Anthropic | ClaudeBot | 允许 |
| Bytespider | 字节跳动 | Bytespider | 允许 |
| MoonshotBot | 月之暗面 | MoonshotBot | 允许 |
| DeepSeekBot | 深度求索 | DeepSeekBot | 允许 |
| CCBot | Common Crawl | CCBot | 视需求 |
| ChatGPT-User | OpenAI | ChatGPT-User | 视需求 |
部署后需验证AI爬虫能否正常访问。