# ===== 搜索引擎与 AI 爬虫抓取协议 ===== # 放行全部搜索引擎与 AI 大模型爬虫,无验证码、无 403 拦截 # 各语种子目录:中文/(无前缀) /en/ /jp/ /kr/ /sp/ /fr/ /pu/ User-agent: * Allow: / Allow: /en/ Allow: /jp/ Allow: /kr/ Allow: /sp/ Allow: /fr/ Allow: /pu/ Disallow: /admin/ Disallow: /admin/login Disallow: /api/ Disallow: /cart Disallow: /*?sessionid= Disallow: /*?utm_ # 新闻分类筛选视图(/news?category=xxx 是 /news 的同内容子集,canonical 已归一,屏蔽以省抓取预算) Disallow: /*?category= # ===== 显式放行 AI 大模型爬虫(GEO 核心策略)===== User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Google-Extended Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / # Anthropic Claude 联网检索/用户代抓(2026 新增独立 UA) User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Amazonbot Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / User-agent: FacebookBot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Baiduspider Allow: / User-agent: DuckDuckBot Allow: / User-agent: YandexBot Allow: / User-agent: Sogou Spider Allow: / # 搜狗官方 token 为 "Sogou web spider"(微信/腾讯元宝联网检索复用搜狗索引) User-agent: Sogou web spider Allow: / # 360 搜索/纳米 AI(360 智脑检索源) User-agent: 360Spider Allow: / # 神马搜索(UC/夸克,阿里移动搜索;通义千问联网检索复用) User-agent: YisouSpider Allow: / # 华为花瓣搜索(盘古大模型检索源) User-agent: PetalBot Allow: / # 科大讯飞星火大模型爬虫(业界观察期 UA,命中即放行) User-agent: iFlytekSpider Allow: / # Brave 搜索(独立索引,AI 答案源之一) User-agent: BraveBot Allow: / # 社交/职业网络抓取(品牌实体一致性与链接发现) User-agent: LinkedInBot Allow: / User-agent: Discordbot Allow: / User-agent: Slurp Allow: / # ===== 新一代 AI 搜索/代理爬虫(2026 最佳实践:显式声明引用许可)===== User-agent: OAI-SearchBot Allow: / User-agent: meta-externalagent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / User-agent: YouBot Allow: / User-agent: MistralAI-User Allow: / User-agent: DeepSeekBot Allow: / User-agent: AI2Bot Allow: / User-agent: Diffbot Allow: / User-agent: Timpibot Allow: / User-agent: iaskspider Allow: / User-agent: ImagesiftBot Allow: / # ===== 七语种 Sitemap 入口 ===== Sitemap: https://www.jojomen.com/sitemap-zh.xml Sitemap: https://www.jojomen.com/sitemap-en.xml Sitemap: https://www.jojomen.com/sitemap-jp.xml Sitemap: https://www.jojomen.com/sitemap-kr.xml Sitemap: https://www.jojomen.com/sitemap-sp.xml Sitemap: https://www.jojomen.com/sitemap-fr.xml Sitemap: https://www.jojomen.com/sitemap-pu.xml