# AI Crawler Directives for hiltons-brushes.co.uk # This file tells AI crawlers whether they may access this site for search and training data. # Policy: OPEN to all major AI search & training crawlers to maximise visibility in AI answers. # Last updated: 8/27/2026 # OpenAI - GPTBot (powers ChatGPT training & responses) User-agent: GPTBot Allow: / # OpenAI - OAI-SearchBot (appears in ChatGPT/SearchGPT results) User-agent: OAI-SearchBot Allow: / # Google - Google-Extended (used for Gemini & other Google AI models) User-agent: Google-Extended Allow: / # Anthropic - Claude (claude-ai & anthropic-ai, powers Claude) User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / # Common Crawl - CCBot (used by many AI training datasets) User-agent: CCBot Allow: / # Meta - Meta-ExternalAgent & Meta-ExternalFetcher (used for Llama & other Meta AI) User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / # Perplexity - PerplexityBot (appears in Perplexity AI answers) User-agent: PerplexityBot Allow: / # Cohere - cohere-ai User-agent: cohere-ai Allow: / # Mistral - mistral-ai User-agent: mistral-ai Allow: / # Apple - AppleBot (used for Siri & Apple intelligence features) User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # Microsoft - Bingbot & Copilot AI User-agent: Bingbot Allow: / User-agent: GPTBot-Copilot Allow: / # Bytespider (ByteDance/TikTok AI) - ALLOWED for broader visibility User-agent: Bytespider Allow: / # PetalBot (Huawei AI) - kept restricted User-agent: PetalBot Disallow: /