# AI Crawler Directives for hiltons-brushes.co.uk # This file tells AI crawlers whether they may scrape this site for training data. # Last updated: 7/3/2026 # OpenAI - GPTBot User-agent: GPTBot Disallow: / # Google - Google-Extended (used for Gemini & other Google AI models) User-agent: Google-Extended Disallow: / # Anthropic - Claude (claude-ai & anthropic-ai) User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / # Common Crawl - CCBot (used by many AI training datasets) User-agent: CCBot Disallow: / # Meta - Meta-ExternalAgent & Meta-ExternalFetcher (used for Llama & other Meta AI) User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / # Perplexity - PerplexityBot User-agent: PerplexityBot Disallow: / # Cohere - cohere-ai User-agent: cohere-ai Disallow: / # Mistral - mistral-ai User-agent: mistral-ai Disallow: / # Bytespider (ByteDance/TikTok AI) User-agent: Bytespider Disallow: / # OAI-SearchBot (OpenAI Search) - ALLOWED so your site appears in ChatGPT/SearchGPT results User-agent: OAI-SearchBot Allow: / # PetalBot (Huawei AI) User-agent: PetalBot Disallow: /