AI_TRAINING_CRAWLERS

const AI_TRAINING_CRAWLERS: readonly ["gptbot", "claudebot", "anthropic-ai", "google-extended", "perplexitybot", "ccbot", "bytespider", "applebot-extended", "meta-externalagent", "oai-searchbot"];

Defined in: jurisdictions/vendors.ts:131

AI-training crawlers and LLM-related user-agents.

Listed as consent-required so an explicit opt-in toggle decides whether site content can be crawled / used for model training. Pairs with the upcoming ai_training category and a future /ai.txt / /llms.txt generator. EU AI Act Article 53 enforcement starts August 2026.