generateAiBotRobotsRules

function generateAiBotRobotsRules(config): string

Defined in: ai-txt.ts:95

Generate a robots.txt fragment with Disallow rules for AI training crawlers based on the ai_training category in the config.

If ai_training.vendors is non-empty, only those crawlers are listed. If absent or empty, all crawlers in AI_TRAINING_CRAWLERS are listed. If AI training is allowed, returns a single comment line so the caller can still concatenate it into their robots.txt without conditional logic.

Append this to your existing robots.txt content.

Parameters

Parameter	Type
config	ConsentConfig

Returns

string

Example

const robotsTxt = `${userExistingRobotsRules}\n\n${generateAiBotRobotsRules(config)}`