Here is a rule that looks like it blocks OpenAI's crawler: User -agent: * Disallow: /checkout/ Disallow: /customer/ User -agent: GPTBot Crawl-delay: 10 It blocks nothing. GPTBot reads its own group, finds one Crawl-delay line and no Disallow , and crawls everything — including /checkout/ and /customer/ . Under RFC 9309 , a crawler that matches a specific user-agent group uses only that group. It does not inherit from User-agent: * . Naming a bot anywhere in the ...