Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people. GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it request...