I ran a scraping platform that processed millions of pages a day at roughly 95% extraction success, around three seconds per page. The fetch-and-parse code, the part everyone thinks of as "the scraper", was a tiny slice of the whole thing. The years went into everything around it. Here's what actually broke, more or less in the order it broke. The queue with no manners Crawling is bursty in a way that surprises you the first time. One category page fans out into a few hundred ...