A short field note. I had to crawl every item in a public registry (~4 million of them) in about a day. One idea did most of the work. The problem Split the sorted keyspace into N ranges, run one worker per range, done. That's the plan. It falls apart because keys aren't uniformly distributed. Cut the alphabet into even chunks and one chunk swallows all the dense prefixes while the rest finish in minutes and sit idle. My first cut had a single shard holding 71% of the work. ...