Tech
MapReduce for Dummies
MapReduce for Dummies
Say you need to grep for "Hi There" across a petabyte of text spread over thousands of files. A single machine reading that much data sequentially could take hours — I/O alone becomes the bottleneck, before you even count the CPU time to scan every line.
The fix: split the work. Hand different files to different machines (workers) so they scan in parallel, and use a central coordinator to hand out tasks, track progress, and reassign anything that fails.
That's Map...
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to