Tech
Using Zstd Frames to Egress Partial Parquet Files
Jump Tables, TLV Footers, and the Real Cost of Reading What You Don't Need
You're paying for bytes you never read.
A data engineer on a busy pipeline touches dozens of Parquet files a day: schema discovery, predicate pushdown, column pruning, metadata scrapes for a data catalog sync. In each case, the application needs maybe 200 KB of context from a file that is 4 GB on disk. Without a seekable archive format and a jump table to find the right frame, your HTTP client fetches the whole thi...
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to