We stored account balances as a mutable column and updated them on every transfer. It worked fine until a race condition double-processed a batch of payouts, and we discovered we had no way to independently verify what the correct balances should have been. We rebuilt the system around an append-only ledger, where balances are computed from history rather than stored directly, backfilled six weeks of parallel verification, and only then retired the old column. Here's what broke, why it bro...