Tech
We measured a week of inference. Routing by task difficulty cuts our cost per call roughly 48x — and flips which users are profitable.
We did the thing everyone building on LLMs does. We defaulted to a strong frontier model, because the demo has to be good and nobody gets fired for picking the strongest model. Then we measured a week of production traffic, and the numbers were embarrassing enough to write down.
One frontier model was carrying 77 percent of our calls and 97 percent of our inference spend. Not because 77 percent of the work needed it. Because it was the default, and defaults are where money goes to hide.
Th...
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to