OpenAI
How do you catch silent regressions when OpenAI updates a model?
If you run anything on the OpenAI API in production, outages are the easy failures. You notice those. The one that gets us is the silent regression: a model gets updated underneath you, the same prompt starts returning slightly different output, and the pipeline keeps going green while quality quietly slips. Nobody gets paged. You hear about it from a user a week later. It sneaks in two ways. If you call an unversioned model name, it can change under you when a new snapshot ships. And if you p...
Read the full discussion on Reddit
This article was aggregated from Reddit. Click to join the conversation.
View on Reddit