We shipped an LLM-powered classification feature for a client last year. It worked well. Three weeks later, after a routine prompt tweak, it started miscategorising a specific edge case — one that the team had explicitly tested for during development. Nobody noticed for four days. The problem was not the prompt change. The problem was that we had no automated check that would have caught it. Our test suite confirmed that the endpoint returned a 200 and that the response was valid JSON. It sai...