A while back I was building a support assistant and hit a simple-sounding question: is this new version of the prompt actually better than the old one? So I did the obvious thing. I wrote thirty test cases, ran both prompts, saw the new one score a little higher, and shipped it. I felt great about it for half a day, until the support queue filled with complaints and I was rolling the change back from a Slack thread that evening. The bump in the score was never real. The test was far too sm...