Tech
Vibes Are Not a Benchmark: A 30-Minute Harness to Test Whether a Free Coding Model Can Touch Your Repo
Last month I wrote about the six questions I ask every new model. Those questions are good for a first impression, but they have a blind spot: they tell me how a model talks about code, not whether it can survive contact with my code. So I built a small, ugly, reproducible harness that runs any coding model against five tasks drawn from my own git history, scores the results with my real test suite, and writes the outcome to a CSV. This post is that harness, the reasoning behind it, and wher...
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to