AI
Show HN: Parsimony – an open-source benchmark for how much code AI models add
Existing benchmarks measure solve rate and token output/cost, but don't punish excess code, which makes a codebase harder to work with in the long term. I wanted to measure this since it matters most for long-term projects. Method: it takes the patches that benchmarks already publish (DeepSWE, SWE-bench Verified, SWE-bench Live) and counts "coding units": syntax elements like statements, calls, names and literals. I didn't use lines of code, so cramming code isn't rewarded. Deleted code is subtr...
Read the full discussion on HackerNews
This article was aggregated from HackerNews. Click to join the conversation.
View on HackerNews