
Do AI-Written Tests Actually Catch Bugs? I Measured It
LLMs wrote pytest suites for three Python projects; every suite was scored against one fixed mutant set. Green suites sometimes killed zero mutants.
Tag archive

LLMs wrote pytest suites for three Python projects; every suite was scored against one fixed mutant set. Green suites sometimes killed zero mutants.

Who the hell writes a unit test for an infinite loop? Well, now that's me 😇 When I started working...
100% Coverage, 4% Mutation Score: The Gap That Explains Why Your AI Tests Pass...
I expected a kill to leave all four with a mutated file on disk. Three of them never write a mutant into your tree at all, and the fourth is left dirty by an ordinary SIGTERM, which is a gentler signal than the one I built the suite to test for.
A test that never fails is not a test, and LLM-generated tests often pass because they assert...
Green tests and 100% line coverage can still miss real behavior. I ran C++ mutation testing on a free...
TL;DR: A test can pass without executing the control it names; mutation and path coverage found 59...
In the final stage of my pytest series, I intentionally broke working code to test whether 87 GREEN tests could actually detect failures. Five of 11 manual mutations initially survived.
A green build can still be blind. Line coverage tells us which code ran. Mutation testing changes...
Quando i tool producono centinaia di righe al minuto, la revisione riga-per-riga diventa il collo di...

Coverage measured execution, not verification. Agent-generated tests pass coverage gates while writing assertions that pass even when behavior breaks. Mutation score is the metric coverage was always pretending to be.
An AI-generated test that passes proves nothing until it can go red. Mutation testing exposes the phantom tests, with a Go example to prove it.