Back to articles

Tag archive

#testdiscipline

Y
Nov 12, 2026

Your Private Eval Is the Only Benchmark That Steers You

A 2026 saturation review flagged 29 of 60 widely used LLM benchmarks as high or very high on a saturation index. A null model topped AlpacaEval 2.0 under automated judging. Multilingual contamination studies report leakage up to 91.8% on popular suites. Public benchmarks still shape a shortlist. A private eval built from the team's own domain vocabulary is the signal that moves the ship decision.

Nov 12, 20267 min read0 reactions0 comments
R
Nov 5, 2026

Red-Green-Refactor Was Always a Provenance Rule

Two 2026 empirical studies land the same shape: tests generated from a task description alone catch roughly 25% of injected faults, while tests generated after the model has seen the implementation catch roughly 14%, and chain-of-thought and chain-of-verification prompting do not close the gap. Red-green-refactor's temporal order was never a workflow preference. It was a provenance claim about the test, and the agentic workflow that generates code and then back-fills tests inherits a measurable disadvantage the human discipline was always built to prevent.

Nov 5, 20267 min read1 reactions0 comments
T
Nov 9, 2026

The Agent's Memory Is Not a Test Suite

The 2026 persistent-memory wave for coding agents is landing hard. The empirical shape underneath the pitch is that memory retrieval without validation propagates past wrong fixes: naive similarity-based retrieval showed a 75% false-positive rate on hard negatives, while selective retrieval of only validated repair episodes resolved 29 of 94 tasks against 19 of 94 without memory. Memory is a secondary channel, and when it disagrees with the running test surface the suite wins.

Nov 9, 20267 min read1 reactions3 comments
S
Oct 19, 2026

Self-Healing Tests Are the Vendor Version of the Corrupted Signal

The 2026 self-healing wave sells an auto-repair mechanism for a test's locator when the app changes: the test drifted, the healing engine finds the visually similar element, the bar stays green, the reviewer thanks the vendor. The empirical shape underneath the pitch is a passing test after a semantic behavior change. The flaky-test post named the corrupted signal as a false red; this is the same failure inverted into a false green, and the tooling silences the alarm on the reviewer's behalf. The discipline move is the one MockWorlds and domain-typed tests already made: assert on named domain state and vocabulary the team owns, not on rendered locators the framework happens to expose.

Oct 19, 20268 min read1 reactions0 comments
O
Oct 12, 2026

Over-Mocking Is a Governance Problem, Not a Review Smell

The 2026 empirical study of 1.2 million commits and 48,563 coding-agent commits (arxiv 2602.00409, Hora and Robbes, MSR 2026) found agent test commits add mocks 36% of the time versus 26% for non-agent commits, agents concentrate on a single test-double type, and the resulting suites are more brittle than the human baseline. The precedent is load-bearing; the smell reproduces at generation speed; per-PR review catches at reading speed. The corrective move lives at the convention layer, not in the review pass.

Oct 12, 20267 min read1 reactions0 comments
T
Oct 8, 2026

The Visible Test Suite Is the Ceiling on What the Next Agent Can Deliver

Two 2026 verification-horizon papers land the same measurement: coding agents saturate whatever suite they can see, and the gap between visible pass and actually correct widens with task complexity. The visible test suite is a ceiling on what a search process can converge toward on the codebase it operates against. Waiting for a stronger model moves the agent under the same ceiling faster; it does not raise it. The lever that raises the ceiling is a suite whose discrimination power outruns the search it is trying to guide.

Oct 8, 20268 min read1 reactions0 comments
L
Sep 17, 2026

Lights Out, Part 2: Earning Confidence

You trust an unattended pipeline the way you trust a test suite: through named regressions and fail-closed design, not through watching it. The 2026-05 incident on this blog's own pipeline (a post merged to main as a base64 blob, two weeks of broken builds, a double-publish before the fault surfaced) is the scar this post carries. The fix was not more human review. The fix was deterministic scripts with structural validation gates, atomic all-or-nothing pushes, tests that pin each incident by name, and a veto window replacing the approval queue.

Sep 17, 20268 min read1 reactions0 comments
H
Sep 7, 2026

Harness Engineering Is the Test Suite, Renamed

The industry named a new discipline in early 2026 and packaged it as harness engineering. Read the load-bearing parts of every taxonomy that shipped and the same shape repeats: CI gates, test-suite feedback, deterministic verification loops, sandbox permissions that fail closed. The disciplined test suite is the load-bearing layer inside the harness, and calling the whole stack one new discipline blurs which piece is doing the correctness work.

Sep 7, 20267 min read6 reactions2 comments