#evals

My skill evals scored exactly zero for as long as I had records. The lesson was not that the skills were bad. Only a broken instrument writes the same

When building AI agents, it is easy to test randomly. You change a prompt. You run the agent...

An un-seeded LLM judge is a coin flip even at temperature 0. How we made signed eval verdicts replayable instead of attesting noise.

Leaderboards are useful for discovery. They are a weak way to decide what your product should run in...

When evaluating AI agents, we often focus on the final answer. Was it correct? Was it useful? Was it...

Our eval dashboard said 94%. Green checkmark, merge button unlocked, everyone moved on. Three days...

TL;DR — The headline error bounds in this series carry machine-checked proof obligations across four...

TL;DR — Production deployments of the primitive (SCAMP, STUMPY) do not run the textbook sliding...

TL;DR — Three verification gate designs can read green while verifying nothing: an empirical...

TL;DR — The deterministic error bound carries a global magnitude factor T★ — a single worst-case...

Most AI agent failures trace back to missing evals. Here's the 3-level framework — unit tests, LLM-as-judge, and online evaluation — that actually works in production.

The practical evals guide covering task-completion scoring, deterministic vs. LLM-as-judge, regression harnesses, and tool-call fidelity — the full production testing loop that tutorials skip.