
AI Agent Evaluation Metrics: Pass Rate Across Repeat Runs
AI agent evaluation metrics for Claude Code: compute pass@k, pass^k, variance across N reruns and judge-score drift to catch flaky or silently wrong runs.
Tag archive

AI agent evaluation metrics for Claude Code: compute pass@k, pass^k, variance across N reruns and judge-score drift to catch flaky or silently wrong runs.

An AI agent evaluation framework for Claude Code: write tasks and graders, run each several times, and track pass rate and pass^k. Includes a harness.

Claude Code regression testing after prompt changes: capture a baseline, grade both versions, and set a flaky-run threshold before you trust a CLAUDE.md edit.
We installed Inspect AI and attempted an offline scorer contract check, then developed a regression-task walkthrough around its task, solver, scorer, and agent interfaces. The framework is compelling, but we have not validated log reproducibility, grading stability, or model-upgrade cost accounting.

Langfuse published a negative result on auto-improvement. Lyft named the real failure: your synthetic eval user isn't your user.

Жизненият цикъл на разработка на AI агенти (или как претеглих 22 ястия в името на...