Tag archive
A small eval harness for an LLM feature in TypeScript: golden cases, exact checks, an LLM judge with a pass/fail rubric, a CI gate and a tracked score.
Benchmarks and leaderboards are useful, but they answer a different question from the one I usually...
Recently, TypeSafe launched Jev, the first of what people are calling System One or decision models:...
There’s a lot of talk about Jev right now. The idea is simple. An LLM judge writes out a verdict and...
Common Sense Media just called OpenAI's teen-oriented ChatGPT an unacceptable risk for young users -...
What agent evals taught me about moving model floors, noisy LLM judges, and treating the evaluator as...
Every LLM feature I've shipped went through the same loop: tweak the prompt, try five examples, feel good, ship, then get a bug report the five examples never covered. The fix is one we already know from testing. Build the eval set first, from real failures, and make the prompt the thing that has to pass it.
Evaluating agents with Jev has become one of the most discussed ideas in AI testing over the past few...
An AI agent can produce an impressive demo and still be a poor production system. The reason is...
You tweak one line of your agent's prompt to fix a complaint. The fix works. You ship it. Two days...
Before you can evaluate an agent, you need something to evaluate it against. Which test cases you can...
What a Passing Eval Still Doesn’t Tell You Before giving an agent access to a production...