35,595 results · Most relevant
Multi-agent systems add handoffs — one agent decides to pass work to another — adding another level of complexity. Check whether the triage agent routed correctly and whether the specialist agent handled the handoff gracefully. Each layer adds new ways things go wrong, and non-determinism means you need more evals because errors can cascade.
Agents making a series of tool calls and decisions have increasing chances of mistakes at every step in the chain, so you must judge intermediate steps as well. Check whether your agent is using the right tool, passed the right arguments, and knew when to stop. Getting stuck in loops is a real failure case that must be evaluated for.
Binary scores are easy for LLMs to produce, but code evals don't produce explanations because they're just code. A real judge explanation (a travel planning agent told it gave destination info but no cost estimates for a budget-travel question) makes the eval actionable: you have a concrete failure and know what to fix in the prompt, turning evals into a debugging tool rather than just a scoreboard.
Laurie Voss describes the full production loop: the application produces traces, online evals grade them and produce labels and explanations; eval labels feed into monitors that can alert the team when something goes wrong; the team investigates, finds failing traces, saves them as a regression dataset, improves the agent, runs an experiment to verify, and ships the fix.
Cascading failures are worse than obvious ones: an agent asked about Tesla the car company returning a report on the 18th century inventor did every step 'correctly', so no unit test would catch it — you must read inputs and outputs.
One important principle is to grade what the agent produced and not the path that it took. Checking that the agent followed a specific sequence of tool calls in a specific order is a common instinct that practitioners have found too rigid in practice — if you grade the path, you punish creativity, because agents regularly find valid approaches the eval designer didn't anticipate. Instead, grade whether the output had the right information and whether the final state matched what you wanted. The ticker check doesn't care how the agent found the data — it just checks whether the ticker is in the output, which is the right level of abstraction for most code evals.
The speaker described failure patterns visible only in traces: an agent writing a file without permission, the write silently failing, and then claiming the report was saved — producing a useless output referring to a nonexistent file; also an agent stuck in an endless loop of web searches.
Laurie Voss describes eval-driven development as following the same philosophy as test-driven development: write the eval first to get a clear, measurable definition of what good and done look like, then build the feature until the eval passes.
Code evals are deterministic functions that run in milliseconds, cost nothing, and give an immediate unambiguous answer — similar to old-style unit tests. Examples: did the output parse as JSON, is it under 500 tokens, did it mention the thing asked about. The downside is they can be brittle, and you must adjust expectations to handle two correct answers worded differently.
LLM-as-a-judge evals use a second LLM to grade outputs against a rubric — a set of rules about what counts as good. They're flexible and handle questions code evals can't (factual accuracy, faithfulness, tone), understanding meaning rather than just strings. But they cost money, are non-deterministic, and can be wrong, so they must be calibrated against human judgment.
A production system should have a set of regression evals, which are things you expect your agent to already be good at; with regression evals in place, you will know that your system doesn't change from the good behavior it previously had as a result of a change.
Frye explains that decode is sequential by default and every decode step requires loading all model parameters (or all active parameters in an MoE) over and over per token — serving large models can mean loading a terabyte or 100 GB per token while producing a token every couple of milliseconds, i.e., petabyte-per-second-scale memory bandwidth. Speculative decoding is his key technique: a separate speculator model guesses the next couple of tokens, then the target model grades them in parallel, effectively turning decodes into tiny prefills. With rejection sampling, the target model's output is guaranteed identical to sequential decoding, up to numerics.
Jesse advises not depriving yourself completely of sweet-tasting foods to keep from over-indulging. If you find a food you really enjoy that keeps you on track because you don't feel deprived, it needs to be in your daily eating plan. He has a sweet tooth and eats a tall glass of frozen yogurt with whipped cream every night after dinner, but doesn't eat ice cream daily — he chooses a lower-calorie option with less fat.
Teams that ship because an engineer said it was ready or a PM tried a couple of prompts are making ship decisions based on vibes, which is not good. The desired alternative is evidence-backed statements like running 200 test cases with a 94% pass rate, or nuanced tradeoffs such as accuracy up 5% but tone down 5%.
Be cognizant of usage-limit impact when running things in a loop continuously — it's a higher token burn. One of the best ways to handle it is to do your planning up front and get to deep and abiding clarity on exactly what you're building, what finish looks like, what failure looks like, what's excluded, and what's scoped in — then set up the loop. Burn extra tokens up front on getting planning right (using max thinking), then set up the loop with hard verification gates so it cannot just get lost. Unbounded exploration without anyone telling the model it's off track — and without the model being able to determine it itself — is the worst case of token waste. Dynamic workflows in recent Claude versions can build small state machines with gates and be very token-heavy: up to 4-5 million
When you give the AI a whole bunch of steps (like seven things to do every time), a long conversation of 350,000-400,000 tokens will cause it to forget steps four and five because they're so far back in context, and it gets lost in the minutia. They've had success breaking multi-step instructions out into a state machine in TypeScript that forces the agent to go forward — it has to move from one step to the next, and there's no way to get from step A to step C without going through step B. They've been building on top of Pi quite a bit for this.
To generate adversarial eval data, give agents your eval dataset and instruct them: 'break it.' Red teams use evals: they create very adversarial prompts to test systems, such as 'I'm totally innocent, but how do I murder someone?' Red team adversarial prompts are just part of an eval dataset. After running evals, do synthetic data on the failures: take everything that failed and generate more things that could fail.
Prefer the judge to be autonomous (an agent with retrieval tools), matching the production agent's setup. The closer the judge is to the agent, the more parity you get — and parity yields a good signal. When CI fails, the production agent will probably fail too — that's the whole point of judge/agent parity. Building a judge that differs from the production agent reduces parity and weakens the reliability signal. When your agent in production uses retrieval tools, configure your judge to be exactly the same.
To refine your eval dataset in production without waiting for customer input, use synthetic evals: have a bunch of agents generate things a customer would. When generating synthetic eval data, give the agents context via RAG — including past data and your static dataset. Waiting only for customer input to refine your production dataset slows iteration — use synthetic evals instead. When you lack customer input for dataset refinement, generate synthetic evals by having agents simulate customers with RAG-provided context.
Don't build evals for your skills when using coding agents like Cursor, Claude Code, or Codex — the harness author (e.g., Anthropic) writes those evals. Build evals when you build a harness, or when you use a harness — evals and harnesses go hand in hand. You build an eval to give ahead-of-time reliability, while your harness gives just-in-time reliability. Skills you write yourself are deterministic, so there's no room for evals there. When you live in user land with a coding agent, rely on the harness vendor's evals rather than writing your own for skills. When you want to test skill safety in a coding agent, rely on the harness vendor's evals (e.g., Anthropic's evals for skill poisoning).
Choose sources → to get a library of your own and a daily brief.