35,595 results · Most relevant
Coach Kyle frames early questions from a woman (like 'how old are you?' or 'do you do this a lot?') as tests to expect and handle, viewing them as signs of investment rather than attacks, and recommends responding with humor, playful stories, and displayed intent instead of defensiveness.
Kent summarizes that getting down to the why-why-why and turning it into a job statement helps ensure you're building the right thing, and that understanding the job probably means you don't need the complexity of the originally requested solution (e.g., facial recognition).
Kent argues that a requested feature (like facial recognition) is not the job itself — the request and the job are two different things — and that repeatedly asking 'why' digs past the request down to the underlying job to be done.
Changing a system prompt to fix one issue (e.g., tone) can cause unexpected problems elsewhere (e.g., hallucinating product features). Every prompt potentially affects every kind of input users send, and since the prompt is a huge block of text, you can't manually test every combination. Without evals you're playing whack-a-mole; with a faithfulness eval you'd catch a hallucination spike before shipping.
Borrowing the Swiss cheese model from safety engineering, the speaker argued for stacking multiple layers of defense: code evals catch format issues but miss semantic problems; LLM judges catch reasoning gaps but miss subtle hallucinations; human review catches the subtle stuff but cannot scale. No single eval catches everything — the combination gives coverage.
Laurie Voss argues the real value lies in the eval-iterate cycle — run an experiment, change your prompt, run the experiment again, over and over — not in the score itself. The cycle: run the evals, look at the failures, read the explanations, identify a pattern, and adjust your prompt (or get Claude to adjust it for you). 'Did it get better?' is a question that can only be answered by evals, and evals let you move from 'I think it's working' to 'I can prove it's working.'
When you cannot say what 'works' means, you cannot say the agent doesn't work; for a financial analyst agent, ask 'What does a good report look like?' to define success criteria.
When you change a prompt and rerun, you must check both whether the fix worked and whether it broke something that was previously working — changing a system prompt to fix one issue can cause hallucinations elsewhere in the application.
Laurie Voss lays out the full loop for building AI products: define what you want, build it, measure how well it works, ship it, monitor it in production, and iterate based on what you see. The condensed loop is instrument, trace, eval, and iterate — each step feeds the next. For AI products specifically, the measurement step is where most teams fall down, and evals are how you measure.
The full loop to build is: instrument, trace, eval, and iterate. Define what you want, build it, measure how well it works, ship it, monitor it in production, and iterate based on what you see.
Laurie Voss argues failures should seem fair: when a task fails, it should be clear what the agent got wrong and why (a principle from Anthropic's eval teams). The problem with one-off fixes is you don't know whether the fix actually worked or whether it broke something that was working before — if you just run the agent again on a couple of examples and eyeball it, you're back to vibes. You need a structured way to compare before and after when making fixes, which is what experiments are for.
Without evals, switching models means weeks of manual testing to determine if the new model is better or worse. With evals, you run the suite, compare the scores, and know within hours. The field moves fast with new models dropping every few months, which makes evals matter more and more.
Laurie Voss explains that the power of experiments is controlled comparison: take the same inputs and the same evaluators so the only thing that changed is the agent's prompts. With controlled comparison, any difference in scores is attributable to your change. You're still wondering a little whether a score change came from your prompt change, because the agent is non-deterministic and could have done 10 web searches instead of five — but controlled comparison eliminates a major source of variation, which is the test cases.
The speaker argued that if you automate evals before understanding your failures, you create an eval that measures what is easy to measure instead of what actually matters.
Laurie Voss argues that you need requirements first — before you can categorize failures, you need to know what success looks like. You can't say an agent doesn't work if you haven't defined what 'works' means. Success criteria seem obvious when written down, but lots of teams never write them and instead ship an agent and react to complaints. Writing requirements down turns vague disappointment into specific testable criteria, and those criteria turn into your evals.
If you don't derive eval criteria from observed errors in traces, the right criteria won't occur to you — you can't come up with them from whole cloth.
You can't switch models without evals — the field moves fast with new models dropping every few months. Without evals, switching means weeks of manual testing with no certainty; with evals, you run the suite, compare scores, and know within hours whether a new model is better.
The speaker observed that most agent outputs appear fine at a glance, but failures are only visible when reading the traces — in his sample, three of thirteen traces were doing something weird.
Evals turn a vague definition of good into a measurable bar, so 'did it get better' can be answered with proof rather than vibes.
The source argues that debugging voice agents requires seeing the conversation beyond just the transcript: audio, transcript, and trace all in one place.
Choose sources → to get a library of your own and a daily brief.