35,595 results · Most relevant
The speakers argue that tools are what allow an agent to physically interact with the computer and verify its own work — checking whether a join actually worked, whether storage updated, or whether a command ran start to finish. Without these tools, the agent cannot see the results of its actions and must rely on human interpretation.
Nick Nisi frames the purpose of verification tooling as reaching a trusted 'done' state faster and with fewer interaction turns with the agent. The moment an agent realizes it is not done and refuses to drop the work on the floor is when the entire verification system pays off. He also emphasizes giving agents tools to prove to themselves they're done and to show evidence the human can trust, so returning to the work doesn't require re-prompting and restarting the loop.
The speakers endorse that verification tools create reproducible evidence that acts as a regression check — next time the page changes, the agent can run it again and verify nothing broke, keeping the team moving forward.
The speakers endorse that once you build durable, reliable, repeatable, uniform loops, failure starts to drive future work — the agent determines on its own that something is wrong and takes another stab. If an agent fails in a loop while you're busy, it can self-correct: run the command, verify again, and confirm it's good without you coming back to the PR at end of day. They call agents self-healing on single lanes of work foundational for speed and velocity.
Nick Nisi explains that relying on prompts alone without verification tools yields the confabulations LLMs are designed to give you, because they process text and try to make you happy. For every detail you don't fill in about the desired outcome, the agent will infer or make something up, and its assumption may not match yours. An agent without clear goals will spend tokens and might reach the expected end, but there's no guarantee of it.
Nick Nisi argues that every failed verification run is valuable: each miss adds a new line to the failure map, reveals why the agent didn't reach the goal, and gets the team one step closer to a fix because the failure can be captured and repaired. He also notes that once a bug like the wrong-email bug is caught and proven, it won't recur.
The speakers endorse that the goal is not to wrap determinism around LLMs — agents are intelligent and capable and you want them to be creative. With the right constraints, agents come up with creative solutions that actually solve the stated problem (constraints can equal freedom). They recommend giving agents the outcome rather than prescribing how they achieve it (outcome-driven development), noting being overly prescriptive can lead to lower performance in some cases.
The speakers frame verification not as an optional add-on but as a foundational concept that benefits the whole team. When agents can prove their own correctness, the entire team moves faster and ships safer, more reliable software. Without it, teams get stuck in a loop of the agent claiming 'done', the human re-checking, and re-prompting while juggling other agents.
Nick Nisi explains that once you build verification loops that are durable, reliable, repeatable, and uniform, the main benefit is that a failure starts to drive future work on its own — without such loops, a failed agent run cannot drive future work and requires human intervention.
Nick Nisi recommends maintaining checks for everything that came before — unit tests, evals, smoke tests, and whatever the application needs — and giving the model the ability to interpret and run all previous checks so that when new features are added, all prior checks run and catch regressions. He also suggests deploying to a preview environment first and verifying the change there before promoting to production, then immediately monitoring after promotion.
If you have verifiers in place, your source code doesn't matter as much because you can rewrite it and prove the new version works.
Zack Proser argues that constraints can equal freedom: if agents have the right constraints, they will come up with creative solutions and actually solve what you said. That's why you give them the outcome — he points to Nick's project experimenting with outcome-driven development. He also notes that teams might score differently or have different criteria, so the approach is super flexible.
Laurie Voss notes that once registered, every call your agent makes is traced automatically — you don't have to write a log line or trace for every single thing the agent does. The Claude Agent SDK already instruments tools and web searches for you, and you don't have to add any logging to your agent for tracing to work — all the useful information shows up without doing anything.
Laurie Voss explains that without traces you would only get a mysterious output saying something was wrong; traces are where you can see the tool call failing and the agent trying to do something it isn't given permission to do.
Laurie Voss argues that defining good is not solely an engineering job: if you are defining good for your product, you have to talk to the non-technical stakeholders in your organization because they are the domain experts who know what good looks like. Engineers have specific guesses and biases about what makes a good agent and good output, and won't always agree with the domain experts. Customer-facing people need to be in the room when you define what your agent looks like.
Laurie Voss warns that if you automate before you understand how your agent fails, you'll end up measuring what's easy to measure instead of what actually matters.
LLM judges have all the flaws of LLMs: unpredictable, never the same twice, can make mistakes, cost money, and can be confidently wrong. Because they can be confidently wrong, you have to check them.
Laurie Voss explains that changing a system prompt can silently break other behaviors, creating a whack-a-mole cycle of regressions, and that evals fix this by giving a trackable number.
Laurie Voss states that synthetic data is a great way to get started testing an agent, and once you have real users with real traffic, real traffic is the gold standard for testing an agent — get it incorporated into your dataset as fast as you possibly can.
Evals exist to solve the problem of knowing whether your agent is good and whether a change made things better or worse. They let you stop regressions when making changes without reading every output yourself, and fix the whack-a-mole problem by giving you a number you can track, compare, and act on.
Choose sources → to get a library of your own and a daily brief.