Principle · MoneyLaurie Voss argues that evals are infrastructure, not an afterthought. Some teams create evals at the very start of development; others add them later once they are at scale, but either way the important thing is to treat evals as a core part of your system, as routine as unit tests. The value compounds, but only if you keep investing — each time a regression shows up before it reaches your users instead of after, you'll understand why all of this work is worth doing.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyUnit tests don't work on non-deterministic AI output, running a query a few times isn't a test suite, and human review doesn't scale, catch regressions, or run in CI.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyAgents can do something helpful but incorrect and fail silently — giving output anyway while doing unnecessary extra steps without you knowing. Traces are how you detect this.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss explains that the agent decides whether a web search returned enough information and runs an indeterminate number of additional searches until it decides it has enough, then writes the report. Agent execution paths are non-deterministic by design: the same input might search for different things, find different results, and write different reports. Because agent output is non-deterministic, you cannot predict the output from the input alone, and that is exactly why evals are needed. Both reports from the same input might be good, or one might be good and one not.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss warns that if you automate before you understand your failures, you will create an eval that measures what's easy to measure instead of what actually matters. This is why reading traces and understanding failure modes comes before writing evaluators.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyUnit tests don't work for AI outputs because output is non-deterministic — the same prompt produces different text each run, so there's no expected string to assert against. Teams fall back on human review, which doesn't scale, doesn't catch regressions, and doesn't run in CI.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyDespite sounding old-fashioned, the speaker argued that reading raw inputs and outputs is one of the highest-value activities in agent development.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Report · MoneyLaurie Voss explains that every single agent decision is captured as a trace, and each row in the trace table is a span. Traces reveal every decision the agent made; without them all you see is that you gave a prompt and got a report, but with traces you see every step. Click into any span to see exactly what the model received as input and exactly what it returned. Observability in practice means not just whether it worked, but how it worked and where exactly it went wrong. In the demo, the agent ran four web searches before finally deciding that was enough information, then wrote the report.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyThe most popular method of shipping AI applications today is running a few queries, judging if the output looks right, and shipping. The problem is that this fails on inputs you didn't test. Relying on human review to validate outputs doesn't scale, doesn't catch regressions, and doesn't run in CI.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss states that you are not done when your dev experiment passes; you are done when your agent keeps performing in production on traffic you've never seen.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneySkipping judgment of intermediate steps lets errors compound silently — at every step in the chain the chance of a mistake gets higher, so judge each step, not just the final output.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyThe speaker advised multiplying severity by frequency to decide which failures to fix first — even a 1-in-100 failure that is maximally severe (e.g., giving bomb-making instructions) should be fixed first.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyAI features are usually shipped by building the feature, running a few queries, eyeballing the results, and shipping — but this fails on untested inputs, and running a feature three times is not a test suite.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyWithout evals, changing a system prompt to fix one issue can cause unexpected failures elsewhere in the application — you won't catch it until a user reports it. This is whack-a-mole: fixing one problem breaks something somewhere else. Every prompt potentially affects every kind of input users send, because the prompt is a huge block of text sent to your AI.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyShipping AI on vibes is the most popular method today but fails on inputs that weren't tested; real teams like Descript, Bolt, and Anthropic's Claude Code all moved from vibes to evals.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyEvals are testing for AI and traces are logs for AI: evals check whether AI outputs are good (given input, expected output, but outputs are never exactly the same twice), while traces record what the AI did — every agent call, tool call, and LLM invocation with inputs and outputs at each step.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss recommends a practice worth knowing: eval-driven development — write the eval before you build the feature. For example, if you want your agent to always verify customer identity before processing a refund, write the eval that checks for that first.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss argues that you can't evaluate what you can't observe, so observation comes first before running evals. Instrumentation and tracing are the prerequisite for any meaningful evaluation of an agent.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyThe source argues that a transcript that looks correct can hide a failed voice agent call, and that reading only text logs hides latency spikes, interruptions, and misheard audio entirely.
AI Engineer · The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize
Creator perspective · MoneyTranscript logs lie: they don't capture what's actually going on under the hood in voice agent calls. Audio-specific issues like interruptions, latency spikes, and misheard numbers are completely invisible if you only read LM output text. Every one of the demo call's failures — dead air, talking over the caller, the wrong order number — would be invisible in a text log. It's also very easy to look at a transcript and conclude it looks fine while missing that tonality matters, and a flat robotic tone is a bad user experience.
AI Engineer · The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize