35,848 results · Most relevant
Kent relays Jack Ryan's suggestion to set up a Slack channel or similar mechanism to collect all user feedback in one place — for example a channel with high-value customers whose decisions you really respect, or one that aggregates Reddit threads, X, and other social media into one place. You don't have to answer everything in it; that's not necessarily your job.
Install the open inference instrumentation Claude-agent SDK package to auto-capture traces from Claude Agents without changing your application code. AX is compatible with OpenAI's agent SDK and whole agent frameworks like CrewAI, LangChain, and Mastra — no matter what framework you built your agent in, it is already instrumented and you can just turn on logging. You can also use OpenAI instead of Claude or some framework — packages exist for all of those and you just need to swap them in.
Laurie Voss recommends writing one code eval next, since code evals are easier, faster, and cheaper than LLM evals, and making it for the thing that matters most: correctness, faithfulness, whatever fits your use case. Then run your evals, look at the results, see what patterns emerge, and build from there.
You don't have to do all of it at once: start by adding two lines of instrumentation and reading your traces for immediate insight, then write one code eval for the thing that matters most.
Laurie Voss advises that you don't have to do all of it at once; you can start very, very small. Start by reading your traces: add those two lines of code and start looking at them — that immediately gives you information about what your agent is up to that you didn't know. 15 minutes of reading real outputs will teach you more about your application than an hour of building dashboards.
Vary the phrasing, complexity, and level of specificity in synthetic test queries — 'research Tesla financial report performance', 'What's going on with Tesla stock', and 'Yo, is Tesla a buy right now' are the same intent with very different looks.
The speaker advised generating multiple phrasings of the same intent and varying complexity and specificity — e.g., 'research Tesla financial report performance', 'What's going on with Tesla stock', 'Yo, is Tesla a buy right now' — to match the range of what users will actually ask.
Laurie Voss notes that LLM judges cost money and at production scale those costs add up, so you don't need to run evals on every production trace — sample 10% or 1% of traffic to get a directional signal without running an expensive LLM judge on every instance.
If you only test positive cases (does it search the web), you end up with an agent that always performs the behavior — also test 'did it not search the web for this very obvious answer where it didn't need to'.
Failing to label examples with XML tags causes the LLM to treat the example as the thing to write and spit it back verbatim. Telling the judge to spit out a specific string like 'correct' or 'incorrect' is more likely to confuse it and make the eval fail — the judge instead calls a tool with correct or incorrect when done.
Laurie Voss recommends telling the coding agent to find themes and not chase individual failures: it's tempting for Claude Code to write a whole bunch of code to fix one particular failed trace when there are actually 10 failing traces with the same cause it should focus on instead.
Laurie Voss recommends using one model provider (e.g., OpenAI) for the agent and a different one (e.g., Anthropic) for judging, so the judge is less likely to agree with the generator due to self-preference bias. He notes he used Sonnet as a judge for Haiku-generated output for this reason, and that using a different lab's model entirely works even better.
Laurie Voss recommends running evals at the span level to evaluate a particular LLM or tool call every time, or at the trace level where the evaluator sees the full tree from start to finish — pick the scope that matches the question you're asking.
Hand a whole pile of eval explanations to a coding agent and ask it what it could do to improve — it will improve the application for the next run. You can also get your coding agent to do everything you do by hand today using the Arize Skills.
Trace voice agents and see each individual span with live audio playable in line from the trace. Attach metrics to voice sessions: latency, time to first audio, interruptions, and sentiment analysis. When debugging, look at inputs and outputs including the transcript plus specific metrics like time to first token, audio token costs, and interruption events. Don't view trace data as a flat list — look at the linkage between spans and events, and stop looking at disparate events from distant systems; look at the entire trace tree for everything that happened in the session.
The source recommends flagging interruptions, accuracy, transcription drift, and task success as audio-specific eval metrics.
The source recommends creating audio evals via natural language prompting with Claude, specifying sentiment analysis with GPT audio, and catching frustrated user tone on audio spans as an eval target.
The source recommends viewing the entire trace tree for everything that happened in a session rather than disparate events from distant systems, including session lifecycle, audio input, transcript, conversation items, output, response lifecycle, and token counts.
The source recommends viewing tool call execution in line with the rest of the audio to debug tool failures, and that when a voice agent calls a tool, view the tool call in line with the audio.
The source recommends replaying the whole conversation using the session ID to stitch every turn into a timeline, and to see who spoke when, when the agent paused, and where the user got cut off.
Choose sources → to get a library of your own and a daily brief.