35,595 results · Most relevant
Laurie Voss recommends grading the tool calls of agents, not just the final output. Tool selection, tool invocation, and tool response handling are the decisions that can make or break an agent, and they all happen before the report is read.
Use LLM judge explanations to identify what to fix in your prompt. At scale, collect many error explanations (50, 100, 1000), feed them to another LLM to detect patterns, or feed them directly to your agent so it can fix the systematic problem itself.
Laurie Voss says you don't have to jump straight to writing a custom rubric; you can write code evals and use built-in evals to get a bunch of obvious stuff done first. The simplest useful code eval he could think of: check whether the report actually mentions the ticker symbol of the company being analyzed. A report about Tesla that writes about Microsoft is a completely useless report — the agent did this on at least one occasion. The ticker check eval function gets the input and output, splits out the tickers asked about, and checks that the expected ticker is present.
Laurie Voss explains that an agent eval file looks like an ordinary test file because it is one — a describe block with a test for each query in the regression set. Each test runs the agent, logs the report as the output, and then runs the same evals on it — you don't have to add new evals for CI. At the bottom, you gate the build on acceptance criteria, which is where guardrails and north stars go. You can run npm test and it runs like any test suite, and at the end the reporter prints out a scorecard.
Laurie Voss explains that step five is writing your own judge — this is where evals get specific to your application. But you don't have to jump straight to writing a custom rubric: you can write code evals to get a bunch of stuff out of the way first, and use built-in evals to do a bunch of obvious stuff without having to invent your own tool calling correctness evaluator. For example, there is no built-in eval that checks for a buy, sell, or hold recommendation in a financial report because that would be a very specific thing.
Laurie Voss recommends using Phoenix's CLI setup command so your coding agent adds tracing to your app in any language, or installing the Phoenix plugin/skill in your coding agent to connect it to Phoenix. Instrumenting the agent is presented as a simple starting step.
Laurie Voss recommends choosing an eval question the judge can actually answer given the data it has.
Look for external points of reference outside your own data sources to enhance signal and avoid the feedback loop where only your own information and golden examples exist and everything looks right. Use the internet when a source claims a number, and get creative about other references.
endorsed advice
Francesco recommends that in data and finance work you have a separate point of reference — a counterfactual — that you can check against.
The hosts recommend that when you're building a harness, you make sure you can make the evidence trail happen for your customers using your agent, so that they trust the results it gives them.
If you don't have a point of reference for what 'about right' looks like, your harness won't have it either, so build external validation points into it.
endorsed advice
Francesco's main takeaway for accountability mechanisms is to design few gates with very high signal, putting some gates in place but making sure those gates are very high signal.
Francesco recommends including on-purpose gates in any harness you design so you can still be accountable for what is produced.
Francesco recommends designing harnesses so you review tasks instead of individual changes, noting the design is intentional. He describes his own shift: you have a set of tasks, you review the task, let it do its own thing, and then review at the end.
The hosts recommend that when presenting a piece of analysis or doing financial analysis on a potential investment, you need to show where your numbers came from and be able to attribute your decision to the body of evidence you had.
The hosts recommend data labeling and making sure you have a curated data set for either training something or running any process, calling it paramount, and say you need to assign the right level of importance to the evidence you're gathering and gather as much of it as you can.
Francesco recommends having evidence to back up where decisions are coming from, what information is coming through, and whether things are being tested, and putting that front and center.
Francesco recommends designing a harness that gives you accountability by making sure you can trace all the decisions that have been made and who made them.
The closer to game day, the shorter the lift: 25-30 minutes furthest out, then 20-25, 15, and 10 minutes. Lifting doesn't have to be heavy and gruesome.
Grubbs recommends giving professional athletes menu options rather than a single rigid training plan, accounting for their preferences and schedule constraints. She coaches athletes by asking whether they want a total body lift or an upper/lower split, and whether the lift comes before or after the game. The host frames the menu approach as: come in 15-20 minutes, pick two movements, get three to four sets done, and you'll feel better.
Choose sources → to get a library of your own and a daily brief.