Principle · MoneyWhat really matters in a judge prompt is the context — this is a financial report, we expect a report that analyzes financial data and tells us real things, and what we care about is whether you could act on it. Don't write 'a good response is helpful and accurate' because a judge can't make a yes or no decision from that. Telling the judge it is an expert actually works — it helps a little, and telling an LLM it's really smart makes it confident and try harder.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyThe eval-iterate cycle is where the value is, not anyone's score: you run your evals, read the failures and the explanations, spot the pattern, change the prompt or the tools or the data, and run the experiment again. After running the experiment again, your evals can answer 'did it get better?' — you can go from 'I think it works' to 'I can show you it works.'
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss warns that automating evals before understanding how an agent fails leads to measuring what is easy to measure instead of what actually matters.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss warns against grading the path (the specific steps the agent took) instead of the outcome; Anthropic advises that checking particular steps is too rigid because agents can be creative and do things in unexpected ways. The ticker check only cares that the right company is in the report — the right level for many evals.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss explains that the Claude Agent SDK runs the same engine as Claude Code, so if you're doing this in Claude Code, it tends to pick up the options, tools, and skills you've given to your copy of Claude Code instead of its own copy — making things extremely non-deterministic in an unhelpful way. He added a couple lines of code to make sure the Claude SDK runs in its own isolated version of the SDK, not stealing your copies of keys and other data.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss notes that human judges also disagree with each other a lot of the time, so don't be too worried about how much you can trust a judge. Inter-expert agreement is measured with a standard score called Cohen's kappa, and for this kind of labeling task it is often as low as 0.2 or 0.3 — experts agree not much more often than random chance. The bar for your judge is human performance, and human performance is not actually that good when you compare humans to each other. Your judge is never going to agree with you 100% of the time. What you want is a judge that does better than another human doing the judging would be, because your judge can run across thousands of examples.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss explains the Claude Agent SDK's turn model: one turn is one query call. You give it a prompt, it streams back messages as the agent works (thinks, calls a tool, reads the result, maybe calls another tool), and when finished you get a result message with the final text and a session ID. The session ID is how you carry on the conversation: pass it back in as resume, and the next turn remembers everything the last turn saw.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss explains that the higher the quality of the data in your golden dataset, the more reliable everything is going to be. One of the best things you can do is continuously add to your golden dataset — when you see a tricky example or something strange in production, figure out what the answer should have been and put it in.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyYour judge should be more capable than the agent itself (e.g., agent on Haiku, judge on Sonnet or Opus), and ideally from a different provider entirely (e.g., OpenAI judging Claude's work) because of same-model bias — Claude tends to like the output of other versions of Claude more than output from OpenAI or other providers. Run cheap and fast models like Haiku for the agent itself, reserving more capable models for judging.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss argues that a prompt change you haven't measured is just a vibe, and that the goal is a controlled comparison: the same inputs and the same evals, before and after the change — that is an experiment.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyApproving every calculation or reviewing every PR line by line that agents ship doesn't feel great — it's the other side of the spectrum from full automation and leads to a lot of toil and unnecessary work if not designed for the use case.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyWhen presenting analysis to your firm, attribute your decision to the body of evidence you had rather than just stating the conclusion; show where your numbers came from — something that bites data professionals weekly.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyBuilding on the shared domain principle, the agent understands that pieces of the plan that are just code or a calculation will always give a consistent, deterministic result. Based on identifying deterministic elements of the plan, you need to implement code — something that gives the same result every time.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyIn an AI-first world, we sometimes forget that AI is only a means to an end, not the point itself. At Monzo, they approach AI from a highly impact-based point of view: not 'we should use AI because AI can X', but 'what's the outcome we're looking for here?' Define how you measure success for an AI initiative — e.g., are you looking for less back-and-forth with a BI team, or to validate a new dashboard — so you can track whether AI supports that outcome. Track metrics like fewer internal ask tickets (because a number looks wrong) to see if AI supports generating that outcome. The way you achieve an AI outcome doesn't really matter to a point, because AI will be a tool that gets you to the result probably faster, with better guardrails.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyIf you don't have traceability even for changes you make in your deterministic code, you leave yourself vulnerable to blind spots when the process changes, even if the inputs are the same.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyFrancesco Galletta argues the goal is a financial analyst harness you can actually trust, rather than ad-hoc chatting with spreadsheets, and that design principles for achieving that trust matter more than specific technologies or product readmes.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyThin slicing is a concept where you have thin slices of your anecdotal experience, and you know what things feel right or feel wrong based on your experience in your domain, whatever analysis you're doing. The model will have an average of everyone else's experiences. When you're building your harness, you need to make sure that you can have a feedback mechanism so that you can verify from what the model thinks it knows all the way to what you think you know.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyThe gates you design need to be fit for purpose — you need to know the use case really well to put gates where it matters. Adding a gate before every decision asking 'ask me' makes it very low signal again.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Creator perspective · MoneyGiving an agent a bunch of instructions that say the right things but still rely on the agent not making mistakes is the non-deterministic way and produces a non-deterministic result every time. The value of building a harness (rather than plugging principles into skills/instructions for an existing harness like Codex or Claude Code) is mixing non-determinism with deterministic behavior you want to control when relevant. The principles of mixing determinism and non-determinism are the same regardless of implementation details like abstractions, methods, or actual code.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyThe design of your harnesses needs to be intentional to increase trust. AI results will likely not be accurate unless you design all the mechanisms so that whatever process runs, you know you can trust the results to a degree. Don't assume that because something is running on AI it's going to be '200% accurate' — it will likely not be.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo