Explore

Build a lightweight annotation queue for domain experts

Doneyli recommends building an annotation queue for the domain expert to review agent outputs and provide input. Make the process very simple — experts have day jobs and cannot spend hours looking at traces. Give them a couple of simple scores to choose from (yes/no, true/false, or a 1–10 scale) and let them add a free-text comment, e.g., noting an SME failure mode like a missing citation. Annotations should take only seconds each — around 10 to 30 seconds per trace — so an hour of their time covers a meaningful batch. Experts can even dictate comments with a tool like Whisperflow instead of typing. Once completed, the human annotation lives with the trace as evidence.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Tracing: deep insight into what the agent actually does

The most important goal of traces is getting deep insights into what your agent is actually doing. A trace should show aggregate time and costs at the root plus a breakdown of each individual step the agent takes, including generations and tool calls. Use the trace as the foundation to track cost, latency, and tokens, and as an inspectable log for humans and coding agents. You always want to know what your agent is actually happening: what steps it is taking, what inputs and outputs it is seeing, and what tools are being called.

endorsed

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Ground truth data sets for auto-improvement: expert curation and shadow runs

Auto-improvement definitely needs the foundation of a ground truth data set. Build ground truth data sets through expert curation by working with domain experts within the company. Use shadow runs: run the process manually continuously while the agent runs in parallel so you can compare. Shadow runs let you measure agent accuracy while also building your ground truth data set. Teams in regulated industries build ground truth data sets that they can loop on and improve over time.

endorsed

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Incorporate experiments into CI/CD so evals run automatically on every change

Doneyli recommends incorporating experiments into your CI/CD pipeline so you don't have to be in the loop. When a change hits your agent, a CI job kicks off to run the experiment against the dataset and evaluate the agent with the new configuration, giving a pass-fail gate across all the scores and evaluators — a consistent way to evaluate agent outputs.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Derive quality dimensions from expert feedback, then decide scoring and evaluators

Doneyli recommends that after extracting the most common categories from the domain expert's corpus, you define your vectors or dimensions of quality (scores). Example dimensions include source correctness, formal register (addressing customers with a more formal tone in some languages), language matching the customer's incoming language, and must-include / non-solicited items. Once you have your quality categories, decide how to grade them: what type of score and what type of evaluator each one gets.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Use prompt versioning as the trigger for automated eval CI jobs

Doneyli recommends versioning your system prompts in Langfuse (free of charge) and treating a prompt version change as the trigger for kicking off the fully automated eval: configure an automation that triggers on any prompt changes in the agent config, with the action set to launch a CI job in GitHub. Keeping multiple versions of prompts also lets you track how the prompt evolved over time, and you can promote a previously tested version back into production when regressions are observed in the current version. All of this can be done via the API and CLI instead of clicking through the UI; tagging the promoted version with a production tag kicks off the eval pipeline.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Perform error analysis on traces: check steps, tool calls, and outputs

Doneyli recommends performing error analysis by looking at your traces to understand how the agent is behaving. When reviewing traces, check whether the agent is taking all the steps it needs to take, calling all the tools it needs to call, and giving the output that you want.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Evaluation approaches: LLM-as-a-judge, code-based evals, deterministic checks

Use LLM-as-a-judge, i.e., another LLM checking what the LLM did, as a scalable evaluation approach. Use code-based evals for classification tasks where you can check for an exact match. Use deterministic checks for behaviors like detecting all-caps messages.

endorsed

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Automated scores for compliance, failure modes, and guardrails

Define scores for your use case that are automatically calculated whenever a new interaction or trace is captured by the system. Score whether the information given to the customer is compliant with banking information — e.g., if the customer asks about the weather, that is not compliant. Score for failure modes and for guardrails blocking answers given back to the customer. Use scores to filter traces for specific intents, e.g., filtering for traces where customer intent was investment advice.

endorsed

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Build golden datasets from a maintained spreadsheet or from production traces

Doneyli describes two main ways to create golden datasets: have a team maintain a spreadsheet of things to test on the agent and upload it as a CSV, or add items that are already part of your production traces. He frames the latter as 'dogfooding' the dataset — building it from production traces as long as you've determined they are good examples of desired future outputs.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Evaluate against specific evaluators with minimum score thresholds

Doneyli recommends evaluating agents against specific evaluators such as source recall, language match, correctness, and non-unsolicited upsell, and defining minimum score thresholds for each evaluator: if a score doesn't clear the bar, it's not considered a pass to go into production.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

PII masking before persistence in observability systems

Enable PII identification so any data deemed PII is automatically masked before it gets persisted into Langfuse. Financial institutions do not want to persist credit card information or social security numbers in an observability system; they want that data kept in systems of record they own. You cannot control what your users input into the system, so you must catch sensitive data (like credit card numbers in clear text) when instrumenting and building your evals. Use a masking function that immediately masks any defined PII so it is not persisted in plain text in your system.

endorsed

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Building data sets: inputs, expected outputs, out-of-scope requests, stable benchmarks

Data set items should contain the input plus the expected output, which can be a sample reply, expected behavior, or keywords to be covered. Include out-of-scope requests in your data set that you want the agent to correctly defer, such as refusing to give investment advice or answer weather questions. Use data sets as a stable benchmark, either for offline iteration or as part of your CI/CD.

endorsed

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Curate multiple datasets per use case and technical requirement

You typically end up with different datasets to measure specific things or specific areas of quality in your agent fleet. Maintain a red-teaming dataset that looks for prompt injection attacks and any attack that would allow someone to circumvent guardrails and obtain information they shouldn't get. Maintain a dataset specific to a use case, e.g., knowing which cases have been disputed and whether the dispute has not been solved. Curate datasets that are specific not only to the use case but also to the technical requirements you have for that agent.

endorsed

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Datasets as test cases: evaluation criteria anchored on expected outputs

Build evaluation criteria anchored on datasets: a set of questions and expected outputs that allow you to measure the quality of outputs of your LLM or AI solution. Think of datasets as your test cases if you come from a software development background. Do not ship software into production without passing it through QA — merging straight into main or prod is not a best practice. Datasets give you the evidence that you passed the agent through a series of steps or requirements that measure all the things you want to capture. The expected output in a dataset does not necessarily need to be the answer the agent will give; you can measure the intent of the agent, what the agent is supposed to do, or behavior.

endorsed

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Ask SMEs for raw commentary, then use an LLM to extract failure-mode categories

Doneyli recommends not asking the SME to invent categories — instead ask for unfiltered raw commentary on the traces. Then use an LLM to find all failure modes or categories that are patterns across the expert's comments, because LLMs are great at pattern matching and extracting information from text. This keeps it simple for the domain expert and uses LLMs for what they are great at.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Blend LLM-as-a-judge and deterministic evaluators for a balanced mix

Doneyli recommends measuring agent outputs across both LLM-as-a-judge scores/evaluators and deterministic evaluators, and building a library of evaluators so incoming traces automatically get their scores calculated — or alternatively having scores calculated directly from the client application in real time and surfaced back to the system. The more balanced the mix, depending on the categories you identify, the better: if you rely 100% on LLM-as-a-judge, there's a non-zero chance some outputs will be inaccurate because that's the nature of LLMs. Deterministic checks always produce exactly the same output for the same inputs — a code evaluator gives the same output 100% of the time for the same input, not 95% or 80% — enabling a blended assessment of agent quality.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Regulated industries: validated ground truth and human-in-the-loop

In regulated industries, make sure you have a ground truth data set that is validated by your customers. Run human-in-the-loop systems when shipping agents in regulated industries. Run a fully human-in-the-loop system first, then over time slice the data into parts you are confident to automate versus parts where you always want a human in the loop.

anecdotal

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

For each DAG node, weigh agent-driven vs human-driven; keep the final decision node human

For each node in a DAG, ask how valuable it would be to have an agent drive the node versus a human drive it. The final decision node in a lending flow should probably be human driven, since it is essentially the check against the policy. A node may be better suited for automation if you can check it against policy easily, run it in a bounded loop, trace it, and fall back to a human.

AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase

Build representative data sets, run evals in repeated series as statistical analysis, and combine evals with traces

Kormos recommends building representative data sets and evaluating them, defining what good looks like. Run evaluations in repeated series as a statistical analysis, looking at outputs in volume rather than as direct comparisons. Combine evals with traces — the runtime logs of what happens — as a confidence mechanism complementing traditional software testing.

AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase