A method for automated quality gates: anchor experiments on datasets, set up CI so any agent or prompt change triggers a GitHub CI job running evals against the dataset, define minimum score thresholds per evaluator, and treat failing scores as a block on production promotion.
A method for assembling evaluation datasets: create use-case-specific datasets (including red-teaming sets for prompt injection), source items from team spreadsheets, production traces, and curated good examples, and keep domain experts in the loop defining golden datasets.
A method for scoring agent outputs: extract common categories from domain expert feedback to define dimensions of quality, build a library of evaluators that automatically score incoming traces (optionally in real time from the client application), and blend LLM-as-a-judge with deterministic checks — relying 100% on LLM judges risks inaccurate outputs.
A method for getting domain expert feedback: build an annotation queue routing agent outputs to experts, give them simple scores (yes/no, 1-10), allow free-text comments, and keep the process very simple since experts have day jobs — ask for about an hour of their time on a curated batch.
Start collecting all the ground truth you can evaluate your agent against — from historical data, production, or samples run with your domain experts. If you're in a regulated space, define your hard limits (things that definitely shouldn't happen, e.g., in financial services the agent should never give investment advice) and ingrain them in your evaluation set. Ideally this ensures they never happen; at least you will know about it when it does.
Doneyli De Jesus advises verifying that an LLM judge catches what you intended by manually judging a representative sample yourself and optimizing the judge for coherence with human raters. Without calibration against human-labeled ground truth, the judge will not judge the way a human would and may miss what you wanted to catch.
For customer-facing or regulated-industry use cases, don't risk the agent doing something it shouldn't: build ground truth datasets before production. Curate them with domain experts within the company, include expected outputs (sample replies, expected behavior, keywords), and validate them with customers. Ground truth is also the foundation for auto-improvement.
To ensure LLM-as-a-judge consistency (inter-rater reliability), judge a representative sample yourself, then run your LLM judge and optimize for coherence with what your human raters do. Academically you'd have a few human raters; practically, one human labels some data and you optimize the judge against that ground truth. Check the judge over time to make sure it stays representative. When running on production data, start with a few too many false positives — catch a little more than necessary but don't miss anything — then tune it down over time.
Define quality gates for your agent and verify they are being enforced and scored on every interaction. Scores are automatically calculated whenever a new interaction or trace is captured by the system. Score whether information given to the customer is compliant with banking scope (e.g., a weather question is not compliant), detect failure modes and guardrails blocking answers, and use a score to flag customer intent (e.g., investment advice) so you can easily filter for those cases in traces.
Companies that successfully ship and continuously improve reliable agents follow a set of practices called the AI engineering loop. The foundation of the AI engineering loop is always tracing. Monitor aggregate metrics on what is going on and detect signals where things are going wrong. What you get from production through monitoring is one of the foundations you can use to build out your data sets. Run experiments by trying different variations of your system to check how it performs against your data sets. Continuously close the AI engineering loop: learn from production and feed it back into data sets.
endorsed
In-season training should work to maintain 85-95% of absolute strength — the first job across three sessions. Assign one back squat session per week, plus a heavy unilateral lower-leg exercise such as a front foot split squat or Bulgarian split squat. For athletes with weekend games, stack the back squat session at the start of the week (probably Monday morning) assuming the athlete is fairly fresh. Maintain upper body strength as well, though it likely has less negative influence on gameplay. The secondary goal is transference into speed and power: clean pulls from rack, power cleans, and faster, higher-speed, slightly lower-force exercises. Accessory exercises go in the latter half of the session and are predominantly prehab/rehab.
The team designed their generation algorithm around the difficulty of changing optical weights by dividing 1,000 diffusion steps into 100 subunits of 100 steps each and keeping parameters fixed within each subunit. With 10 optical devices, one device can be used 400 times, then pass the activation to the next device and use it 400 times, allowing the full denoising procedure without changing any weights on the physical hardware.
provenanceStatus possible_numeric_error on the step counts
Coinbase's approach: each team and repository has a 'brain' implemented as markdown files that team members can review at any time. Every time an agent comes in to make a change to a service, it ingests the brain for that team and repo. When a human reviewer fixes something an agent missed, that context must go back into the brain so it is fixed in all future cases — the simple manual-edit-and-ship response is the wrong move. This causes the accept rate for one-shotted PRs to tick up over time, creating a recursive self-improvement system.
Frontier AI labs should publicly disclose whether their models have been used to attack standardized and deployed cryptography — the key question is the status of actually attacking standardized and deployed crypto, and labs should be asked what they've done trying to use their models on basic cryptography questions.
Doneyli advises not auto-merging to main just because the eval gate passed; let the result sit for human review. He acknowledges you could set up a full end-to-end process that creates a PR automatically and waits for someone to review it if the eval clears the bar, and notes some teams push the PR straight into main if it's clearing the bar — but he discourages pushing a PR based on vibes or on what you think the agent is good at. Instead, keep a paper trail of eval evidence documenting why you made the decision for a new agent configuration.
Doneyli recommends building an annotation queue for the domain expert to review agent outputs and provide input. Make the process very simple — experts have day jobs and cannot spend hours looking at traces. Give them a couple of simple scores to choose from (yes/no, true/false, or a 1–10 scale) and let them add a free-text comment, e.g., noting an SME failure mode like a missing citation. Annotations should take only seconds each — around 10 to 30 seconds per trace — so an hour of their time covers a meaningful batch. Experts can even dictate comments with a tool like Whisperflow instead of typing. Once completed, the human annotation lives with the trace as evidence.
The most important goal of traces is getting deep insights into what your agent is actually doing. A trace should show aggregate time and costs at the root plus a breakdown of each individual step the agent takes, including generations and tool calls. Use the trace as the foundation to track cost, latency, and tokens, and as an inspectable log for humans and coding agents. You always want to know what your agent is actually happening: what steps it is taking, what inputs and outputs it is seeing, and what tools are being called.
endorsed
Auto-improvement definitely needs the foundation of a ground truth data set. Build ground truth data sets through expert curation by working with domain experts within the company. Use shadow runs: run the process manually continuously while the agent runs in parallel so you can compare. Shadow runs let you measure agent accuracy while also building your ground truth data set. Teams in regulated industries build ground truth data sets that they can loop on and improve over time.
endorsed
Doneyli recommends incorporating experiments into your CI/CD pipeline so you don't have to be in the loop. When a change hits your agent, a CI job kicks off to run the experiment against the dataset and evaluate the agent with the new configuration, giving a pass-fail gate across all the scores and evaluators — a consistent way to evaluate agent outputs.
Doneyli recommends that after extracting the most common categories from the domain expert's corpus, you define your vectors or dimensions of quality (scores). Example dimensions include source correctness, formal register (addressing customers with a more formal tone in some languages), language matching the customer's incoming language, and must-include / non-solicited items. Once you have your quality categories, decide how to grade them: what type of score and what type of evaluator each one gets.
No published protocol matches this selection.
The creators behind most of what cleared.
323 cleared insights match
179 cleared insights match
154 cleared insights match
Choose sources → to get a library of your own and a daily brief.
