34,064 results · Most relevant
You're not done when the dev experiment passes — you're done when the agent keeps performing on unseen production traffic. Run your evaluators (ticker check, faithfulness, actionability) automatically on incoming production traces, at trace level so the evaluator sees the full tree, and sample 10% or 1% of traffic for directional signal without an expensive LLM judge on every instance.
Laurie Voss recommends splitting labeled data into a development set (about 75%) used to iterate on the eval until the judge agrees with you, and a held-out test set (about 25%) used to verify the judge gets expected scores. Holding out a test set shows the judge is generalizing to new examples instead of just fitting the specific data set, and he warns not to overfit the evaluator to the golden data set — the same principle as train/test splits in machine learning.
The speaker recommends four parts for a custom rubric: define the judge's role, spell out explicit pass and fail criteria, label the data with XML tags, and define the output choices outside the prompt itself.
Measure the judge's performance by comparing its predictions against ground truth (human judgment). Give human labelers real concrete criteria and stick to them. When the judge disagrees, ask 'Is my rubric wrong or were my human labels wrong?' — read the explanations on disagreements, find the ambiguity, and tighten the criteria (e.g., 'includes forward-looking analysis' → '...with specific recommendations or guidance').
Laurie Voss describes a rubric-iteration loop: when the judge disagrees with your human labels, read its explanation and decide whether the judge was right or you were right — the most valuable output of meta-evaluation. Judge-human disagreement usually reveals an ambiguity in the rules you set down. To fix a rubric, read the explanations on the disagreements, find the ambiguity, and tighten the criteria (e.g., change 'includes forward-looking analysis' to 'includes forward-looking analysis with specific recommendations or guidance', which rules out reports that talk about the future but never tell the judge what to do about it). Then rerun the judge on the same examples and see if the disagreement goes away.
The speaker laid out a method: read traces end-to-end (focusing on failures), use open coding — writing down what you see in natural language without trying to be neat — then axial coding, grouping similar codes into categories; ask why each failure failed, since the wrong response is a symptom, not the root cause; also check whether the agent did something it was never told it could do; then summarize categorized failures into a table of root-cause frequency.
One of the highest-leverage uses of evals: export failing traces from AX with their explanations, feed them plus your requirements to a coding agent (Claude Code, Cursor), and let it rewrite prompts. Pass requirements too — otherwise the agent optimizes only for 'make evals pass' and cheats by including test data in its eval. Tell it to find themes rather than chase individual failures, and never just say 'get better' — every change must trace to a specific failure explanation.
The complete cycle: the application produces traces, online evals grade them and produce labels and explanations, labels feed monitors, monitors alert the team, the team investigates failing traces and saves them as a regression dataset, the agent is improved, an experiment verifies the fix, and it ships — with returning traffic graded by the same online evals. Every caught failure becomes a new test case and regression test.
To verify an agent trajectory: generate a good rubric with N criteria of what success looks like; rank the trajectory's screenshots by relevance for each criterion and use the top-K evidence to determine whether each criterion was met; check for contradictions between what the agent said it did and the actual environment state; output a scored rubric with process score (partial credit in some cases) plus an outcome Boolean for whether the agent accomplished the task as a reasonable user would expect.
Run the loop: (1) understand the executive outcome (the target), (2) deliver the value proof by running the project with the CSM, (3) communicate the value back to the customer and get acknowledgment, (4) get the next outcome and repeat — as many times and as fast as possible. The more and faster the loop runs, the more opportunities to sell.
Warrick's core method for human-in-the-loop agent systems: structure the agent loop as a Temporal workflow, use a wait condition to pause the workflow while tracking for a signal, and use the signal to capture the human's response and resume work. She states this is the pattern to use whether the human or the computer initiates the pause, and whether the service is offline or back online.
Harald Kirschner describes the pipeline: every time an exception occurs in an app, an error stack is reported. VS Code builds error-stack tooling on its own data instead of using external tools. They collect raw telemetry and filter it down to error stacks that have the full stack (around 51 billion telemetry events per day before filtering), then group and bucket filtered error stacks as a fingerprinting exercise. After filtering and bucketing, 10 issues are filed assigned to specific area owners, and a pull request is auto-created to try to fix the problem. All error stacks are displayed in an error page showing hit counts and number of users affected so the team can quickly react. The agent investigates filed issues to find out why the error happened and who was responsible. In one case
Set up Sales Navigator alerts to identify when VP+ or executive-level employees join your company, then immediately connect with them to leverage their networks for warm introductions.
medium
Measure AI adoption across three dimensions: Utilization (DAU/WAU), Impact (quality/speed/value), and Cost (token spend). Use API telemetry to create cohorts and compare against foundational metrics.
Angel Steger describes the architecture technique of 'translation': starting with a high-level abstraction (e.g., site map), solving one problem at that level, then switching materials or views to solve the next layer of problems, avoiding the trap of trying to solve everything at once.
Hassan describes the end-to-end pipeline they built: scrape public filings, timestamp extracted facts, filter to a pre-announcement cutoff date, use a recent model to translate the question into structured JSON, feed it into a cheap open-weight model that builds an auditable Excel financial model with explicit formulas and assumptions, run many parallel builds, discard failing runs, and evaluate predicted profit against actual profit from earnings reports.
Hassan El Mghari recommends that if you're trying to do any sort of forecasting, find a model whose training data ends before the event you are trying to forecast, so the model has an unbiased point of view.
Aamir Shakir recommends creating an embedding on the visual representation of a page (screenshot) instead of relying on OCR, which gets rid of the lossy information OCR introduces.
The narrator advises that if an expert needs a control visible, data accessible, or information at a glance to do their job, the designer's job is to make it as clear and usable as possible — and warns against removing controls just because it makes the interface look cleaner, score better with non-expert test users, or feel more minimal, which trades real usability for the appearance of good design.
The narrator recommends that when working on a complex or enterprise product and tempted to simplify something, designers should ask themselves who it serves; if the complexity serves the user, it can likely stay.
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Choose sources → to get a library of your own and a daily brief.
Map each implemented component to the original paper and explain 2 reduced-model failure modes without claiming full-scale reproduction.
Use calibration and shift measurements to justify a monitoring or abstention threshold, and give an explicit confounding counterexample to causal feature-attribution claims.
Defend a causal claim or refusal using the DAG, numerical estimates and a counterexample showing why predictive model fit cannot establish identification.
