35,684 results · Most relevant
Go back into your historical data and constrain a set of companies where something happened (positives) and companies where it didn't (negatives), then train and iterate your model on that set.
Commercial context: The creator describes this as their own product.
Seinfeld's production process used the live audience as a real-time feedback mechanism: pre-recorded location scenes were screened to the audience so their laughs could be recorded; if a scene didn't earn a real laugh, production paused, writers/actors/director huddled on stage, and the joke was rewritten live while a band kept the audience entertained — and no artificial laugh track was ever used. On camera block day, cameras and actors rehearsed together as a unit so each camera could reposition within seconds. The week ran Wednesday-start/Tuesday-shoot to buy extra rewriting time, with pre-production for future episodes running in parallel.
A practical trust-building approach: start by giving an agent $5 for a job, then $50, and if it works, $500. People already trust agents with access to their systems, often even write access, which sometimes leads to disasters or data leakage. Giving agents money is actually safer than giving them system access: losing $50 is sad but not a disaster, whereas somebody leaking your email is much worse.
The speakers describe a saboteur/detective game: the saboteur picks a snack and randomly puts a secret bug in it, and you, the detective, find it with snack alone, name the bug, and give proof that you found it. Run 'pnpm snack up --bug random' from the repo to start; with the bug-random app running (not headless), the bug is hidden until you stop up; leave it running and in another terminal run 'pnpm snack doctor'. Possible outcomes include: it named the bug with output, said it works but it didn't, said it works but it did, blamed a bug that wasn't there, and had the right bug but no evidence. 'None' is a real answer — there's nothing to find and everything works.
Consequential creator advice requires independent verification before Today can promote it.
Let the agent determine on its own that something is not correct, take another stab, run the command, and verify again without your involvement. When an agent self-heals and self-corrects on its own, you avoid coming back to a PR at the end of the day still needing to push it through. When the agent realizes it is not done and it's not going to drop the work on the floor, that's the moment you've gotten the value out of the entire verification system.
Preserve what a human would consider an excellent customer response in a help ticket system as a receipt. Everything your agent does today, every run all day long, is very valuable data because you can feed it into LLMs and get fixes that you need. Build a dataset of exactly how your system performed so you can create comprehensive evals for agents to hill climb on — loop, see their performance, and continue going until they get better and better.
Define how the work can be verified before handing it off to an agent. The agent builds a check, figures out what tools it needs, runs them, verifies they all pass, keeps the evidence, and decides when to stop (as with the Riker agent). The agent should also alert you when it needs you instead of you babysitting it.
Build up trust with checks for everything that came before: unit tests, evals, smoke tests, and whatever the application needs. Give the model the ability to interpret and run existing checks and improve them so that when new features are added, all previous checks can be run to ensure no regressions. Mount and climb the evidence ladder, building evidence as you go, to ensure the app is working exactly as expected. The accumulated checks and evidence become the spec that shows what the app does.
endorsed
Write good error messages because they are more important than ever. Make your app's failures explicit about why they failed and give as much detail as possible about how it got into that state so your agent can self-heal from that. When your app throws an error, an agent can interpret and run with that error really well.
Start verifiers as a simple CLI before hardening them into reusable pieces. Build standardized verifiers in a library, skills, or tooling that you share across your org. Use standardized verifiers to accelerate PRs and reduce human oversight over time.
reduced oversight framed as a possibility ('could be one of the ways')
Wisedocs onboarding starts with a few sample files in various formats the customer is actively working on or back-testing, then a review process compares what the customer extracted versus what the system extracted against their standards. Business objectives are aligned (e.g., date of injury vs. report date), a few standard cases are processed first, and if things work well a broader pilot precedes full integration. With a clear business outcome and transformation goal, onboarding is fairly straightforward.
The creator recommends constraining agents by knowing what intent the agent is trying to achieve and what mission it is trying to accomplish, claiming real-world cases show the agent loop becomes better, faster, and stronger as a result.
Ian and Davis discussed consent fatigue before AAuth and similar concepts emerged in the ecosystem, and there is always a trade-off between how secure you make a system and the impact on the human. Minimize human-in-the-loop interventions to only when absolutely required, because excessive human approval gates block agent adoption and autonomy. Human-in-the-loop is somewhat of an anti-pattern in many ways because reviewers get fatigue — but you still need a way to bring the human in at the right time.
None stated.
Laurie Voss describes his iteration loop: run your evals, read the failures and explanations, spot the pattern, change the prompt or tools or data, and run the experiment again. A single line in the prompt (only use numbers that appear in your research) fixed what was measured — actionability went straight to perfect — but faithfulness stayed bad and the agent was still inventing data some of the time. The next thing to fix for this agent is that it is inventing the data some of the time; judgment calls are fixed but data invention is not.
An LLM judge has three separate parts: a judge model (the LLM doing the judging), a rubric (the prompt telling it what good looks like), and the data (the examples it's judging). Phoenix keeps these separate so you can mix and match — try the same rubric with different models, or the same judge with different rubrics.
Laurie Voss describes building the golden dataset in Phoenix by finding every report with a human label, making each an example (input = report, expected output = human label), sorting by span ID, and assigning every fourth example to the test split with the rest in dev. Span ID links each example back to the trace it came from so you can always click through to the original run. He runs the judge against human labels only on the dev split, not the test split.
Laurie Voss advises running npm run batch to execute 15 queries in a row, since one test is not enough data to draw conclusions. He also recommends including edge-case queries in the test set: comparing two companies, non-public companies, different industries, and fake tickers. Synthetic LLM-generated queries are only a seeding method, so a human should review the list.
Write down exactly what makes a report actionable: it makes a specific recommendation, it names concrete risks, and it looks forward, not just back — things a judge can actually check. Your criteria come from your error analysis: 'only summarizes data with no interpretation' is a problem seen when reading traces, and it goes in the rubric because it's a problem you're trying to avoid.
Tracing takes about five lines: call register from the Phoenix OTEL package to set up tracing and point it at Phoenix. In TypeScript, use manually instrument, which hands you back a copy of the instrumenter — use that copy because the Claude Agent SDK is an ES module with read-only exports. Import the Claude Agent SDK from the instrumentation file, never straight from the package.
Laurie Voss advises workshop attendees to ask themselves what would make them act on one of these reports and what they saw in their own traces, write that down, and let it become their rubric. He also notes that the criteria and examples in the rubric file can be changed.
Choose sources → to get a library of your own and a daily brief.