36,612 results · Most relevant
Erina Karati recommends approaching agent state problems systematically: run controlled experiments and keep only changes that survive measurement, rather than trusting one demo or endlessly hand-tuning prompts.
Irina's team recommends that instead of manually tuning a prompt or watching one nice demo, you should define a scenario suite, run the agents, collect traces, score the data, and only keep changes that actually improve the score.
Irina's team recommends against using a single vague metric like 'agent quality' because it will hide all the interesting failures. Instead, use a balanced scorecard.
Irina's team recommends keeping the editable policy surface really small and freezing the harness, the scenarios, and the metrics so only the part of the system you actually want to optimize is exposed. The auto research layer should not have permission to randomly rewrite the whole code base. A small editable policy surface gives the search process room to improve behavior while preventing it from gaming the evaluation directly. The policy space should prevent the LM from gaming the evaluation directly.
Irina's team recommends against optimizing only one metric because it can create bad behavior: if you only optimize for diffusion, agents may learn to overshare everything, and if you only optimize for memory recall, you might create noisy or stale memories. The balanced scorecard keeps the system honest and prevents the auto research agent from gamifying the system to increase one specific score.
hedged
Irina's team recommends using controlled scenarios to evaluate long-horizon agent behavior. Scenario design matters because social behavior is otherwise fuzzy: letting agents wander may look cool and produce nice interactions, but it is very hard to evaluate whether the system actually improved.
An observability layer is the most important utility for an agent architecture. Agents must be able to log out what's going on with their sessions, turns, tool calls, and any errors happening. Log 100% of traces to an observability tool like W&B Weave so the team can learn from them, and use Weave to move from production to offline analysis, driving insights and identifying behaviors.
Tim Sweeney recommends investing in agent-oriented observability, saying it is critical that agents log what is going on with their sessions, turns, tool calls, and any errors that happen.
Tasks and evals are the new world of CI. Implement tasks, which are essentially unit tests for your models, and evaluate those models in a loop. Emit evaluation results to Weave to have a common dashboard for making go, no-go decisions on prompt or architectural changes, and use a shared dashboard to promote the best model forward with confidence.
For managers: invest in agent-oriented observability — log your sessions, turns, tools, and feedback — which introduces the ability to catch a new class of bugs called behavioral bugs, not exceptions or performance issues. Develop a practice where researchers sit on the same scrum team as software engineers, developing tasks and viewing performance metrics as true go, no-go decisions. Use humans as a necessary judge to complement LLM judges, because there are behavioral nuances that LLMs will not catch. Manually review traces as a team at the end of the week on a board, looking at the best and worst traces to understand how your model is performing.
Speaking to managers, Tim Sweeney offers tips for being successful with agents: invest in observability, treat tasks/evals as CI, use human judges, and add value through context and tools.
Recoverability should be part of the admission decision rather than something strategized or planned only after the AI agent is deployed.
Vinoo Ganesh recommends that the way to actually solve these document access problems is scoping searches with security built in.
Vinoo Ganesh recommends creating a system where the models have to dereference their own pointers, saying that by doing so you can actually create a very powerful, very verifiable AI platform.
For any tool you use, ask whether you can actually verify the numbers it produces. Ask whether you can see where the sources are — not as after-the-fact annotation, but by clicking on the number, whether it's a calculation, an extraction, or a combination — to make sure you trust the insight.
Dziri advises using a strong lead agent with cheap worker agents to get near-frontier results at a fraction of the cost, noting the team does not have to be all strong models.
Splitting work among a team of agents lets each worker start with a fresh context, read files, run tests, dig through logs, and send back only a compact result. The lead keeps the full picture and can finish faster if the task is decomposable and agents can work on independent pieces.
Dziri advises the audience to find the bottleneck in their agent pipeline and scale the team there, arguing that where you put agents matters more than how many agents you have.
Takeaway 2: Where you put agents matters more than how many — find the bottleneck and scale the team there.
Takeaway 3: The team does not have to all be strong — a strong lead with cheap workers can get near-frontier results at a fraction of the cost.
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Choose sources → to get a library of your own and a daily brief.
Map each implemented component to the original paper and explain 2 reduced-model failure modes without claiming full-scale reproduction.
Use calibration and shift measurements to justify a monitoring or abstention threshold, and give an explicit confounding counterexample to causal feature-attribution claims.
Defend a causal claim or refusal using the DAG, numerical estimates and a counterexample showing why predictive model fit cannot establish identification.
