Build best-in-class experiences for emerging AI capabilities
Roman Ugarte advises building the best-in-class experience for things that are going to happen, translating AI capabilities into amazing products for customers as the models get stronger.
Explore
35,682 results · Most relevant
Worth exploring
WHY · 34950 cleared results from 506 sources on record
Solve agent state problems with controlled experiments, not demos or endless prompt tuning
Erina Karati recommends approaching agent state problems systematically: run controlled experiments and keep only changes that survive measurement, rather than trusting one demo or endlessly hand-tuning prompts.
Instead of manually tuning a prompt or watching one nice demo, define a scenario suite, run the agents, collect traces, score the data, and only keep changes that actually improve
Irina's team recommends that instead of manually tuning a prompt or watching one nice demo, you should define a scenario suite, run the agents, collect traces, score the data, and only keep changes that actually improve the score.
Do not use a single vague metric like 'agent quality' — use a balanced scorecard instead
Irina's team recommends against using a single vague metric like 'agent quality' because it will hide all the interesting failures. Instead, use a balanced scorecard.
Keep the editable policy surface really small and freeze the harness, scenarios, and metrics
Irina's team recommends keeping the editable policy surface really small and freezing the harness, the scenarios, and the metrics so only the part of the system you actually want to optimize is exposed. The auto research layer should not have permission to randomly rewrite the whole code base. A small editable policy surface gives the search process room to improve behavior while preventing it from gaming the evaluation directly. The policy space should prevent the LM from gaming the evaluation directly.
Do not optimize only one metric — it can create bad behavior
Irina's team recommends against optimizing only one metric because it can create bad behavior: if you only optimize for diffusion, agents may learn to overshare everything, and if you only optimize for memory recall, you might create noisy or stale memories. The balanced scorecard keeps the system honest and prevents the auto research agent from gamifying the system to increase one specific score.
hedged
Use controlled scenarios to evaluate long-horizon agent behavior
Irina's team recommends using controlled scenarios to evaluate long-horizon agent behavior. Scenario design matters because social behavior is otherwise fuzzy: letting agents wander may look cool and produce nice interactions, but it is very hard to evaluate whether the system actually improved.
Invest in agent-oriented observability and log everything
An observability layer is the most important utility for an agent architecture. Agents must be able to log out what's going on with their sessions, turns, tool calls, and any errors happening. Log 100% of traces to an observability tool like W&B Weave so the team can learn from them, and use Weave to move from production to offline analysis, driving insights and identifying behaviors.
Invest in agent-oriented observability and log sessions, turns, tool calls, and errors
Tim Sweeney recommends investing in agent-oriented observability, saying it is critical that agents log what is going on with their sessions, turns, tool calls, and any errors that happen.
Use tasks and evals as the new CI, with a shared decision dashboard
Tasks and evals are the new world of CI. Implement tasks, which are essentially unit tests for your models, and evaluate those models in a loop. Emit evaluation results to Weave to have a common dashboard for making go, no-go decisions on prompt or architectural changes, and use a shared dashboard to promote the best model forward with confidence.
Manager tips: invest in observability, tasks as CI, and human judges
For managers: invest in agent-oriented observability — log your sessions, turns, tools, and feedback — which introduces the ability to catch a new class of bugs called behavioral bugs, not exceptions or performance issues. Develop a practice where researchers sit on the same scrum team as software engineers, developing tasks and viewing performance metrics as true go, no-go decisions. Use humans as a necessary judge to complement LLM judges, because there are behavioral nuances that LLMs will not catch. Manually review traces as a team at the end of the week on a board, looking at the best and worst traces to understand how your model is performing.
Tips for managers succeeding with agents
Speaking to managers, Tim Sweeney offers tips for being successful with agents: invest in observability, treat tasks/evals as CI, use human judges, and add value through context and tools.
Make recoverability part of the admission decision
Recoverability should be part of the admission decision rather than something strategized or planned only after the AI agent is deployed.
Scope searches with security built in to solve document access problems
Vinoo Ganesh recommends that the way to actually solve these document access problems is scoping searches with security built in.
Create a system where models must dereference their own pointers to build a verifiable AI platform
Vinoo Ganesh recommends creating a system where the models have to dereference their own pointers, saying that by doing so you can actually create a very powerful, very verifiable AI platform.
For any AI tool you use, ask whether you can verify the numbers and see where the sources are
For any tool you use, ask whether you can actually verify the numbers it produces. Ask whether you can see where the sources are — not as after-the-fact annotation, but by clicking on the number, whether it's a calculation, an extraction, or a combination — to make sure you trust the insight.
Use a strong lead with cheap workers for near-frontier results at a fraction of the cost
Dziri advises using a strong lead agent with cheap worker agents to get near-frontier results at a fraction of the cost, noting the team does not have to be all strong models.
Split work among agents so workers start with fresh contexts
Splitting work among a team of agents lets each worker start with a fresh context, read files, run tests, dig through logs, and send back only a compact result. The lead keeps the full picture and can finish faster if the task is decomposable and agents can work on independent pieces.
Find the bottleneck in your agent pipeline and scale the team there
Dziri advises the audience to find the bottleneck in their agent pipeline and scale the team there, arguing that where you put agents matters more than how many agents you have.
Find the bottleneck and scale the team there
Takeaway 2: Where you put agents matters more than how many — find the bottleneck and scale the team there.
Protocols you can run
WHY · 60 of 60 published protocols
Adversarial Generators and Failure Analysis
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Backpropagation Without Autograd
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Bayesian Inference and Graphical Models
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Source analyses
WHY · 672 of 672 current source analyses
Choose sources → to get a library of your own and a daily brief.