Recommendation · MoneyIrina's team recommends that if public facts stay local, the policy change to search over might be to classify useful public facts differently and make agents proactively share important source evidence.
hedged
AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
Recommendation · MoneyIrina's team recommends giving each agent its own memory namespace backed by RAG so memory does not bleed between agents.
AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
Recommendation · MoneyIrina's team recommends tracking agent emotion as a small vector of values like joy, success, sadness, fear, anger, or disgust, updated after events or conversations.
AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
Recommendation · MoneyIrina's team recommends giving agents belief scores toward other agents and the player, functioning as a trust matrix. After an interaction, the LLM decides whether the trust score should go up, go down, or stay unchanged.
AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
Recommendation · MoneyIrina's team recommends that in the auto research loop, you first define a controlled scenario, such as one agent learning a public fact or one agent hearing a rumor. During the simulation run, collect structured traces: observations, conversations, memory writes, retrievals, belief updates, and whatever else is relevant.
AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
Recommendation · MoneyIrina's team recommends that if rumors harden into facts, the policy change to search over might be to store confidence, mark first-hand versus second-hand, and require hedging when retelling uncertain claims.
hedged
AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
Recommendation · MoneyIrina describes how teams can replace manual prompt tuning and demo-watching with a loop: define a scenario suite, run agents, collect traces, score the data, and keep only changes that actually improve the score.
AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
Recommendation · MoneyIrina's team recommends that after rerunning, if the score improves and the guardrails hold, you keep the improvement; if not, revert back. The loop should basically be like a ratchet: try a change, score it, keep it only if the scorecard improves and guardrails hold.
AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
Recommendation · MoneyIrina's team recommends measuring diffusion with reach (how many agents know the fact after N steps), provenance with source retention among agents who know the fact (how many remember where it came from), rumors with uncertainty preservation and false certainty rate, planning with action consistency and time to replan, and privacy with containment.
hedged
AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati
Recommendation · MoneyTim Sweeney advises manually reviewing traces as a team at the end of the week on a board, looking at the best and worst traces to understand how the model is performing.
AI Engineer · An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases
Recommendation · MoneyTim Sweeney advises logging your sessions, turns, tools, and feedback as a foundational practice for AI agent development.
AI Engineer · An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases
Recommendation · MoneyA candidate model that scores higher than the prod model on the eval suite, e.g., 73% vs 72%, should be pushed forward — in Tim's case, definitely pushed forward that Friday.
AI Engineer · An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases
Recommendation · MoneyTim Sweeney advises developing a practice where researchers sit on the same scrum team as software engineers, developing tasks and viewing performance metrics as true go/no-go decisions.
AI Engineer · An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases
RecommendationElie Bakouch recommends having the agent read the training logs itself and decide whether a run constitutes a new record, which is how the race agents operated.
AI Engineer · We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect
RecommendationElie Bakouch recommends letting agents submit jobs via sbatch on a Slurm cluster, restricted to available nodes under preemptive permission so human jobs are not blocked and agents get preempted when someone else needs the node.
AI Engineer · We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect
RecommendationElie Bakouch recommends normalizing agent comparison plots by the number of active hours and re-plotting the x-axis by number of output tokens, because token consumption differs drastically between models and tells a different story.
AI Engineer · We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect
RecommendationElie Bakouch recommends running a novelty track where the goal is to beat the record using only novel ideas, which proved harder for the models.
AI Engineer · We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect
RecommendationElie Bakouch recommends treating an agent stopping and saying it cannot improve as a signal to restart with a new direction rather than accepting the answer, and when human records improve during a run, feeding the new records to the agent and instructing it to improve upon them.
AI Engineer · We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect
Recommendation · MoneyJemima Abu recommends setting up an agent that carries out audits on your code base to look for accessibility issues and then fixes them.
AI Engineer · Teach Your AI Agent to Build Products Everyone Can Use - Jemima Abu, CAIS
Recommendation · MoneyNouha Dziri recommends giving a heavy prompt that tells the model exactly what to do: follow a plan of delegating agents throughout the whole process from exploring and planning to implementation, create adversarial workers, and verify the task. A heavy prompt with lots of guidance made the solo agent jump almost six points just from better instructions, such as 'don't step on each other's toes' and pointing to the right folder. Once you tell the lead how to coordinate, it knows when to wait — the async lead catches up with the blocking one under a heavy prompt.
AI Engineer · How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere