33,353 results · Most relevant
The fix is never just to make the number higher or nicer — it's to build a harder eval: read the traces, find the artifact, harden the harness, and then rerun. Never trust your own scoreboard.
endorsed
A benchmark has to be transparent so you can look at the traces and see what happened; then surprising results make sense. If a benchmark result is surprising, take a look at the traces to find out what is happening. A bar is a claim, a trace is evidence. Click any bar on WolfBench to get taken to the Weights & Biases Weave page where you can inspect all the traces.
When showing early prototypes, Josh Woodward says you're looking at people's eyes — that is the metric, not DAU or D7. If someone's eyes light up and they lean in, those are the signals to look for in the early days. He describes asking 'Hey, did you see this? What do you think about this?' and watching for the reaction.
Josh Woodward explains that larger lab projects risk crowding out zero-to-one work, so the team uses explicit stage labels — zero-to-one labs, one-to-ten labs, ten-to-one-hundred labs — to keep the distinction clear. The people suited to each stage differ a little too, so you create commonality while respecting the differences. In zero-to-one you optimize for eyeballs; in ten-to-one-hundred you look at waterfalls, CPAs, and conversion to magic moments.
The Copilot app has a command called orchestrate for horizontal scaling across repos. Orchestrate lets you set a series of GitHub issues as goals for the agent and spin up sub-agents to address all of those tasks. It spins up agents in the repository they're supposed to be working in, so the agent has permissions in that repository.
Create a repository named .github private to expose agent files at the org level. Agents published there appear in the cloud agent dropdown and can be called on the platform. Publishing an agent in the .github private repo pushes it to everyone's IDE or CLI environment so they can call the same agent. This is how you share an agent internally to everyone inside the org; it is purely internal, and an admin can push it out.
CodexExec spins up a non-interactive interface great for scripting and CI-CD tasks like auto-fixing a PR, while the app server shines for comprehensive clients and spinning up many Codex threads at once (eval harness, slash goal) because you get full harness control without hand-rolling parallelization.
You cannot dump a massive amount of retrieved context back to your agent; you need a router and summarizer. The router sends incoming prompts to the right place at the beginning, deciding who should handle it. The summarizer takes data from synced context and live context, summarizes it, and sends it back to the agent that sent the prompt.
The definition of 'which customer was upset last week' can mean something very different to one company versus another. Companies need skills like 'how do we determine if a customer is happy' encoded in their context graph. Document company processes — e.g., what to do when a customer complains and how to route to the right place — as skills in the context graph.
Use live MCP lookups for simple, single-action queries (e.g., a Stripe lookup for a specific customer). If you need semantic lookups or analysis beyond a simple lookup across a vast dataset, you have to sync a copy of the data. Prefer live API calls when you know you'll have simple lookups and retrievals because they're easier to build, easier to add on, and cheaper to run.
Every answer has to be traceable — never leave yourself in a position where your AI made a bad decision and you have no idea why. Making business-critical decisions on versioned or outdated data without provenance is dangerous: you need to know where data came from, when it was fetched, under what ID and scopes, who accessed it, what permissions were used, whether it was transformed, what it originally looked like, and what should invalidate it. Letting AI make decisions on data synced a year ago without checking provenance leads to bad business-critical decisions. Build provenance tracking into your platform so you can investigate when something goes wrong — dropping provenance tracking leaves you unable to investigate.
Viren Baraiya's take-away slide: the responsibility of a non-deterministic agent is to plan what should happen next — not to actually do things; the harness is the one that does the actual execution. The harness is a deterministic piece of code that actually understands what is involved in executing a piece of work. When the agent says a cluster is unhealthy and should be restarted, the harness should decide how to restart the cluster and what is involved in restarting it. This is the clear separation between, as he would like to call it, the brains and the hands: the LLM is the brain, the harness is the hands — the agent writes the plan, the harness executes the plan.
Viren Baraiya's operating rule: if the agent writes the plan, the harness executes the plan. When the LLM outputs steps for what should happen next, the harness executes them rather than the LLM doing it directly. Depending on how far ahead the agent can plan, you can execute multiple steps at a time.
Viren Baraiya argues that when determinism matters, the harness should deliver it rather than the LLM. If the LLM decides how to restart a cluster, you lose determinism and every restart could be different. When the agent plans a cluster restart, the harness executes it deterministically so every cluster restart is identical. Non-determinism slipping into critical operations like payments or cluster restarts costs you predictability and safety.
The last mile is where someone has to read the code, test the behavior, own the trade-offs, notice the missing requirements, and decide whether the thing deserves to be part of the system.
Not every line of code needs to be personally read, blessed, and understood in isolation — that has never really been the standard, and large systems have always contained more detail than any one person can hold in their head. But the logic has to be understood: the main flows, the boundaries, the data model, and product behaviors.
From the beginning, the team's hypothesis has been that one of the big bottlenecks in automated materials discovery is automatic characterization: mixing powders is easy, but without automated characterization and analysis you can't intelligently decide the next step. Mixing powders randomly yields little benefit — the value comes from automatic characterization and closing the loop with simulations, so you can try many things, make informed decisions, and plan the next day.
Rather than training on the final output of science, train on the process of doing science — the process data is the scarce, valuable resource. Build reasoning traces over entire scientific campaigns, not individual experiments: capture the full lineage of conversations, intuitions, lab executions, computations, and code, stitched together. The team says this type of data basically doesn't exist anywhere else and that they spend much of their time getting the full lineage of the scientific process into the model.
A classification algorithm can't be trained on all-positive samples — you need negative samples, which is why unpublished failed syntheses are a critical missing data source in materials science. The team deliberately seeks null results and learns from them: a null result means you intended to produce a structure and all evidence points to you not producing it, and sometimes the null reveals a previously unidentified structure. A string of negative results followed by a positive one is valuable 'process engineering' data — the iteration path to success is itself a training signal. They also run negative controls in materials experiments, because impurity phases can kill the property you're measuring, so controls are needed to distinguish real nulls from contamination.
As a product leader, the ideal time horizon to live in is two to three months out, not years. Predictions of the future years out are almost always wrong, so don't plan too far ahead. Tara keeps a personal doc of incorrectly predicted things she thought were going to happen and didn't, to remind herself of her inability to predict years out. If you build exactly for today, you're going to get left behind.
endorsed
Choose sources → to get a library of your own and a daily brief.