34,785 results · Most relevant
For an intelligence producer, rent is one part of the cost. The operator still has to turn compute cost into work the customer can actually use. To compute full cost per useful output, count everything spent to deliver the service and divide by the output the customer can actually use. Operating capability affects both sides of that equation: what you spend and how much useful work you can deliver.
Owen models RSI progress as a ratio: capability improvement rate equals performance (numerator) divided by cost/difficulty (denominator). He stresses always accounting for the denominator — when evaluating AI progress claims, divide capability gains by capital raised, since model release times may be shrinking but improvement normalized by investment may not be.
The host explains that once you save a search, every new person fitting the criteria automatically appears in it, so you never have to recreate the search — you just respond to new prospects.
When evaluating an acquisition-prediction system against the live web, searching 'now' leaks future knowledge (the acquisition is already visible), producing a false 100% true positive rate. The fix: use Snapshot to set the snapshot date to just before the outcome was announced, so the only information informing the system doesn't leak the exact confirmation of that outcome. For example, when designing a system to understand company acquisitions, you might search one day, one week, or one month before the acquisition to check whether the search would have found the relevant signals.
When a pilot tests poorly with the general audience but a small group loves it, treat it like a startup: aim to be a 9.9 for a very small group of people rather than a 7 for everyone, then get those superfans to tell their friends.
'Status comes from records, not model pros.' Show me the receipts — the actual tangible proof that it did what was expected. Don't just accept 'yeah, it passes, it looks good' from an agent — there's nothing to back that up; demand actual screenshots, videos, test results, and smoke tests.
If your web app does not already have a bespoke CLI that allows any agent to fully flex the functionality you care about verifying, you'll have a much rougher time and an unpaved path using prompts alone. Give each agent the ability to say: yes, I actually built that feature, it is actually performant, it didn't cause any errors, and the application looks correct.
The tired way — one giant massive PR from a single agent session running three or four days being told 'fix this, fix this, fix this, make it better' — is being replaced. What works essentially perfectly: decomposing tasks really cleanly, giving each task an agent, and in the prompt giving exactly what it needs to do to know it is actually done and to verify the state of done — those will just rip.
Putting a lot of verification instructions in a prompt is what a lot of skills and saved prompts do, but if you forget anything or things change, it leaves a lot up to interpretation. Instead, have a script that can prove the work out in different ways so you can run it over and over as a deterministic check. The script feeds its output back into the agent so it knows exactly what was wrong and how to fix it quickly, closing the loop. The agent can spin the script, run it again, check, verify, and repeat over and over until there are no more errors. LLMs are not deterministic — giving 30 agents the exact same prompt has potential for variance in outcomes — so pinning verification down to a single command backed by actual bash commands and bash calls is significantly more deterministic.
There should be a separate supervising agent that decides what the worker agent gets to do, creating a separation of concerns where the supervisor can't do any of the work itself. If a user tells an agent to go read the CEO's email, that is outside the scope of that agent and should be rejected as intent. A request to read somebody else's email would probably get caught in deterministic policies right away because it's not a vague thing and good controls exist for it. Telling an agent to forward the CEO's email to the outside world is vaguer and is the kind of thing that ideally comes back to a supervising agent, which would say that's probably a bad idea and no, you can't do that.
The creator describes that without intent — some capacity to relay the user's or business's intent — we can't get to agent autonomy. The creator recommends giving the agent the plan — what you're trying to do — as context, so that a mission identifier tracks through everything and authorization decisions are made in the context of that mission. A mission identifier also lets you audit and see all the things that happened and the decisions made in the context of one mission. Today there isn't a good concept of agent identity or the mission — things make decisions without knowing which agent instance is doing it or what mission it's involved in.
In the demo, Voss gates the build on acceptance criteria: mentions-ticker is a guardrail (the agent must talk about the right company) and actionability is a target (the average must stay 0.8 or better). He gates on the average score rather than a single report because an LLM judge can be a bit noisy on a single report.
Laurie Voss describes several biases in LLM judges. Position bias: given two options, depending on the model, they tend to prefer whichever comes first or last — so swap the order of options presented to a judge so it's not always looking at them in the same order. Length bias: judges prefer long answers. Confidence bias: they prefer answers that sound fluent and confident. Self-preference bias: a model judging its own output tends to rate it higher — so judge haiku with sonnet, and ideally judge haiku with something from OpenAI, to avoid self-preference bias.
When a capability eval passes reliably (100% of the time), Voss says you don't throw it away — you turn it into a regression eval. A regression eval is the kind you put into CI: it used to pass 100% of the time, and if it suddenly stops, you call that a failure and stop the release.
The important thing when improving an agent is whether anything that used to pass now breaks. In Phoenix you can filter down to the passes split to check whether previously passing reports still pass. In Laurie's case, all five reports that passed previously still passed after he improved his prompt. He also notes that when a test passes 100% of the time you don't throw it away — you turn it into a regression eval.
When improving an agent there is an order of operations based on how useful a change will be. First is data quality: if the underlying data is bad, hard to read, or difficult to interpret for the agent or the judge, nothing else you tweak will be useful. Second is prompts: once data is good, move to the prompt level by changing your rubric or agent. Third is model choice: it has a lot of leverage but changes many things at once, and the usual change (a bigger model) makes things more expensive — a trade-off you don't always want. Fourth is hyperparameters: they are so useless that modern models have stopped allowing you to tune them — you can't pass temperature to modern Claude models, and if you're tempted, it's the least useful thing you can pass.
Precision asks: when the judge says 'not actionable', how often is it correct? Recall asks: of all the reports that aren't actionable, how many did the judge catch? The two sound very similar but measure different things. In most cases you should prioritize recall, because high recall catches everything but will give you false alarms. A false alarm costs you a minute or two when you look at a report that turns out to be fine, but a missed failure gives a bad report that reaches a customer. If recall goes up, precision will go down, and vice versa — you have to trade one for the other. For your application, you have to decide whether discarding some actually-correct things is better or worse than getting false alarms where you flag something important that turns out not to be.
Laurie Voss advises treating eval prompts as code, because eval prompts going into a custom eval are extremely sensitive to small changes in wording. As an illustration, he forgot to put XML tags around his examples; with examples as plain text, the judge returned a string of not-actionable results because it was confused about what was data and what was examples. When he wrapped the examples in XML tags, everything suddenly became actionable — with no other change to data, instructions, input, or output. The Coca-Cola report was the example that flipped when the prompt changed, showing the judge got better with the examples.
Laurie Voss warns that a trap with evaluators is trying to write one evaluator that does everything — the 'God evaluator'. The best practice is to split evaluators into as many dimensions as you are testing — one for faithfulness, one for correctness, one for actionability. He also advises knowing which one of your evals is a guardrail and which one is a North Star.
After launch, Voss says you should keep evaluating: run your evals on a sample of real traffic and alert when scores drop and stay down. He warns against alerting on every single failure — one failure would be noise — and says only a trend in failures is worth alerting on.
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Choose sources → to get a library of your own and a daily brief.
Map each implemented component to the original paper and explain 2 reduced-model failure modes without claiming full-scale reproduction.
Use calibration and shift measurements to justify a monitoring or abstention threshold, and give an explicit confounding counterexample to causal feature-attribution claims.
Defend a causal claim or refusal using the DAG, numerical estimates and a counterexample showing why predictive model fit cannot establish identification.
