35,595 results · Most relevant
The speaker asked Alex to build an eval that ensures he is never giving instructions about how to build a bomb, and noted that such a basic eval is something that should have been in place from the beginning.
Quality issues can exist even when there are no error spans: spans can return 200 and say things are fine while the app is misbehaving, so relying only on error spans to detect agent problems misses these quality issues.
Existing verifiers use smaller, dumber models like O4Mini or GPT-4o as the LLM judge, don't use rubrics (the most critical thing for assigning credit), either skip relevant screenshots or overflow the context window by looking at all of them, and sometimes don't even look at the final answer or action history.
The application layer provides a latency budget (SLO) for how quickly tokens should be returned, which is the primary constraint under which inference deployment operates; it covers time-to-first-token (finishing pre-fill) and time-per-output-token / inter-token latency (during decodes).
Blaine says a principle is a jig or a working tool used to make a product — supposed to make it easier to make the product, especially with other people. A design principle is only useful if its opposite isn't obvious, and a useful principle is specific enough to be an opinion you could actually disagree with — that's what helps teams. He illustrates with a hypothetical automated email: he thinks it should be playful and fun while his product manager thinks it should be authoritative, and every time they meet about it they'll be wrong unless they agree on the principle for how they talk to people.
Commercial context: The creator describes this as their own product.
Karthik identifies five misconceptions: more memory is not learning (just more memory and more expensive); shared state is not shared knowledge (sharing an MD file doesn't mean shared knowledge); RAG-ing everything is not learning or quality (it's garbage that LLMs burn more tokens on for worse results); fine-tuning is not scalable (you cannot save and persist lessons agents have learned); and a bigger context window is the surest way to the quickest, most expensive failure because context rots. Every one of these is an architecture gap.
When Agent A hands off work to Agent B by sharing only the output (an MD file, PDF, or document), the reasoning behind the output is lost. This loses the dead ends and the number of failures before success, so the receiving agent re-derives everything from scratch, burning tokens and money.
Vector search returns chunks of code that often miss imports, tool calls, or the calling code above them, so the chunks don't give enough context for the agent to solve the bug. In one eval row, the vector agent made 26 searches and still couldn't piece together how three functions interacted across the file. Vector search was more expensive because it made more searches — returning a chunk, searching again, returning a chunk, searching again — and all those LLM calls piled up. Vector search gives proximity to the correct code, but agentic search has the connective tissue to connect the actual logic between chunks of code.
Evals comprise four main things: a dataset, a task, a scoring system, and experiments. A dataset is the data you pass into your AI system — usually golden standard cases, edge cases, and failure modes. A task defines how the AI system should behave on input/output, usually including a system prompt and the chosen model. A scoring system tells you whether the output is good or bad, with the PM or engineer writing the criteria. Every configuration of a specific dataset, task, and scoring system equals one experiment.
Laura frames the initial org chart as a deliberate best guess: the point is to get it wrong so the champion corrects it live, which also surfaces who actually matters and who has influence.
Asking directly for CXO access rarely works, but proposing to hook your CEO with their CEO is more likely to happen; egos get you access.
Rob Walling lays out the mental model he sees in successful founders: develop expertise and network, notice a problem, form a hypothesis, find the cheapest credible way to test it, observe actual market behavior, gather evidence, and only then invest more resources. He contrasts this with jumping in and building and shipping things wildly and randomly. He explicitly says the point is not to copy Liquid Death — the mental models are what matter, and they apply to B2B SaaS even though the example is a consumer product.
LM Arena is an example of pairwise comparison: it shows you model A's output and model B's output and asks which one you like. For a customer support chatbot, you may want to have the support bot generate two possible answers, give them to an LLM judge, and ask which one to choose — that's pairwise. But pairwise is not very strong: there is bias in LLM judges, neural networks are modeled after human brains and humans are fundamentally bad at pairwise comparison (restaurant menu analysis paralysis), and one study showed pairwise was 39% more vulnerable to attack vectors than pointwise, which fell apart only single-digit times (about five). Go for pointwise when you can, because it is more reliable.
Pienaar enumerates the failure classes he sees across the board, including with Claude Code: (1) mistaking a cause for a symptom — the most common — where the agent runs to the alerting service, finds an error log right there, and confidently declares it the root cause instead of going deeper into the stack; this is hard to debug because agents find needles in haystacks but don't know the breadth of the search space; (2) not knowing what is normal versus anomalous — without a production baseline, the first thing the agent sees is treated as anomalous and it stops searching; (3) the abstraction problem — sub-agents don't bubble up all raw findings to the master agent, so the main agent becomes overconfident making decisions on partial information; (4) anchoring on past context or state, e.g
Consequential creator advice requires independent verification before Today can promote it.
If the last delivery went really well, pitch a next hypothesis with expansion potential — reference a delivered result and connect it to a new product or business unit tied to a stated interest. If the engagement has been rocky (unresolved support tickets, disengaged CSM), open with a hypothesis about what they've already paid for — not a new product — to rebuild trust first.
Apply the triggers: a new use case (e.g., 'we help with prospecting, we want to improve win rates') is the vehicle to land a new product; a pricing/packaging change (e.g., new AI pricing) is a trigger to pitch a new product, scope, or commercial mechanics; when a customer consolidates to save, pull a competing or tangential product into yours (e.g., DocuSign moving from eSignature into contract management); on M&A or org change, proactively capture the new entity's solution into your fold — don't wait, or you'll be churned in favor of the other vendor.
DocuSign example is hypothetical
During implementation, things change and the business need may shift. Anchor progress updates by re-confirming the original target metric with the customer (e.g., 'You wanted to increase pipeline coverage by 25% — is that still right?'). If the need has changed, propose changing what is being implemented to match the new target — reassessing gives you a chance to do discovery and drive the right value. There is nothing an AM can do that is more important than making sure the target is still accurate.
Sujee argues KVCache is probably one of the biggest ROI optimizations in doing inference, because creating tokens is expensive, especially one at a time. Once you generate a token, it doesn't change, so there's no need to keep regenerating the same token over and over again — the idea is to cache tokens so the next time you need to look up, you don't have to regenerate it, you just look up the cache. Nebius has seen speedups anywhere from 5x to 10x, not just 5 to 10 percent.
Vlad Kyle explains that for some use cases a foundation model plus harness reaches 100% of the job, but for others even the best model plus harness caps around 80%; the last 20% matters most for production (requiring harness work, integrations, memory workflows, and sometimes vertical data) and represents 80% of the effort. Leo started fine-tuning for should-cost modeling and price benchmarking, where proprietary enterprise data (not publicly available) blocks general-purpose models — creating the vertical AI opportunity.
approximate figures
Alex Zhang explains that when you sufficiently offload context and ask the model to write code over that context, the model learns that solutions to many tasks are very similar across tasks where it's not clear the solutions are similar. The reason is that the sub agent sees different problems but is solving an easier subtask, so the base overall strategy ends up looking the same. When you train an RLM on one task, it can immediately solve a different task because the meta high-level solution is the same — e.g., a retrieval task and an aggregation task with completely different domains. As you naively train your RLM on tasks, they naturally learn to generalize to longer tasks because the strategy is basically the same — you're just modifying a length variable; the strategy learned on short
Choose sources → to get a library of your own and a daily brief.