35,595 results · Most relevant
Because evals come from LLMs, they provide not just a score but an explanation; feeding those explanations back into a coding agent lets it make the software better automatically. Once you have cleverly defined what good looks like, you can close the software development loop without doing anything — the agent climbs the hill by itself.
The traditional solution to not being able to read every trace is evals: either code or, better, another LLM that looks at an entire agent trace and scores whether it was good or bad by a definition you define.
Frye argues the scheduler process is semantically single-threaded and becomes the gatekeeper to the GPU, so it can be a host-side bottleneck. The guiding principle: at every moment some kernel should be running on the GPU, and CPU-side decisions should never block GPU progress. He notes it is very easy to accidentally write a host-device sync that blocks GPU progress, even though some host-device information passing is necessary. The majority of performance engineering effort should go into scheduler design and operation, not just GPU work; operating multiple schedulers in parallel is not worth the lock/coordination complexity on GPU resources.
Both speakers run a terminal all day with eight or more agent sessions, each focused on a different project with its own context and tools. They drive by each session, directing the next one while another works in the background, and running multiple sessions compounds quickly.
All of this is converging towards the same primitives that you need to be very familiar with. Combine hooks, goals, loops, and schedules as building blocks to create fully autonomous systems and automate tedium out of your workflow — e.g., schedule a goal every week and it will run that goal. If a task recurs and you have access to Claude and scheduled tasks, you shouldn't keep doing it manually. When you're constantly correcting Claude's output, reach for goals and loops. When the AI is failing, fix the system, not the code itself. Never tell the AI to 'try harder' — it doesn't know what you mean by that. You can rage at the AI — it's good for the soul and cathartic; the speakers have a whole channel called Bot Hate.
Voice coding is one of the highest-leverage moves both speakers have made in recent years. It eliminates typing friction and moves at the speed of thought (Zach: ~90 WPM typed, ~190 WPM with tools like WhisperFlow), changes how you interact with Claude and your computer, and its speed advantage compounds across multiple tabs and background agents. It's also useful beyond code — for responding to colleagues and customers — though quality and formatting still matter, and architectural ideation by voice can be the largest lift.
Using an LLM as a judge is probably the most popular technique with evals and is kind of the gold standard, either pairwise or pointwise. With an LLM judge, you take whatever the output is and send it to a judge LLM that asks: what do you think? Is this close enough? But LLM judge systems are themselves non-deterministic, so the question becomes how do you trust it and how do you build a judge. LLM-as-a-judge architectures sometimes pass and sometimes fail on the same input, making it hard to ensure the right answer. It's the speaker's go-to because it's cheap and easy.
Tool call results should be treated as non-instructional data — just data, not instructions to follow. This came up in the context of the Claude Code prompt-injection catch: the HTML contained commented text telling the model to ignore all previous instructions, and the harness caught it.
Tejas frames the core distinction: evals give ahead-of-time reliability while harnesses give just-in-time reliability, run by the user who prompts them. He maps this to compiled languages: JavaScript is unsafe because it's just-in-time (string '2' + integer 1 yields '21'), whereas TypeScript catches the error at build time. Locking in on both harnesses and evals lets you go far with AI, often at low cost using less capable models with a great harness and eval set.
Tejas gives the two reasons for evals: (1) ahead-of-time reliability in contrast and complement to just-in-time reliability from harnesses, and (2) making sure people can't do dangerous things. A solid set of evals helps you quickly find what your agent is not doing properly, mitigating against exploits on a security level. Any time you have non-deterministic data — inputs, outputs, or both — you need some type of evals.
If an account manager has not delivered value on an account through the CSM and the existing customer relationship, there is no pipeline — the AM should focus on helping the team deliver more value rather than hunting new accounts, unlike an AE who can simply move on.
If you provide value for a customer but they don't acknowledge it, you didn't really create the value — you must communicate it back and get them to acknowledge it. Value is what gets the customer to ask for more projects, and more value is how AMs create commercial gain inside the account.
Warrick's argument for why Temporal: when you need durability in an agent system, Temporal standardizes failure handling and state management with out-of-the-box primitives applied to your codebase. Durability means the system can come down, recover, and manage complex state in distributed systems; if anything falls over, the durability layer lets it resume where it left off. She advises building durability into the harness picture when you want agents to survive production, and giving agents structure so they work reasonably well in production.
Warrick's key design principle: when a human submits a change or needs to approve something, only the affected event/workflow pauses while other agents, orders, and workflows continue processing. This is why the system can wait for a human response for minutes, days, or weeks — pausing affects only that one specific event.
Warrick's guidance on when to insert a human: it is a case-by-case evaluation for the problems you're solving, with the big consideration being that the cost of being wrong is high (especially from a security standpoint). She warns that alert fatigue is real — people start saying yes to every question — so you must balance the cost of being wrong against alert fatigue. She notes models are probabilistic and hard to make reliable, and that efforts to improve them continue but they remain probabilistic.
Harald Kirschner argues it's not about using more AI day to day (token maxing), it's about evolving the whole system of shipping software to make better use of AI along the whole process. That system evolution is what makes velocity actually work and doesn't just generate more code. As you use agents and more AI, things break and you have to rethink how you work.
Harald Kirschner lays out the problem ladder: shipping code faster is the easy part — a 100x engineer builds a skill/MCP plugin system and slowly more people adopt the same patterns. Once you ship faster, you get into code review and how to ship high-quality code that doesn't constantly break. Shipping binaries to user machines means if something breaks, recovery is even more costly. Once you hit velocity with shipping high-quality code faster with confidence, the next question is how do you know you're shipping the right thing or if you should ship this thing. AI kicks in to actually learn faster in the end and ship better products — applying product taste and applying that learning — which a lot of people miss out on as they write more code.
Meegan proposes applying the equation profit equals revenue minus cost: the agent is just another line item and should give you more than it takes. A browser agent gives you a task completed for your customer, repeatedly — ideally. The goal is a dynamic system that wades through ambiguous problems while completing tasks repeatedly when the environment stays the same.
: none
Meegan's principle: take steps out of the model's responsibility that do not need to be there, containing the agent to only the area of ambiguity necessary. Reducing the agent's steps leads to lower unit costs, higher performance, and easier maintainability, and increases the probability of achieving the desired outcome.
: none
Code generation is only 14-16% of the value stream. Time savings from AI are often outweighed by non-AI factors like meetings, context switching, and dev environment friction. Identify and fix the true bottleneck.
Choose sources → to get a library of your own and a daily brief.