Turn capability evals into a regression suite over time
As your agent matures, turn capability evals into a suite of regression evals that you run all the time to make sure previous behavior has not regressed.
35,595 results · Most relevant
As your agent matures, turn capability evals into a suite of regression evals that you run all the time to make sure previous behavior has not regressed.
The speaker advised writing code evals with no model configuration and no API calls — just Python — for checks like whether the expected stock ticker appears in the report, output length limits, parsable JSON, and forbidden phrases; an LLM judge is overkill for string matching. Code evals can also do complicated things like querying a database or hitting an API to verify stock prices, as long as the grading logic is deterministic.
Generate a body of test data — you need more than the one trace you've generated so far to run meaningful evals. Include a bunch of obvious queries that should work, edge cases (e.g., two tickers like Apple and Microsoft for a comparison), and the same tickers multiple times so you can see the non-deterministic output. Different tickers, questions, and complexity levels cover the range of things users will actually ask. In reality, users will ask really weird things you can't predict in advance — another reason evals are important.
Extract the research context from the traces produced by the first step, attach that output to all spans as the context for the judge, instantiate the faithfulness evaluator in code just like the correctness evaluator, and after attaching context at the column level call evaluate_data_frame with those spans and the instantiated evaluator.
Eval prompts are very sensitive to wording — treating them like code without versioning and testing turns evals from a useful tool into noise. Version your eval prompts and test them on examples where you know the right answer.
Karthik recommends storing reusable agent knowledge in a shared memory system rather than empty files, so retrieval costs fewer tokens.
endorsed
Pienaar's core recommendation: replicate what works in coding by giving the agent critical signal that comes back quickly, so it can ground on it during or at the end of an investigation. In coding, the agent tries a change and gets immediate feedback — it didn't compile, or here's an error message — and that grounding loop is what to reproduce for production, where sync time can be on the order of hours.
After the open, make the conversation about the woman without asking any questions — elaborate on what you noticed about her in the moment. Going straight into interview-style questions kills the interaction; most men do this wrong.
Cut the chilled shrimp into roughly peanut M&M sizes (about a half inch). Don't go too small — a shrimp roll is all about large, plump, sweet pieces of shellfish with full texture. Slide the chopped shrimp into a medium bowl, then agitate them with a little paper towel to absorb any last bits of surface moisture, since water on the shrimp goes into the mayo dressing and makes the salad wet and sloppy.
Add 120 grams of mayonnaise (Duke's is the best off-the-shelf mayo, in the creator's opinion), 3 grams of salt, and about 15 grams of minced chives (saving the rest for topping). Zest one whole lemon with a microplane (a box grater works; avoid digging into the bitter pith), then chop the zested lemon and add 15 grams of juice — you can go a bit more if you like things zippier. Stir very gently to combine, then taste mainly to check that salt and lemon levels are dialed in.
Weigh out 3,000 grams of water into a deep stock pot. Add 120 grams of salt and 90 grams of sugar. Squeeze in the juice of two lemon halves and drop the squeezed lemons directly into the pot. Bring the whole thing up to a simmer.
Use 600 grams of red Argentinian shrimp. Kill the heat on the boiling water, then gently add all the shrimp. Adding the shrimp cools the water from a high boil into poaching territory, around 190-200°F. Set a three-minute timer and give the shrimp a quick stir to distribute them evenly.
Harald Kirschner recommends using a self-correcting loop — because VS Code runs a web browser and opens a website (it's all HTML), Playwright can automate a lot in the browser. Use a slash launch skill that opens up VS Code and gives it a specific scenario to click through in VS Code. The slash launch skill is a powerful way to diagnose issues because you can also get the logs along the way. Use the launch skill to validate any fixes before and after without having to click through it yourself — hit launch, move on to the next task, and come back once the agent verified its work.
Harald Kirschner recommends investing in a component browser — an automated build that runs every time VS Code changes something. Example: adding a back button to the customization screen changes width, triggering the automated process that takes screenshots of every component in VS Code and points out the differences. The component browser allows detecting unexpected changes — like changing one component causing the ripple effect of something else moving or some other icon disappearing — and allows quickly reviewing PRs because you get directly a view on what the PR is actually doing. Previously, VS Code asked every developer to at least attach a video or screenshot of what they implemented so reviewers don't have to run everything from scratch; the Component Explorer automates attaching,
Laurent recommends an automated harness that picks the right model at the right time based on task outcome — this is what Cast AI built with Kimchi Coding.
endorsed
Jan Čurn endorses progressive tool discovery — adding tools to context only when needed instead of all at once — noting it saves a huge amount of context and makes agents more effective, cheaper, and faster. He reports Anthropic introduced a Tool Search Tool in Claude that finds tools and adds only those to context when needed.
Endorsed with moderate confidence.
Alex Zhang finds that the most successful research from grad students or in academia comes when people care about problems that most people in industry are not looking at. He advises grad students to avoid working on benchmark harnesses and meta-harnesses for popular tasks just because they look good to industry labs, and notes that when people react to your idea with 'what is even the purpose of this?', it's almost a good sign that the field isn't thinking about the problem yet.
Alex Zhang advises that as a PhD student you're in a unique position to work on literally whatever you want, and the biggest advantage over anyone at an industry lab is freedom from bureaucracy. If you're unwilling to take big bets in research, go to an industry lab instead of academia. Expect many of your big bets to fail — that's natural — and recognize that one of the luxuries of being a PhD student is that there are so many big bets to take, most of which will probably yield nothing, but it's an exciting time to be in research.
Developers should prioritize automating 'rote and simple' work that constitutes the majority of economically valuable tasks, rather than chasing complex AGI. This includes automating easy work before hard work.
Seek roles in companies that have achieved product-market fit but are not yet fully systematized, as this phase offers maximum opportunities ('at-bats') for creating scalable systems and rapid career growth.
high
Choose sources → to get a library of your own and a daily brief.