Explore

Stop babysitting: record mistakes, run retrospective agents, build memory

To stop babysitting the agent, let it make its mistakes but record them and keep track so it doesn't make them again. For every loop, run a retrospective agent at the end that reviews the loop's performance — what was done, why time was wasted, where corrections were needed, where it ran tools without changing things. Work through the retrospective with Claude so it builds up a memory file (Markdown) loaded at the next session, with some curation. Structure memory as short-term (filtered out over time) and long-term (core tenets you never want forgotten, like where the design system lives or how to test a component), broken up with progressive disclosure — only load testing memories when talking about testing, React memories when touching React files.

AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS

Schedules: durable cron-like recurring tasks in cloud sessions

Schedule is for durable, recurring work like a cron job (every two weeks, every Monday/Wednesday/Friday, every day) — it kicks off and saves a durable cloud session in a VM that does the work on a repeatable basis, and may involve secrets. Triggers don't have to be time-based; they can fire on events like new PRs or Slack messages. There's a whole interface: list what's running, modify, tune, locate, delete. Reach for schedule for anything tedious you'd like to automate — dependency bumps, status reports, eval runs, meeting prep, weekly PR/work summaries before performance reviews.

AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS

Multi-model adversarial review: Codex via hooks, fresh-context reviewers

Add a hook so that every time you work on something sensitive, urgent (unblocking a production issue), or of certain complexity, you must fan out the code diff for adversarial review from Codex via the CLI (install Codex, auth with your OpenAI account via OAuth). Codex ships skills — review, adversarial review, rescue — easy ways for Claude to call it and bring advice back in. A model still in the same session will confabulate helpfully and claim it completed the goal; a fresh model with a succinct prompt (inputs, outputs, goal) finds issues immediately. Adversarial Codex reviews find significant issues regularly, reducing overall time by catching issues before expensive builds and paid reviews. In an experiment with four different agents from different companies reviewing the same code,

AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS

Parallelization: Git Worktrees, claude --worktree, and cloud environments

Use Git Worktrees to give each independent agent an isolated, safe-to-work-concurrently copy of the repo; run 'claude --worktree' from inside your repo to create a worktree and start a Claude instance in it, enabling immediate parallelization. Break a full afternoon (5-6 hours) of development into discrete blocks assigned to run concurrently, pressing time down to 45 minutes or an hour. Caveat: running multiple instances of a large monorepo in different worktrees is difficult because of gigabytes of node_modules. Alternatively, use cloud environments (Claude Tag, Devin, others) to parallelize work in the same codebase without conflicts, opening independent PRs that never bleed changes into each other.

AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS

The safest way to put evals in production: make production an agreement gate on CI (red under 80%, green 80–90%)

The safest way to put evals in production is to make production an agreement gate: run evals on CI and don't ship if the judge agrees with you less than the 80% threshold. CI should be red on less than 80% agreement and green on 80–90% agreement. A failing eval test will block CI because it is red, and ideally you want no failing tests. Modify the eval test to only fail if human-AI agreement drops below 80%, otherwise pass, and console log the agreement percentage. When you ship the eval to production, it becomes the quality gate that runs on CI. The eval set provides ahead-of-time reliability; it will never really fail, and if it does, a harness can pick up the slack at runtime.

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

A well-defined eval suite has four components: test cases, expected behavior, a judge, and aggregation

A well-defined eval suite has: (1) test cases — a set of inputs and outputs, such as customer requests to a chatbot and generated answers, against which you can create a verdict; (2) expected behavior — the answers you expect, which is not an exact string but the direction you wanted the output to go; (3) a judge — somebody to say yes, that's right, which is harder in non-deterministic systems because it's subjective; and (4) a way to aggregate all results and come up with a probability score, e.g. 'this is 80% the right direction, therefore it's good.' Almost every eval set in production today of any quality has these components. Eval components come together to answer: given a scenario, did the agent follow policy? Did the agent use the right tools? Did the agent ultimately resolve the (

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

To train a judge: build a golden dataset of ~30 question-answer-score triples scored by a human, remove your scores, let the judge score, and compute agreement

To train a judge (not ML training): you have an LLM you want to use as your judge, and you make sure it agrees with you using a dataset. Step one is to have a long list of data — question, answer, score triples — where the score is one that you or a subject matter expert human has scored, a trustworthy score. Start with about 30 examples, then let it grow. To train the judge, remove your scoring from the dataset, keep only the question and answer, send that to the judge, and let it produce its own scores. Compute the delta between your scores and the judge's scores to find your percentage of agreement. Aim for 80–85% agreement, maybe a bit more. Under 80% you won't get much value, because even a team of human support agents only agree about 80% of the time. If your judge agrees 100% of the

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Once in production, eval data goes stale: skim real traffic, have a team score it manually, and check judge agreement to stay relevant

Once in production, there is a high chance your eval data will go stale, so you need to actively stay relevant. To stay relevant, skim real traffic: collect new user inputs and your agent's outputs, have an internal team score them manually, and check the judge's agreement against those scores. Your eval data set should be a living data set — continuously updated with fresh scored traffic. A living data set with an up-to-date judge is very difficult to break and provides reliability ahead of time. After skimming public traffic, you will have even more data to refine your evals. In production, skim the top of the data, get a little more data, and refine and refine.

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Evals are afforded strategically: start with deterministic tests, then regex, then traces, then a judge with the cheapest model

To keep evals cheap, start with actual deterministic tests (unit tests). If deterministic tests can't be done, the next level up is regular expressions or pattern matching. If pattern matching fails, the next thing to do is examine your traces. Ideally, everyone is using some type of observability solution and has traces of each message envelope, inputs, outputs, tool calls, etc., which can be validated per object. Eventually you will come to a place where you need to use a judge, and then you start with the cheapest model. Evals are afforded strategically, because it will not be cheap if you use the most expensive model, and you want to build up to it. Regular expressions can be used for testing AI outputs but have their limits.

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Judge training is iterative: investigate dissonance, find the knowledge gap, fill it with policy; agreement grows toward the threshold

When humans and AI don't agree, it's usually because of a knowledge gap — and the way to reach agreement is to introduce context to bridge that gap. To improve judge agreement, examine the cases where you disagreed and fill the knowledge gap with policy. In production you would supply policy to the judge via a RAG pipeline that retrieves enterprise context; in a demo you can just hard-code the policy. Write the policy explicitly (e.g., a 14-day return window; defective items can be returned) and interpolate it into the judge prompt. Building your judge is arduous iterative work — investigating failures and filling knowledge gaps is literally the job. After adding policy, check whether agreement is growing even if it still fails — improvement indicates you're on the right track. When the

AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM

Engage the account immediately at AE-to-AM handoff and verify the executive outcomes

The minute the account is yours, engage right away so the value loop starts turning — the AM not being involved in implementation is a mistake. At handoff, verify (don't assume) the executive outcomes the AE used to win the deal, grab them, keep the CSM on target to them, and deliver those as the first set of value; that proof is what lets the next outcome happen.

30 Minutes to President's Club | No-Nonsense Sales · #604 - Upsell & Expansion Masterclass: 7 Lever to Grow Any Account

Environment ID routes to the same sandbox; state persists across turns

The environment.id parameter enables routing to the exact same context so you can use the same sandbox. As long as you preserve the environment ID, it will return you to the original sandbox you had. Any packages installed or files created remain accessible to the model through each turn, as long as it passes back the interaction ID and the environment ID. You don't need to manage any persistence of state yourself; you only need to add a few lines of code.

AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind

Mix and match inbuilt tools in a single API call; the model reasons autonomously

With the Interactions API you can mix and match different inbuilt tools, which wasn't possible before. Example agent setup: give the model Google Search (the same index Google has), the URL context tool to retrieve information from web pages, and a custom tool called File Incident, to look up the latest security reports for a React application. With mixed tools, the model can reason autonomously, figure out what information it needs, and generate a response — all in a single API call. Giving models internet access makes them powerful because they can work with information beyond their training cutoff date.

AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind

Interaction ID preserves context across turns via previous_interaction_id

With the responses API, the first turn (e.g., 'hi, my name is Phil') returns an interaction ID; passing that ID back as previous_interaction_id on the second turn preserves all context. The interaction.id parameter lets you preserve and work with context and ensures you're not busting your cache.

AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind

Managed agent provides a persistent sandbox out of the box with a single API call

In the past, building a coding agent required queueing in the harness, finding a sandbox provider, managing the infrastructure, and figuring out how to preserve context between runs. With the anti-gravity managed agent, a single API call gets you a persistent sandbox out of the box that you can keep hitting and treat as a personal cloud without any modifications.

AI Engineer · An Interaction Is All You Need — Ivan Leo, Google DeepMind

Regulatory Strategy: Self-Regulation vs. Nationalization

Industry self-regulation (e.g., Hollywood's MPAA) can prevent heavy-handed government intervention. However, if risks are deemed 'non-zero' for catastrophic outcomes, the logical regulatory response is nationalization, as seen in critical infrastructure sectors like banking and power.

medium

The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

Grounding AI Safety in Concrete Risks

AI safety discussions should be grounded in concrete, identified risks (e.g., novel cybersecurity threats) rather than speculative super-intelligence scenarios. Security claims should be validated using physics-based and systems-based constraints (e.g., entropy, bit rates) rather than abstract 'super-intelligence' capabilities.

The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

The Shift to Probabilistic Programming

The center of AI innovation has shifted from model creation to the integration of models into traditional software systems. This involves using language models to return probabilities for conditional logic (e.g., 'if X percent') rather than binary true/false checks, reviving 1960s-70s probabilistic programming research.

medium

Commercial context: The creator presents this as their own offering and directs viewers to get it.

The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

AI Agent Threat Model: Untrusted Oracles

AI agents should be treated as untrusted entities with oracle-like capabilities in threat models. Unlike human employees, they can execute thousands of security probes rapidly without fatigue, requiring granular access controls and monitoring of internal APIs for denial-of-service style attacks.

The a16z Show · Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

Recipe for improving long-horizon agents: freeze harness, define scenarios, log traces, score behavior, expose small policy surface

Erina Karati lays out an ordered recipe for long-horizon agents: (1) freeze the harness, (2) define scenarios, (3) log traces, (4) score behavior, and (5) expose only a small policy surface. The steps are presented as an ordered method for making agent behavior measurable and safely improvable.

Method, not a verified protocol.

AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati