Explore

Many published RSI papers fail when tested at scale; simple loop-closing approaches are what scale

The guests report that many papers claiming recursive self-improvement fail when tested in their internal infrastructure at scale. They attribute this to papers being cool, complicated human-introspected ideas that sound good but don't scale on benchmarks or real economic problems. Their finding: the simpler thing that closes the loop is the thing that scales.

based on the guests' internal testing experience

The a16z Show · The Race to Recursive Self-Improving AI

Recomputing shared prefixes wastes latency and GPU cycles; reuse cached tensors to cut TTFT

Recomputing a shared prefix on every request wastes both latency and GPU cycles; when a new request shares a prefix with a previous one, reuse the cached tensors to skip prefill computation and directly reduce time-to-first-token (TTFT).

AI Engineer · Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI

LLM judge system prompts may leak unredacted PII or MNPI

Al Chen challenges listeners to look at the system prompts for their LLM judges and check the traces to see if the traces, with the judges' reasoning, contain any PII or MNPI. A seemingly innocuous line in a judge's system prompt — added so a human reviewer can verify the agent did a good job — may have allowed the judge to leak unredacted PII or MNPI, so the UI looks good but the backend judge's reasoning may unredact customer information.

partial

AI Engineer · Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco

Firing leaders and regression to the mean

Coaches frequently get fired after a losing season, and CEOs get fired if the company has a poor year. When a coach or CEO is fired after a poor result, the replacement usually does better the next year because about 70% of the variance in outcomes is due to factors that are random with respect to the leader. Diamond compares firing a leader after a bad result to firing a dice thrower who rolled a one: for statistical reasons the next thrower will do better five out of six times.

Andrew Huberman · What Makes Great Leaders & How to Live an Extraordinary Life | Dr. Jared Diamond

Keep notebooks and make lists as a core method for memory and research

Diamond began keeping a notebook and making lists of items in categories he wanted to master (rivers, animals, birds), recites lists from his field of expertise when falling asleep (sometimes by foreign names, sometimes Latin names) to reinforce memory, and reports getting a lot of scientific mileage from making lists to extract generalizations from large datasets.

anecdotal

Andrew Huberman · What Makes Great Leaders & How to Live an Extraordinary Life | Dr. Jared Diamond

How LLM tool calling works

Pamela Fox explains that tool calling (also known as function calling) is the basis of all agents. LLMs are trained during training to look at function definitions and user input and decide which function to call with which arguments. The full JSON schema of a function — its name, description, parameter names, types, and descriptions — is sent to the LLM, which decides whether and how to call it. The LLM itself doesn't call the function — it only returns the function name and the arguments; executing the call is up to the developer.

AI Engineer · The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

Exa Snapshot: content versioning and version search

Exa Snapshot is a new capability that lets you see what search results would have looked like at a point in the past. With content versioning you provide a URL and a date and Exa shows the content as of that date or the nearest date before it, guaranteeing a version that would have been served at that time. With version search you execute a search as you would now, but as it would have been executed in the past.

AI Engineer · Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa

Three key questions for evaluating acquisition targets: trusted customer base, human judgment, regulatory accountability

Naidu outlines three key questions Long Lake uses to evaluate acquisition targets: (1) is there a customer base that trusts the business with real end-to-end outcomes, (2) is there human judgment in the service being provided, and (3) are there regulatory reasons that govern accountability when making decisions.

AI Engineer · Why Buying Companies Beats Selling Them AI — Prathik Naidu, Long Lake

The learning loop: how it works

The learning loop works by: the product does something (responds to a customer ticket, closes an invoice), the customer responds (pays the invoice or is happy), the system observes success or failure of that outcome, and then the next action improves, ideally automatically.

MicroConf · SaaS Moats in the AI Era: Sell the Outcome, Not the Tool

A defensible learning loop requires a real-world clock

The ideal learning loop requires a real-world clock — observations that happen in the real world, not something you can get by simulating the outcome. Real-world learning is not fast-forwardable, so competitors cannot simulate the outcome and catch up.

Also discussed by Startups For the Rest of Us (unclassified source — not counted until classified). Not counted as independent sources.

MicroConf · SaaS Moats in the AI Era: Sell the Outcome, Not the Tool

Reallocate a lower-priority budget to the project you believe in

When you need budget for a project you believe in, cancel a lower-priority planned item and reallocate its budget — the late night and specials department canceled a Bob Hope special to fund Seinfeld's first season.

Acquired · Seinfeld

Rehearse high-stakes performances by running them live many times in low-stakes venues

The hosts endorse preparing a high-stakes short performance by performing it repeatedly in low-stakes settings, citing Jerry Seinfeld's approach of running his five-minute Tonight Show set roughly 200 times in clubs over months, like an Olympic athlete training.

The ~200 repetitions figure is flagged as a possible numeric error.

Acquired · Seinfeld

How TV syndication works: definition, the 100-episode threshold, and who gets paid

Syndication is when a show gets licensed to any distribution option besides the primary first-run network — international, off-network, cable, and eventually streaming. The 100-episode threshold exists because buyers want a large block of episodes to demonstrate people liked the show; they don't want 10 episodes. When NBC buys a show it buys directly from the studio (Castle Rock), but once a show enters syndication the reruns have nothing to do with NBC — they go through local affiliates. Local affiliates are four-letter stations starting with K or W, broadcast over the air, and may be NBC, ABC, CBS, or Fox stations; they only take network programming during primetime or sporting events and must make or buy their own programming the rest of the time.

Acquired · Seinfeld

Brains-vs-hands: keep credentials and access separate from the agent

In the brains-versus-hands architecture, credentials and access are kept separate from the agent itself — there is no broad standing access; AOP allows agents to run without API keys.

AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth

AAuth: mission and policy as primitives of intent-based auth

In AAuth, the concept of a mission is being developed: what the agent should accomplish big picture, with each decision made within the context and intent of that big-picture mission. Policy is another primitive of intent-based auth. A mission identifier should track through everything so you can audit all the things that happened and the decisions made in the context of one mission.

The mission concept is described as being developed, not shipped.

AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth

Binary authorization decisions don't work for the future; context must be paired with intent

Davis describes that most authorization decisions in organizations today are still a single binary decision, and that doesn't work in the future. He recommends pairing the intents of what a user is trying to do with the ecosystem the user is doing it in — where they're doing it from, time of day, and more advanced context — and bringing more context into the authorization environment through different teams but more importantly through sensors. He gives the example of deciding whether to let an administrator use Firefox to access a server right now when the server has high-risk vulnerabilities and their system hasn't been patched.

AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth

Design golden dataset tasks a human could pass, with known-good answers

For a good golden dataset, Laurie Voss advises writing tasks a human could pass — if your agent fails a task every single time, that is almost always a broken task, not a broken agent. For each task, you should have an answer that you know is good, which proves the task can be done. He also advises testing in both directions: if you only check that the agent searches the web when it's supposed to, you'll accidentally train it to search the web every single time, even when it doesn't need to.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Agent improvement order of operations: data quality, then prompts, then model choice, then hyperparameters

Laurie Voss lays out an ordered sequence for improving an agent based on how useful each change will be: fix data quality first, then prompts, then model choice, and finally hyperparameters. He explains that if the underlying data is bad, nothing else you tweak will be useful; once data is good, move to prompts; model choice has a lot of leverage but changes many things at once and makes things more expensive; and temperature tuning is the least useful lever since modern cloud models often don't even allow it.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Use Phoenix, Arize's open-source AI observability platform, to collect agent traces regardless of framework

Voss introduces Phoenix, Arize's open-source AI observability platform, as the tool used to collect traces from your AI agent regardless of framework. You install and run Phoenix on your own laptop, then open your browser to localhost:6006 to see the UI. Phoenix also has Python SDKs and examples, though the tutorial repo uses TypeScript.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Write the eval before the feature — eval-driven development gives 'done' a measurable definition

Voss says more and more teams that ship AI quickly write the eval before the feature — that's called eval-driven development. For example, if you want your support agent to always check who the customer is before giving a refund, you can write the eval that checks for that first, and now 'done' has a definition you can measure from the start.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI