Explore

With semi-autonomous agents, users must be able to predict what will happen — privacy and norms understanding matter more than ever

Users understanding privacy, their data, and how it's used is way more important than it always has been, as is users understanding the norms of your product. Before, you could get away with unpredictable behavior because there was a settings menu or a pop-up that could appear. When the agent is doing things semi-autonomously, you really need to make sure that at any given point a user can predict what is probably going to happen with a feature or primitive you're introducing.

endorsed

Lenny's Podcast · Stop planning for 2027: how OpenAI builds product 90 days at a time | Tara Sesha and Nan Yu (OpenAI)

Inject human taste and judgment into AI outreach, and never auto-send

The speaker argues that AI content all sounds the same and will feel like spam without human judgment, so the voice, taste, and judgment have to come from him. He also explains that automatically sending AI-generated emails leads to broken drafts reaching recipients — which is why the pipeline does not automatically send emails, and drafts are reviewed manually before sending.

Ahrefs · I Used AI Agents to Replace Our Marketing Agency

Human-in-the-loop plan approval as the core agent design pattern

Design AI agent products so the human reviews and approves the agent's plan before execution: the user states intent, the agent returns a plan, the user edits it, and only then does the agent execute. This keeps the vendor responsible for outcomes while making the right choices obvious to the user.

The Official SaaStr Podcast: SaaS | Founders | Investors · SaaStr 876: Shipping Enterprise AI Agents with the CPOs of Rubrik, Glean, and Harvey

Be specific and verifiable about done-criteria when using GOAL

Charlie Guo's advice for GOAL: be really specific with the criteria you give the model for knowing when it's done, and make that completion criteria as verifiable as possible. When given a GOAL, the agent keeps evaluating itself against the success criteria, checking at each turn whether the task actually completed and whether to keep going. He compares GOAL to a genie: when it works well it is insanely magical, but when it doesn't it can be a monkey's paw situation where it technically did what you asked but not in the way you expected. GOAL suits very large projects — code migrations, large refactors, retry loops, experiments, games, and one-shotting apps with detailed specs — and he cites Thomas's slash GOAL that ran 40 hours reimplementing Doom in native Swift.

endorsed

AI Engineer · Cooking with Codex — Charlie Guo & Gabriel Chua, OpenAI

Read your traces before you write evals — the most important practice

Laurie Voss calls reading your traces before you write evals the most important practice in the entire workshop: before writing a single evaluator, start with your data and read a dozen or more traces end-to-end. When reading traces, focus on the ones where something went wrong: what was the input, what was the output, what specifically is broken. Reading raw inputs and outputs sounds old-fashioned but is one of the highest value activities in agent development. A trace tells you whether the agent made a genuine mistake or whether your graders rejected a valid solution. Anthropic invested in tooling specifically for viewing eval transcripts, and their team spends a lot of time just looking at eval transcripts every day.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Grade outputs, not paths — path grading punishes creativity

The speaker advised grading whether the output had the right information and whether the final state matched what you wanted, regardless of how the agent found the data; checking a specific sequence of tool calls is too rigid in practice, and grading the path punishes creativity because agents find valid approaches the eval designer didn't anticipate.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Write the eval first, then build until it passes

Writing the eval first gives a clear, measurable definition of done; then build the feature until the eval passes. Some teams write evals at the very start, others add them at scale — either way treat evals as core.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Regression evals verify behavior after system changes

When an agent's system changes (model, prompt, tools) and behavior might drift, the solution is evals — specifically regression evals already in place before the change. Running them confirms the system doesn't change from previously good behavior, so you rely on evals rather than manual inspection.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI

Feed eval failure explanations into a coding agent to close the software development loop

Eval failure explanations can be fed back into a coding agent, which can then make the software better automatically and close the software development loop without manual intervention. Once you have defined what good looks like, your agent can climb the hill by itself.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI

Validate before building: research, conversations, and cheap tests beat building blind

Rob Walling argues that founders should validate an idea before committing to build, launch, and market it. Validation means doing research, putting thought in, having conversations, or testing the assumption that what you want to build is valuable. He frames this as a way to avoid spending months or years building something nobody wants, and notes that serial entrepreneurs do more validation before selecting an idea than first-time founders who flail. He also cautions that validation can never reach 100% certainty — you only get that by building something and selling it.

Startups For the Rest of Us · Episode 848 | What Liquid Death Can Teach Us About B2B SaaS (Rob Solo)

Reliability as the Core Value Proposition

Reliability is the primary value driver for AI tools. Each additional 'nine' of reliability (e.g., 99% to 99.9%) unlocks new applications. Developers should prioritize background reliability over flashy demos.

The a16z Show · AI Can Write Code. Why Isn’t Software Better?

The Power of 'No Deal' Calm

A calm, unemotional energy that holds the line on agreements builds respect and makes deals work better, unlike a 'no problem' calm that lets things slide and leads to being walked over.

Charisma on Command · This Scene Will Change How You Handle Conflict

How to perform the incline drag curl

Set the incline bench to 60 degrees, use light weights, start with elbows back behind the body, curl by bending only at the elbow while twisting the dumbbell into supination, allow the elbows to come forward slightly with an extra twist at the very end, then come down slow and re-establish elbow positioning, contracting the triceps at the bottom for a deeper biceps stretch.

Method, not a verified protocol.

ATHLEAN-X™ · The ONLY 2 DB Biceps Exercises You Need (NO, SERIOUSLY!)

How to perform the preacher curl on the 60-degree bench

Perform the preacher curl on the same 60-degree bench angle, press the entire back of the triceps against the pad, push back into the pad to engage the lats and support the shoulder girdle, keep tension in the bottom position, and twist the wrist into supination as much as possible at the top.

hedged

Method, not a verified protocol.

ATHLEAN-X™ · The ONLY 2 DB Biceps Exercises You Need (NO, SERIOUSLY!)

Save money on AI spend with caching, no spending limits, and smart routing

Tereza recommends three levers for saving money on AI: use caching to stop prefilling context every time, set no limits on spending but require people to show results when they spend a lot, and use smart routing between LLMs to optimize tokens instead of spending so many tokens.

AI Engineer · Rise of the Software Factory — Tereza Tížková, Factory

Onboard early users manually and fix the worst pain point before the next session

Roman Ugarte describes manually onboarding 200-300 early users, sitting on a 20-minute call watching for confusion or technical issues, and immediately fixing the worst pain point so the next onboarding goes better.

Lenny's Podcast · How we built Grok Bot in a month | Roman Ugarte (SpaceXAI)

Build new products from scratch with a small, isolated team

Roman Ugarte recommends starting completely from scratch rather than adding features to an existing product, and creating a small team that goes off into a cave for about a month with a single objective, completely isolated from the rest of the company.

hedged

Lenny's Podcast · How we built Grok Bot in a month | Roman Ugarte (SpaceXAI)

Onboard early users manually and fix pain points within 24 hours

Roman Ugarte describes a two-week period of intensive manual onboarding where the team onboarded 200-300 early users via calls. Whenever they observed an onboarding pain point, they fixed it immediately - within 24 hours - because the next person would be onboarded the following day.

Lenny's Podcast · How we built Grok Bot in a month | Roman Ugarte (SpaceXAI)

Give the agent hardware context, since low-level GPU features change generation to generation

Tejas Bhakta says you really need to make sure your agent is aware of the hardware — e.g., on a B200, the warps, Tmem, TMA, and so on. These low-level operators change generation to generation (H200 won't have Tmem, which is new with B200), which is why this context is needed. In practice this is a bunch of MD files you give the agent so it has context.

AI Engineer · Autoresearch Made Our Models 3x Faster — Tejas Bhakta, Morph

As the human, review profiler output and flag dumb high-level approaches for the agent to fix

Tejas Bhakta advises that the human's job when reading a profiler is to look at the top and identify dumb approaches — for example, recognizing that loading 32K chunks into context is unnecessary for DeepSeek attention and should only be done every 32K instead. At a high level, all you have to tell autoresearch is that the top method is dumb and to pipeline it instead; everything else like sizing, chunk sizing, and context chunks should be decided by autoresearch.

AI Engineer · Autoresearch Made Our Models 3x Faster — Tejas Bhakta, Morph