Explore

Non-determinism: run evals multiple times and understand pass@k vs pass^k

Because agents are non-deterministic, each run of each example can mislead you in either direction — Laurie ran his experiment three times for this reason. As k goes up, pass@k and pass^k diverge: pass@k goes up (succeeds at least once) and pass^k goes down (fails at least once). A coding assistant that gets it right on the second try is fine, so pass@k is the right metric for it; but a support bot that is wrong one run in every five is a huge problem, and a financial analyst agent giving inaccurate recommendations must pass every single time (pass^k).

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Ask why a failure happened; classify the cause

Laurie Voss argues 'was wrong' is a symptom, not a cause: bad search results are a retrieval failure, right data with wrong conclusion is a reasoning error, made-up numbers are hallucination, and answering the wrong question is a scope problem. Each cause has a different fix, which is why reading traces is useful.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

You can't switch models without evals — evals make model switching safe

Voss states that you cannot switch models without evals; evals are what make model switching safe. To climb the hill of a failing eval, you can upgrade the model (e.g., pass a new model like Haiku 5.5) and run the experiment again. But a model upgrade that improves consistency by just picking the same answer every time (e.g., always saying hold) is probably worse and needs iteration — a model can get more consistent just by getting more conservative. And when an upgrade improves consistency but costs four times as much, whether it's worth it is a question you need to make for yourself in production.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Don't tune on your test set: split labels into tuning and test groups

Applying a rule from machine learning to judges, Laurie Voss advises not tuning on your test set. He recommends splitting labels into two groups — one to tune evals and one to test that they are working. With only 16 answers in the golden dataset, the test set is very small — the rule is usually 75% tuning set, 25% test set, so only four in the test set; if you were up to 100 answers, the 75/25 split would be fine. Phoenix datasets have splits built in, so splitting labels is very easy to do.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

When to build a harness: convoluted copy-paste workflows and repeated manual steps

If you're copy-pasting text between chats, opening new sessions, and the workflow feels convoluted, build a harness; repeated manual steps done three to five times are the signal to design a harness around them.

endorsed advice

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Ensure numbers come from tools and scripts, not the language model doing arithmetic

Francesco Galletta recommends ensuring numbers come from a correct place via tool calling and scripts rather than the language model processing arithmetic itself.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Skills try to cover edge cases with prompt instructions; back-end code covers them deterministically

Teams frequently write skill instructions trying to cover unsupported cases ('if the user tries to do this, tell the user back that this skill does not support that'); determinism via back-end code like Python in a harness covers out-of-the-box for cases that skills try to cover with prompt instructions. The prompt-based approach is 'please follow all of this and make no mistakes, please, please, please' — and 'please don't make any mistakes' doesn't work quite well.

described and endorsed

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Five trust principles build on each other: shared domain, consistency, evidence, accountability, verification

The presenters present a sequence of design principles that build on top of each other in a logical order — shared domain, consistency, evidence, accountability, verification — meant to funnel effort into where it is most useful to get something you can actually trust.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Five principles for building harnesses you can trust with financial data

The framework for building harnesses you can trust with financial data and money is built on five principles: having a shared domain, having consistency, showing the receipts, keeping yourself accountable, and making sure you create intentional loops for verification outside of what you think you know.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Year-round baseball and hard throwing from seventh grade is an injury risk pattern

Grubbs discourages the pattern of playing baseball year-round from seventh grade while throwing hard all year, linking it to overuse injuries in young athletes.

Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs

Everything in exercise selection is risk-reward, judged against the athlete's context

Grubbs frames exercise selection as risk-reward: consider whether a movement (e.g., barbell bench, cleans) is more provocative on the wrist, shoulder, or rotator cuff, marry that to the athlete's context, and factor in how much time you have to teach movements (e.g., a three-week intro block before season start).

Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs

Independence is key: NTSB, range safety officers, and AI safety

The key thing for aviation safety is independence — the NTSB was set up as an independent agency in 1967 and is used globally (if an accident involves a US-manufactured component, they'll go to other countries and investigate). In the space industry, a range safety officer is completely independent of the mission and makes a go-no-go call based on objective safety metrics. Independence is really important for AI safety evaluation — it would be a shame if all safety were left to the labs or friends of the labs.

Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club

Agentic vs process automation: two tracks with opposite economics

Eugene Cheah explains that AI workloads split into two major tracks with very opposite economics: agentic workloads and process automation. Agentic workloads involve long-running, long-context requests that are extremely cache-heavy and input-heavy, with only a small percentage of output — in agentic deployments only 1% of the request is output tokens — because they repeat multiple turns in a rinse-and-repeat pattern, so even smaller input and cache costs get repeated heavily and decide your token cost. He advises separating and tracking the agentic and process tracks of token economy respectively. He notes CI/CD is a very common current use case of AI running in the background, and that you can control token speed and cost per task priority at a key level, such as setting up faster tokens

figures described with moderate confidence

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Every 2x of speed costs 3x, and it compounds

A rule of thumb the speaker sometimes says: for every 2x of speed, it's 3x of cost, and it compounds.

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Planner/worker/reviewer split: cheap models for token-heavy implementation

Eugene Cheah describes a common pattern in organizations scaling AI agent deployments: having the best agent do planning, a faster cheaper agent do implementation, and the best agent do review again. This gives the best of both worlds because the implementation phase is very token-heavy, involving repeated processes like modifying the file, testing, checking results, and rinse-and-repeating. He claims faster cheaper models like JRM 5.3 Flash can do implementation tasks at the same quality level as stronger agents like Astra or Opus, assuming those agents do the planning. Some organizations Featherless worked with use the best frontier closed-source model for planning and review, and GRM 5.3 Flash or DeepSeek v4.1 Flash as the implementation workers.

described pattern; model names flagged possible_numeric_error

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Agentic workloads are long-running, context-heavy; process automation is a volume game

Agentic workloads tend to be long-running multi-turn agents — coding agents are a particular example — that maximize context usage. Process automation workloads (call centers, email routing, meeting-notes transcription) are much shorter in context, very process-driven, task-isolated, and essentially a volume game: think millions per day, or even tens of millions per day, of decisions in call centers, lines of code scanned, emails and text messages filtered. Maker-checker processes, approval processes, and document processing are all volume-game AI workloads for multiple organizations. Financial institutions have high-volume OCR, document, approval, and transaction-risk workloads suited to process automation.

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Dynamic workflows: the agent writes the workflow and a server runs it in the background

For a task with too many pieces for one agent — such as scanning 300 contracts to find change-of-control clauses — dynamic workflows has the agent write the workflow and a server runs it in the background; when the workflow runs, results come back from all the parallel work and are combined into the final output.

described

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Benchmark AI tools fairly: same prompt word-for-word, fresh out-of-the-box builds, and exclude heavily-iterated internal tools from head-to-heads

Methodology guidance from the bake-off: use the exact same prompt word-for-word in each instance for a fair comparison; run a fresh out-of-the-box bake-off on each platform rather than benchmarking through your existing customized setup, to see what a new user would actually get; and when comparing a heavily-iterated internal tool against fresh builds, exclude it from the head-to-head and show it separately — it's not a fair comparison. A mature internal agent build like 10K (connected to 20 apps for a year, representing hundreds of hours of iteration) shouldn't be treated as comparable to a fresh out-of-the-box platform build. When benchmarking agent platforms, also compare them against your actual current workflow — the speaker realized the real benchmark was their year-old 10K agent, a

None; presented as methodology the speaker actually followed.

The Official SaaStr Podcast: SaaS | Founders | Investors · SaaStr 882: Claudeforce vs. Replit vs. Lovable. Who wins? (The Agents #016)

Push design-layer edits back to the code layer to keep them in sync

Failing to push design-layer edits back to the code layer leaves code and design out of sync, so edits made in the design layer should be pushed back to keep both layers aligned.

Figma · Explore ideas in code | Figma

Start as a low-pressure side project with a small, specific investment

Dad Gang began with $250 from each of the three founders ($750 total) while they still had full-time jobs. They avoided grand plans and long to-do lists, treated it as a fun side project, and deliberately did not put stress or pressure on themselves to grow quickly, letting it expand organically.

Foundr · We Started Dad Gang with $750 and One Hat - and did $40M in 4 Years