Explore

How many eval examples you need: 10-20 in dev, hundreds for production

Laurie Voss advises that 10 to 20 examples give a rough direction of where things are going and are very useful in development, but for production you want hundreds of examples — it takes a long time but is very worth it for accuracy. To halve the error bars on your accuracy you need to quadruple the number of examples. Going from 25 to 100 examples is really helpful and not that much effort, but going to 400 requires a big jump in effort and gets into territory you're never going to bother with. The bigger the better, and the more production failures you throw into the dataset, the better.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Write the calculation once and run it every time; keep arithmetic out of the model's head

Write the calculation once and then run it every time you need it. The model can pick tools, but your harness must make it very strict that it will not do any arithmetic or try to solve these problems by parsing tokens. If calculations aren't in code, the arithmetic is in the model's head and gone once you start a new session, and new code gets generated for every new question or session until it becomes a mess. Asking the same question and getting different answers is non-determinism — you're asking a stochastic machine to produce a consistent output, so you're likely not going to get it.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Establish a shared domain with the model so you talk in the same terms

Francesco Galletta recommends sharing a domain with the model and agreeing on a plan so you and the model talk in the same terms, noting that your definitions likely differ from what the model knows because its training differs from your context, and that going back and forth with the model really makes a difference in getting a shared understanding.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

A planning harness should refuse vague input and force every needed data point

Francesco Galletta explains the harness should not go into yes-man mode and implement an unclear idea; instead it is guardrailed to force the user through every single data point needed to come up with a plan, helping them drill down into what actually needs to be done and challenging assumptions they may not have known they had.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Pitching as a whip: arm care through mobility and assessment

For a baseball pitcher, the arm is a whip — many big leaguers are super skinny, with force coming through the hand and shoulder. From an arm care point of view, a key priority is good mobility: assess T-spine mobility, the ability to separate (dissociate) shoulders and hips, and shoulder external/internal rotation control. The elbow will be taxed in some way regardless.

her coaching framework

Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs

Hitter development: strength foundation, fast contraction, then transfer

Grubbs outlines a pyramid for hitters: first assess a good foundation of strength through the lower body, posterior shoulder, or lat to stabilize, plus the ability to contract and relax. Develop rate of force development through plyos and jumps on top of that foundation. Once strength and fast contraction are established, the hitter can contract fast to do their job, and you can get creative with transfer of training — the 'sexy stuff' like rotational med ball throws.

her coaching model

Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs

Split API keys by department and use case to enable measurement

Eugene Cheah advises starting to split up API keys by departments and by major use cases so you know where all your token spend is, noting that splitting keys this way informs your workload decisions from then on. He stresses that if you are not measuring your token spend, you should ignore everything else in the talk and start with measuring first, because the biggest ROI is just from measuring, and that token usage should be measured through separate token keys. For those not yet tracking token spend and token-view split, he points to Featherless's open-sourced tokenomics tracking tool, which he describes as a fancy way of splitting up API keys and tracking which API key spends how much. Attendees can scan a QR code for updates on the tool and fill a form to be notified about the talk's.

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Store skills by meaning via embeddings, not plain text

The most important thing when storing skills is capturing the meaning of the sentences and what the skill means itself, rather than having it as text. Storing a skill as plain text in a file on the file system limits agent interaction to keyword or regular-expression word search; regular expression matching (like grep) is inefficient for huge files, and if the file system grows linearly, the performance of regular expression matching decreases quadratically, so it doesn't scale with file size. Pre-computing embeddings of a skill captures hidden meaning between tokens that text descriptions alone do not; embeddings are typically 384 minimum to a thousand-plus dimensions and make skills much more discoverable by semantic search.

speaker's argument; dimension figures are approximate

AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle

Single agents struggle researching long ticker lists; flows split work across isolated scan agents

A single agent struggles when it tries to research each ticker on a list by itself. With flows, split up the list of tickers and pass each ticker to a scan agent, giving each scan agent an isolated context where all it cares about is its one ticker, so it isn't influenced by other tickers or search results. After the scan phase completes, pass results to the next phase (review), which is its own isolated context.

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Tiered evaluation benchmarks reveal true generalization

Design ML evaluation benchmarks with multiple difficulty tiers that vary chemical similarity — from shared building blocks, to shared hub only, to completely different libraries — to reveal true generalization. In the Kaggle competition, the winner performed okay on shared-building-block splits, saw some boosts on shared-hub splits, but no one beat chance on the completely different library split. Models that win on chemically similar test splits may fail completely on distribution shift — three of the top five entries were the noisy tutorial notebook.

Latent.Space · Lab-in-the-Loop at Compute Speed — Ian Quigley, Leash Bio

One trifecta capability is usually fine; two is risky; all three is never acceptable

Having any one of the lethal trifecta capabilities is usually fine, because realistically your AI tool won't be very useful if it can't have access to any of these things — though even one can lead to disaster if you're not careful and the rest of your infrastructure has a lot of issues. Having two together is risky and not something Jiang would recommend, especially in a regulated industry like banking or financial services. Giving agents access to all three at the same time is never acceptable — not for a prototype, not just for one sprint, not for just one repository — because that combination of vulnerability is an open door to compromise your whole organization.

AI Engineer · Your AI Isn’t the Problem. Your Infrastructure Is. — Jiachen Jiang, Coder

Zero-trust approval: agents cannot approve their own changes; approval must come from someone else

Agents cannot approve their own changes, and the engineer who prompted the agent is also not the approver. Ensure the approval comes from someone apart from the bot that was prompted by the same person. The control is making sure the change was approved and tested by someone who did not write it.

AI Engineer · Rebuilding CI for AI-Scale Commit Volume — Gautam Korlam, Sonar

Only allow commits onto the trunk if they pass CI in combination with all incoming commits

Only allow a commit onto the trunk if it passes CI in combination with all other incoming commits — this keeps the trunk green.

AI Engineer · Rebuilding CI for AI-Scale Commit Volume — Gautam Korlam, Sonar

For hypertrophy, at least one of frequency, intensity, or volume must be high

Dr. Andy Galpin states that to succeed at hypertrophy you only need to make sure the training is hard enough, and that as long as one of the three variables — frequency, intensity, or volume — is high, you will achieve hypertrophy. The only wrong combination is infrequent training combined with low intensity and low volume. This makes crafting a hypertrophy program fairly simple: you just have to make sure you do the work.

Andrew Huberman · Essentials: Optimal Protocols to Build Strength & Grow Muscles | Dr. Andy Galpin

Chaos management: adapt disrupted workouts to your goal

Dr. Andy Galpin explains that your goal (hypertrophy vs strength) determines how you manage disrupted workouts — what he calls chaos management — and that 10% to 50% of your workouts will be chaos management situations (short time, hotel, not feeling it). For hypertrophy when short on time: lower the load, lower the rest intervals, and just get to the burn and get going as much as you can to hit total volume. For strength when short on time: cut your volume in half, get the few repetitions done at high load, and just don't do very many sets that day. He gives the example of a plan that normally takes 60 minutes when you only have 35 minutes available.

Andrew Huberman · Essentials: Optimal Protocols to Build Strength & Grow Muscles | Dr. Andy Galpin

Volume targets: 10 sets minimum, 15–20 realistic, ~25 for the well-trained

Dr. Andy Galpin lays out weekly working-set volume targets per muscle group: the minimum is 10 working sets per week; a more realistic number for most people, especially intermediate or advanced, is 15 to 20; very well-trained individuals should push toward around 25. He notes that past 25 working sets per week there is not a lot of research. He illustrates how easy the minimum is: three sets of 10 done three days a week gets you to nine working sets — almost 10 — and going to the gym one day a week doing three sets of 10 across three exercises also yields nine. Hitting 10 sets per week per muscle group is fairly easy (not even counting indirect activation of secondary muscles), and hitting 20 is actually still not that hard.

Andrew Huberman · Essentials: Optimal Protocols to Build Strength & Grow Muscles | Dr. Andy Galpin

If sets aren't hard enough, rep range doesn't matter — get close to failure

Dr. Andy Galpin explains that if you're not seeing adaptations despite adequate volume, the repetitions in your sets may not be hard enough. A set of 25 that only starts getting hard at the end is not enough stimulus — and neither is a set of five or six that feels the same way. When sets aren't challenging enough, the rep range itself doesn't matter; the effort within the set is the issue. Rep range doesn't matter if you're not getting somewhat close to failure, though sets don't need to go to complete failure.

Andrew Huberman · Essentials: Optimal Protocols to Build Strength & Grow Muscles | Dr. Andy Galpin

Match kickoff tone to team performance — never fake rah-rah when reps are missing quota

When the team isn't winning, Katie Givens recommends a 'true grit' perseverance theme that acknowledges it's harder to be excited, focusing on pushing barriers out of the way rather than fake enthusiasm. She says never plan a massive celebration kickoff when reps are missing quota because reps feel the fake theme — their pay is tied to winning. In a losing locker room, run a session where reps feel heard about what's getting in their way, with an honest conversation about why. In a down economy, use a chip-on-your-shoulder 'we're gonna get this done' tone instead of a tone-deaf celebration, while still celebrating individual successes.

Presented as the guest's design philosophy from experience.

30 Minutes to President's Club | No-Nonsense Sales · #612: Sales Kickoff Masterclass: The Ultimate Planning Guide

Fold the year's first QBR into kickoff so reps leave with concrete plans

Fold the rep's first QBR of the year into kickoff itself, while the theme is fresh and reps are fired up and undistracted, so they launch into the year immediately. Teams that don't do QBRs at kickoff suffer a post-kickoff delay because reps return without proper account plans and spend time re-orienting instead of selling. Structure a QBR as: retrospective on last quarter (numbers, misses, wins, three biggest lessons learned or a start/stop/continue), then forward plan — this quarter's number, the gap, and three to five very specific goals. Make goals hyper-specific with the format 'I will make X calls by X date to X customer' — never vague goals like 'I'm just going to make some more calls.' In QBRs, have reps deliver in-depth account planning on their top three to five deals to peers —

Presented as recommended design; the delay claim is the speaker's observation.

30 Minutes to President's Club | No-Nonsense Sales · #612: Sales Kickoff Masterclass: The Ultimate Planning Guide

Limit teaching to one focused concept per day, delivered interactively by multiple voices

Limit the kickoff teaching agenda to one heavily-focused concept per day — reps won't remember material if you try to teach everything at kickoff. Put more than one person on stage during teaching moments so attendees hear different points of view and avoid enablement-person fatigue. Talking at attendees for hours is a fatal flaw: even with a room of 1,100 reps, build in interactive elements. Roughly four hours of the general session day go to the single new concept, broken into multiple sessions with breaks and lunch in between.

The four-hour allocation is the guest's described practice.

30 Minutes to President's Club | No-Nonsense Sales · #612: Sales Kickoff Masterclass: The Ultimate Planning Guide