Method · MoneyJesse Hall lays out a benchmarking method: take a complete working agent with a real backend, put simulated callers against it (callers that pursue goals, change their minds, and introduce background noise), define each scenario as what the caller wants, what actions the agent should take, and what outcomes should be, then score those outcomes; swap the model stack while keeping the same agent and scenarios so every model sees identical input, and run the loop again.
Method, not a verified protocol.
AI Engineer · How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
Method · MoneyTo pick models, test the conversation: take a complete working agent with a real backend and put simulated callers against it. Simulated callers can pursue goals, change their mind, and have background noise added to make the scenario real-world. Run text-based simulations to test business logic, and audio-based simulations to test speech recognition, turn-taking, and the entire voice pipeline. Define each scenario as: what the caller wants, what actions the agent should take, and what the outcomes should be. Score agents on outcomes — in a hotel benchmark, did the correct reservation hit the database; in a financial benchmark, did the correct authorized amount get charged. Don't score on completion alone, because agents can game completions — verify they took the right steps: check that a
Method, not a verified protocol.
AI Engineer · How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
Method · MoneyWhen the happy path doesn't happen and a prospect gives a lukewarm feel, Rob Snyder recommends throwing the 'Hail Mary': asking 'Is there anything on your to-do list that you would pay me to do right now — that you would pay somebody to take off your plate?' He reports that quite a few startups have totally pivoted and taken off from this question.
The pivot outcomes are described anecdotally.
Method, not a verified protocol.
Lenny's Podcast · How to find product-market fit (but, actually) | Rob Snyder (Author, The Power of Pull)
Method · MoneyRob Snyder describes a sales call as mostly two steps: pull discovery, then the conceptual pitch of how you might help. Approach it leaned back and excited to chat — 'let's see if I might be able to help you' — not wired up to convince; if they have pull, they'll convince you to let them buy. Open by asking why they took the call, which ideally lets them fill out the entire pull framework. If they say 'I don't know, you reached out to me,' use what they just told you and propose it as the agenda, getting their thumbs up. Then earn the right to ask discovery questions by asking permission — this prevents the call from feeling like an interrogation, because if it does, they're not telling you anything. Make it a 30-minute calendar block (smaller blocks don't get seen), and be iffy about a on
None stated.
Method, not a verified protocol.
Lenny's Podcast · How to find product-market fit (but, actually) | Rob Snyder (Author, The Power of Pull)
Method · MoneyRob Snyder walks through a call script: ask why the prospect took the call, ask what they're trying to do but have been unable to do with existing tools, ask whether the problem must be solved now, and ask what other options they've looked into and whether those are good enough.
presented as the guest's own practiced script
Method, not a verified protocol.
Lenny's Podcast · How to find product-market fit (but, actually) | Rob Snyder (Author, The Power of Pull)
Method · MoneyBuild a five-step regression loop: (1) select a dataset by pulling raw real production traces; (2) add a scoring mechanism by bringing in evaluators; (3) run a baseline experiment by running the agent against all production traces and datasets, then running the eval testing suite on the output — in the SDK via run_experiment, passing in the dataset, task, and evaluation suite; (4) add the one change, rerun the evaluation, and compare eval results; (5) automate the regression process by pushing the experiment into CI (e.g., GitHub Actions) so each pull request triggers it. Use observability as a way to build a robust regression process that feeds automatically into CI/CD.
Method, not a verified protocol.
AI Engineer · Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI
Method · MoneyTo bring in a new hardware backend: characterize the hardware, then specify what each of its operations means by equating them to operations the compiler already understands via rewrite rules. New rewrite rules propagate out to the rest of the equalities already specified in the compiler, making it a fairly low-effort process even for very different backends.
Method, not a verified protocol.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Method · MoneyHarneet Kaur created a benchmark that assesses each layer of the AI reasoning process to improve assessment of the models' process. On the extraction layer, LLMs can mess up, though with public companies it's pretty solved; extraction with private loans, Excel documents, and PDFs can still be a challenge. The reconciliation or derivation layer comes after extraction — having EBITDA and debt, you can derive leverage. The judgment layer asks whether, given the leverage, this is a distressed credit or not. To get to the prediction layer, you must first assure the data points are good, the reconciliation layer is good, and there is some sort of reasonable judgment. When analyzing a distressed credit, you should look at whether you agree or disagree with the different assumptions at each layer.
Method, not a verified protocol.
AI Engineer · Can AI Understand a Distressed Credit? - Harneet Kaur, Axim Intelligence
Method · MoneyOnce you know who your audience is, figure out what they like. Associate your brand with what your audience likes by making content about the thing they like, making products they like (free and paid), and appearing next to the people they like.
Method, not a verified protocol.
Alex Hormozi · How I Would Build a Personal Brand in 2027 (If I Had to Start Over)
Method · MoneyStart AI evaluation with a clear requirement, such as 'advice must match the approved policy'. Challenge the AI chatbot with edge-case questions (e.g., about reimbursement after travel) to expose incorrect advice before deployment. If eval tests expose incorrect AI advice, add a policy check or human escalation as a control. After adding controls, test again to verify the controls catch errors before they reach the customer.
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Method · MoneyStart by looking at your system claim and testing it to try to break it, then identify where the system claims break. Map out the uncertain region and high-risk region where the AI system's performance cannot be trusted, creating a full failure region map. Then either restrict the AI system from operating in failure territories or add supporting systems you are confident of, and iterate until reaching the desired confidence level. Apply this build-break-fix-break-again approach before deployment to expose failures that narrow evaluations miss in the real world.
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Method · MoneyA first-principles approach for any evaluation system: start from your system claim and test it, trying to break it; identify where the system claims are breaking; when failures are found, either don't let the AI system run in the territory where there are failures, or come up with supporting systems you are confident of; keep iterating the build-break-fix loop until you reach the level of confidence you need. In practice: define the claim, actively challenge it, and use the evidence to guide changes and make a deployment decision.
endorsed method from the speakers
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Method · MoneyUse a structured framework with 13 dimensions and three core questions — technical strength, deployment controls, and business context — to investigate AI failures. Technical strength covers data quality, task design fit, predictive validity, fairness checks, failure modes, and explainability. Deployment covers decision authority, human mediation, exposure control, and monitoring. Business impact covers financial impact, legal consequences, and scale of users affected.
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Method · MoneyBreak your entire AI system into core components: technical failure, deployment controls, and business context. Assess your AI use case's materiality using a scale, which could be company-specific, and recognize that risk can change over time. Once you have the risk assessment, apply system-level stress testing on the AI use case. Only make the deployment decision once you are confident about the region; otherwise it may be too expensive.
Method, not a verified protocol.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
MethodTom Foley outlines an ordered method: first trace a typeface's calligraphic root (broad nib, pointed nib, or expansion), then compare structural proportions like x-height, ascenders, and cap height, then match stem weights and weight range/color/texture to narrow down pairings quickly.
Method, not a verified protocol.
Design Better · Tom Foley: How to pair typefaces and why the best brand fonts go unnoticed
Method · BodyThe coach uses a four-day training split where every day develops a different piece of the explosive power puzzle: day one focuses on increasing max lower body strength, day two on speed strength (possibly with upper body work), day three is 'athlete day' with lots of jumps and plyometric work, and day four is the impulse session focused on putting out a large amount of force in a short period of time.
Method, not a verified protocol.
Garage Strength · I Trained 1000 Elite Athletes. This Is How You Jump Higher.
Method · MoneyInstead of only grading agent answers, plant known problems or answers into inputs and test whether the agent finds them back. Mix modified contracts (with removed clauses) with existing unmodified contracts that should show no flags, so the agent must distinguish real defects from normal contracts — the team found their agents were missing most removed clauses.
Method, not a verified protocol.
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Method · MoneyWhen pairs of original and modified contracts cannot be obtained, pivot to generating test cases from known rules: take a rule (e.g., a $1 million cap), create a clause that breaks it, generate the contract around it, and verify the agent flags the violation. Source rules from experts and their documentation, supplement with your own rules, and design generated tuples so some break rules and some do not — full rule coverage cannot be guaranteed.
requires knowing the rules; coverage of all rules cannot be guaranteed
Method, not a verified protocol.
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Method · MoneyReuse the templates experts already use, which contain blanks; generate an array of values for each blank to produce plenty of test contracts. The speaker identifies this template approach as the most powerful technique, while noting it is very hard.
speaker notes the technique is very hard
Method, not a verified protocol.
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Method · MoneyRun a weekly skill that reviews your video creation process, shows where you spend the most time, and suggests repetitive tasks to automate. Use it to identify which skills need updating because they're not being effective, consolidate your task triage into one place (it identified reminders being sent to a Slack thread that was never checked), and remove unnecessary recurring checks (it identified a Premiere import check that took a lot of time and wasn't needed).
Method, not a verified protocol.
AI Engineer · The AI Content System Behind 200M Views — Sarah Chieng, Cerebras & Cova Lee, OpenAI
