Find big external goals that scare you and have real stakes as belt moments
Find big moments and big external goals that you can drive towards — things that scare you and where you actually have real stakes — as belt moments.
Explore
34,858 results · Most relevant
Worth exploring
WHY · 34135 cleared results from 496 sources on record
Identify your acceptable risk level before deploying an agent system
Before deploying an agent system, Jarmak recommends explicitly identifying what level of risk you are willing to accept.
Optimize for observability and realistic deployment environments to trust measured values
Jarmak recommends optimizing for as much observability and as realistic environments as possible where the system will be deployed, so the values you measure can actually be trusted.
Start from zero by making internet content about what you know
Gary Vaynerchuk says that if he were starting today with no audience, money, reputation, or resources, he would make content on the internet about things he knows — for him, that would mean talking about business.
Post generous, not selfish, content
GaryVee says small business owners should ask whether each piece of content is selfish (asking for the sale) or generous (building connection), claiming that selfish content — like only posting listing photos and 'buy from me' messages — is why most businesses struggle with social media.
Make one bet: focus on a single offer when time is short
Alex advises focusing on only one thing when driving traffic to an offer because you don't have the luxury of time — make one bet. He chooses summer camp over season pass as the one bet because summer will make more sense. He also notes the business has one shot at getting the summer camp offer right.
Run continuous evals to mine failure signals and prevent repeat failures
Find an approach to continuously mine for signals where AI agents are going wrong, so failures don't happen again. Manual log review is not feasible at scale with increasing automation and longer traces, so scalable approaches like LLM-as-a-judge are needed, with occasional manual inspection of eval outputs.
Balance human taste with agent automation; avoid both extremes
Don't fully automate an agent with no human oversight and ask it to 'fix everything' — that produces slop, low quality, and unsatisfied customers. But also don't go the full manual way: don't instrument everything yourself or try to catch all failure modes and review all data yourself, since that can be automated. Find the ideal balance: bring your taste and domain experts in to build datasets and curate the problem and direction, work with your coding agent to automate instrumenting, failure-mode finding, and evals (e.g., 20 experiments instead of one), and gradually automate the eval loop over time. Avoid ending up in the bottom-left corner (full manual); be the tasteful AI engineer.
Set up bulk automated evals before production
Pamela Fox recommends setting up bulk automated evaluations that can be reliably run and compared before putting an agent into production, with lots of test cases, and making it easy to add more test cases to the eval suite.
Multi-thread fast — laterally and up — and don't wait on an unconvinced champion
The faster you can multi-thread and get to power, the better — more often than not. Multi-thread laterally, not just up: get two marketing managers on the next call instead of one — double the quantity of people you call. You might have to multi-thread across before you multi-thread up — that's the more conservative path. Don't stick with an unconvinced champion for 20 steps waiting for them to be convinced — you may delay the board send, but still go get other allies in the deal cycle.
Backtest agentic features one by one against true positive matches
Test each agentic feature individually against cases where the outcome you care about actually happened, to check whether the system could have correctly predicted it.
Backtest with a holdout set to guard against overfitting
Overfitting to a known evaluation set is dangerous because the algorithm likely won't succeed in the future; backtest with a holdout set, and verify performance beyond publicly visible sets since knowing the existing set enables overfitting.
Build app-specific tools and evals; no generic tool fits every application
The speakers recommend building up the tools that fit your own application and giving them to the models — no generic tool works for every application — and note evals will wildly differ by domain (interior design vs speed-critical businesses), with value in building them with your team.
Write down requirements for a good agent before building evals
Laurie Voss notes that before writing requirements down, different people on your team can have different ideas of what a good agent does. He advises writing down exactly what makes a report actionable: it makes a specific recommendation, names concrete risks, and looks forward not just back. A report that is factually true but lacks a concrete recommendation is not actionable.
Read your agent's traces before writing any evals
Laurie Voss advises reading a dozen or more traces end to end before writing a single eval, because without traces a failing agent just produces mysterious output and you can't see the failing tool call. He also suggests pointing your coding agent at the traces to look for weird things, since coding agents are aces at reading traces.
Read your traces before writing a single eval
Manually read a dozen or more traces end to end before writing any eval; it's an incredibly high-value activity that finds unknown unknowns, and if you automate before understanding how your agent fails you'll measure what's easy to measure instead of what matters.
Distinguish guardrails from North Stars and gate shipping on guardrails
Laurie Voss advises knowing which of your evals is a guardrail (should block release, e.g., agent making up a stock price) versus a North Star (nice to have, e.g., suggests other stocks worth a look), and only gating shipping on guardrails.
Define what 'good' means before measuring your agent
You can't claim an agent doesn't work without first writing down what working means; define the standard of good before building evals.
Build a better team through elimination
Hormozi recommends raising the team's bar by removing everyone who underperforms: get out everybody who sucks, and what remains is awesome. He frames elimination, not retention, as the path to a high-performing team.
Six infrastructure building blocks ordered by leverage
Xiaochen Jiang lays out six infrastructure building blocks that you should know about, ordered roughly by leverage, where number one takes the most risk off the table and number six does the least. The blocks covered include agent sandboxing, short-lived task-scoped credentials, identity provider enforcement, audit logging you own, model proxy layers, and isolated workspaces.
Protocols you can run
WHY · 59 of 59 published protocols
Adversarial Generators and Failure Analysis
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Backpropagation Without Autograd
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Bayesian Inference and Graphical Models
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Source analyses
WHY · 664 of 664 current source analyses
Choose sources → to get a library of your own and a daily brief.