34,197 results · Most relevant
Manually read a dozen or more traces end to end before writing any eval; it's an incredibly high-value activity that finds unknown unknowns, and if you automate before understanding how your agent fails you'll measure what's easy to measure instead of what matters.
Laurie Voss advises knowing which of your evals is a guardrail (should block release, e.g., agent making up a stock price) versus a North Star (nice to have, e.g., suggests other stocks worth a look), and only gating shipping on guardrails.
You can't claim an agent doesn't work without first writing down what working means; define the standard of good before building evals.
Hormozi recommends raising the team's bar by removing everyone who underperforms: get out everybody who sucks, and what remains is awesome. He frames elimination, not retention, as the path to a high-performing team.
Xiaochen Jiang lays out six infrastructure building blocks that you should know about, ordered roughly by leverage, where number one takes the most risk off the table and number six does the least. The blocks covered include agent sandboxing, short-lived task-scoped credentials, identity provider enforcement, audit logging you own, model proxy layers, and isolated workspaces.
Per Seth O'Neill, people with genetically small tendons should try to build up the tendon's cross-sectional area with a lot of heavy slow resistance training over a really long time. Plyometrics and sport play also contribute to building tendon cross-sectional area in addition to heavy slow resistance.
Attributed to Seth O'Neill; the plyometrics point is a lower-confidence quoted addition.
Coach Kyle recommends breaking approaching women into small, easy, executable steps you can do one at a time instead of thinking about the whole interaction. He explains that thinking about the entire interaction structure — how to ask her out, make it flirty, what if she has a boyfriend — puts you in your head and makes approaching uncomfortable, and that overthinking the whole interaction prevents action entirely.
Hormozi's core advice: if you want to get rich, deliberately pursue it as early in life as you can, because every advantage he describes compounds with youth.
You need to fully lean into and embrace the parts of you that make you different — typically the parts you hide for fear of judgment and mockery. The parts that make you different are the ones other people think are weird or cringe or don't fit in with the pack — which makes sense because you can't become exceptional without being different. Sometimes the parts that make you different are things you don't like about yourself, and that's okay. Embrace the components of yourself that don't necessarily belong together.
Be violently yourself with all of your idiosyncrasies, quirks, and atypical ways of behaving, and accept them as obviously different — in so doing they make you exceptional. X factor typically comes from a level of agency of self-acceptance: stars like Dolly Parton and Joe Rogan are willing to accept themselves as they were and just say, this is what I'm into. By accepting yourself as you are, people either admire that level of courage or see a part of themselves in you.
Anthropic's advice for building agent evals is to get high-quality data to evaluate over, iterating on your data and making sure the problem is set up correctly. They warned that garbage in, garbage out: ill-defined eval ideas can yield 100% scores that mean nothing about performance. They suggested looking at historical sessions and their results, then scoring them, as a good way to build an eval dataset. They noted Claude Code is pretty good at building an eval for you, and that while this is not an out-of-the-box solution, it is generally the advice that has worked.
Use boundaries and separate dev, staging, and production environments as in the normal software lifecycle. AI agents should never touch production directly — only through monitored CI/CD pipelines with safety controls in place.
Always put a human in the loop for approvals for things like deployment to production or dangerous operations such as deleting a database or making a user root.
The hosts advise maintaining evaluation criteria over time so evaluators don't over-index or stop providing meaningful information, and maintaining the evaluation dataset because initial assumptions about user behavior will miss unaccounted-for edge cases; they emphasize dataset maintenance as the most important part of continuously improving an agent.
Wolfram Ravenwolf recommends inspecting and controlling the system prompt because it is a very important evaluation factor. Telling an agent 'you are a helpful assistant' vs 'you are an autonomous agent' produces very different benchmark results.
Wolfram Ravenwolf recommends an isolated sandbox for each task, which is super important, because otherwise you would change your own system with every evaluation run and results would be even more random.
When choosing a model, you want a very high ceiling, a very stable average, and a very high solid base. When two models have almost the same average score, pick the one with the higher solid base. Also look at worst and best scores in addition to averages, and check what percentage of benchmarks a model can solve at all and how consistently.
Wolfram Ravenwolf recommends running agents in a sandbox instead of locally on your own system, because running locally causes interference between runs and corrupts results. Docker is a very common option, and cloud sandbox providers exist when Docker is hard to parallelize on your system.
Use both graph and vector approaches together rather than choosing one over the other. Graph is not going to completely replace the vector layer; the recommendation is to add graph on top of vector and use the benefits of both. A mnemonic for remembering retrievers: they are like a golden retriever that fetches the necessary information.
Using both is actually the key: use both graph and vector rather than replacing the vector layer with a graph. Vector-only RAG and vector-only databases fall short on tasks requiring node traversal and historical context.
Choose sources → to get a library of your own and a daily brief.