34,068 results · Most relevant
GaryVee advises focusing on winning over the 10% of people who can detect your intentions. When they smell that you are not actually trying to sell them anything — that you are just listening to where they're at — you completely dominate the 90% and win over the 10% who are rarely won over, making you an unstoppable salesman.
Al Chen points out that most scanning tools (Sysdig, Wiz, Prisma) only check whether a package has been bumped to a specific version and say nothing about whether the exploit is still in the application code. One solution is creating a Harbor task — like a harness or world for an agent to try things out — containing instructions for the agent, an environment with the actual Docker images, and tests to run against the agent.
endorsed
When evaluating models for RAG, Pamela Fox says to check how likely the model is to make stuff up (hallucinate) and whether it is willing to admit what it doesn't know. She also forces the model to prove every piece of information in its answer came from the provided sources, to reduce hallucination in RAG.
When evaluating a predictive system, set the snapshot date to just before the outcome was announced. Live search leaks future knowledge into an evaluation of the past, so constraining the snapshot prevents the system from being informed by the confirmation of the outcome.
Nick Nisi recommends leaning away from relying on skills alone for verification because it's up to the model whether it follows that path; instead prefer CI enforcement of the commands. Verification checks can run regularly as part of CI. Hooks are another option — though a git hook is probably the last one he'd reach for — and hooks are relevant for local development across a whole team because they can be shared. He also suggests putting an agents.md (not a claudemd) in the app with wording like 'on change, verify things by running this' or 'before you open a PR, do this'; with that in place, agents have to go out of their way to ignore or avoid the verification instructions. One could even build a full pi extension to keep verification running in the background and feed in information.
Nick Nisi recommends that if you have a web app and don't already have a custom verifier CLI, building one is one of the highest-leverage things you can start with for your codebase. He stresses that you must build tools that fit your specific application — without them the model has no way to prove correctness for your particular app — and that the tooling should produce detailed output about what happened and the state of the system so it can be audited by the application and the model.
The speakers warn that a successful screenshot is not proof: make sure it actually shows the state you expect and that the commands to reach that state actually ran, otherwise it might just be another lie. A screenshot might have taken a different path that's not actually possible within the UI or caused side effects — verify it ran start to finish.
The speakers describe giving the agent a map with a verifiable destination — 'you must get here. You'll know you're here when this command returns this' — and note you don't necessarily have to prescribe how the agent gets there. If you prescribe an exact step-by-step path, agents may not sway from it, and you want them to sway a little while still reaching the right end goal; given a destination without a path, they may wander, catch more edge cases, figure something out, or ask a question that unlocks something.
Nick Nisi emphasizes that the value of keeping receipts from every verification run cannot be overemphasized. For an e-commerce app, receipts should cover the full user journey: login, page hits, errors seen, catalog load time, cart actions, and cart abandonment. Those receipts can be fed to LLMs, or a system can run all night aggregating the latest receipts and reporting failure patterns — for example, eight failures in the caching module — and a PR can be issued that night to fix it, with proof included in the PR that the issue is fixed.
Davis says the more successful enterprises onboard agents like they onboard developers: agents do not get full commit access on day one and must add levels of capability as they mature. When determining an agent's autonomy level, enterprises should evaluate the context that makes the agent safe, including the environment around it, not just what the agent attests about itself. Whether an agent is running in a sandbox is a simple example of environmental context that can change the level of autonomy allowed.
None stated.
Dick believes there should be a separation of concerns between a worker agent that does the work and a separate supervising agent that decides what the worker agent gets to do; the supervisor cannot do any of the work itself. The supervising agent should be independent of the agent doing the work — it runs your policy and decides how things get done. Without an independent supervising agent, guardrail decisions about what an agent can and can't do get decided by the agent's vendor — and Dick does not want to cede that control. A request to read the CEO's email is outside the scope of an instance agent and should be rejected as intent from the user; a request to forward the CEO's email to the outside world is vaguer and should come back to a supervising agent, which would decide it is a bad
None stated.
Laurie Voss argues you have to write down what good means before you measure it — you can't say your agent doesn't work if you haven't specified what working means. For a good financial analyst report: it should use real current data, be about the right company, stick to what it actually found, and (for his purposes) say what to do — buy, hold, or sell the stock — not just give analysis. Requirements sound obvious once written down, but before that they could be vague, or different people on your team could have different ideas about what a good agent would do. Writing requirements down turns vague disappointment into specific things that you can test.
Laurie Voss insists you have to read your traces before you write a single eval: read a dozen or more of them end to end — what was the input, what was the output, what specifically is wrong. Manually inspecting traces sounds manual and old-fashioned, but it's an incredibly high-value thing you can do when building an agent. You have to manually inspect what is going on so you can find the unknown unknowns that you weren't expecting. Anthropic's team regularly sits down and reads eval transcripts by hand and has built a bunch of internal tooling to make that easy to do.
Laurie Voss describes the three eval types (code, LLM judge, human), their individual flaws, and recommends using all three simultaneously because they are complementary.
Laurie Voss explains that the span kind tells you what type of step it is — an agent tool, an agent, a tool call, an LLM, and so on. In tool calls and web searches, the span input is JSON, and that structured data makes the first eval easy to write. Spans also include timing and token counts, and often your agent will record what it costs — these are the fields to pull out and grade when writing evals.
The principles build on top of each other and form a logically sensible order: build a shared domain (plan mode) first, then set up consistency, then evidence, then accountability, then finally verification.
Treating every input as a fact when some are estimates can invalidate a whole financial model. The model cannot tell them apart on its own, so you must enforce the distinction through harness design.
Include review gates on purpose when designing a harness so you can remain accountable for what is produced; without them, your only answer is 'the stochastic machine said it was fine.' Capture important judgment calls and disambiguations — not trivial ones — at those gates.
The main takeaway for designing accountability mechanisms is to use few gates with very high signal. Breaking tasks down in a way that you can reason about the issues and how they connect is a good example of a high-signal gate. You can be a little hand-wavy with reviews before functional tests pass, but you wouldn't push anything into production that hasn't passed all your tests.
If you don't make sure calculations run in code, the arithmetic is in the model's head and is gone once you start a new session. Make the harness strict so the agent doesn't do arithmetic by parsing tokens.
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Choose sources → to get a library of your own and a daily brief.
Map each implemented component to the original paper and explain 2 reduced-model failure modes without claiming full-scale reproduction.
Use calibration and shift measurements to justify a monitoring or abstention threshold, and give an explicit confounding counterexample to causal feature-attribution claims.
Defend a causal claim or refusal using the DAG, numerical estimates and a counterexample showing why predictive model fit cannot establish identification.
