Method · MoneyThe implementation: query Phoenix for project spans filtered to parent_id null (top-level agent run spans), use the getAgentRuns helper to fetch each run's span ID, input, output, and research turns; wrap an ordinary function with createEvaluator from the Phoenix evals package, marking it a code eval so no LLM is involved; the function checks each requested ticker appears in the report and returns a label and score, naming the missing ticker so the failure explains itself; run it on every agent run and log span annotations back to Phoenix. Run code evals with npm run eval code.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyHave the experiment task hand back the report and the research, then give each eval a small wrapper that picks out the fields it needs (e.g., faithfulness gets the research as context). Run the experiment script with 'npm run experiment'; it takes about four minutes to complete.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyMap every prompt change to a specific eval finding: judge said 'makes no recommendation' so ask for an explicit recommendation; 'mentions risks with no evidence' so ask for risks with supporting data; faithfulness evals find invented numbers so instruct the agent to use only numbers that appear in its research; trace reading finds the agent trying to save files constantly so change the prompt to tell it to reply with the report instead.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyWire up a classification evaluator by giving it a name, a model, a template, and its output options (actionable with score 1, not actionable with score 0).
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Recommendation · MoneyIf you're getting golden examples of how the data should look, a good argument is: why would your process give you something different to that? Why don't you just check with your golden examples? Golden examples should be checked against the process output — if your process gives something different, check why against the golden examples.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Anecdote · MoneyThe speaker showed a colorful example to senior leadership; when asked where a number and assumption came from, they couldn't answer ('I don't know where that came from'), which damaged trust in their harness and its results.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Method · MoneyTo start a harness session, describe the problem you want to achieve in the chat window — the chat window on the right-hand side starts essentially empty and is the place to start with a financial worry or analysis you want to do. Walk from the goal you want to achieve and try your best to give an explanation to the agent, and you can keep adding clarifying information to the chat as the planning conversation continues. If you want to run the harness for a problem you care about, start by describing that problem in the chat; if you pick your own direction, you can enter things like planning your own wedding, a trip, or a new investment in your company or personal life — anything relating to financial life where the same principles apply.
Method, not a verified protocol.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Method · MoneyYou can open a branch on a tagged challenge — a separate agent tasked with reviewing the plan, reviewing the assumptions, and doing some healthy challenge: does this read reasonable? Click the tagged challenge open, and you will see how Felipe did — he will join back into the trunk and change the plan, so there is a consequence. You can try opening another branch and just clicking on dismiss — some conversations don't actually change anything or aren't material to your process, and you will know better.
Method, not a verified protocol.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Anecdote · MoneyA person using a shared master file didn't know its embedded assumptions had changed; they did nothing wrong themselves, but their results became entirely wrong. Traceability for changes in shared files prevents this.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Recommendation · BodyAdvice Grubbs received and passes on: strength coaches should 'wear their suit and walk around' — meaning what you're saying is what you're actually doing. She practices this in hydration: wake up, wait to drink caffeine, and drink water first.
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
Recommendation · BodyGrubbs tells people they can get into the field knowing no one — she did, with parents who knew nothing about sports or coaching. But be strategic and tactful about where you pick to do internships, and consider the 'projectability' of an internship: where it will take you next (e.g., the Clemson tree, then Texas or wherever). Projectability is super important.
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
Recommendation · BodyDon't treat NordBord, Force Frame, and 1080 as testing-only tools: use Force Frame for safe arm care isometrics at 30-50% of max, use the 1080 for plane-specific work like frontal plane missions or linear sprints, and use the 1080 to measure limb-to-limb asymmetry during acceleration; force-velocity profiling with varied loads can reveal asymmetry.
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
MethodClark Barrett describes a workflow: with VerisForge, you enter a specification in English of what you would like to build, and it builds the code and does the formal proof. VerisForge (and the CryptoProver agent) generates a structured specification, generates Rust code and a Veris specification, then verifies it with the Veris tool and checks the spec is unchanged, producing fully automatically verified code. Somebody still has to write the specifications — someone has to say what properties the code should have.
Method, not a verified protocol.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
RecommendationThe benchmark ecosystem needs version control, pull requests, and issue filing. There also needs to be some kind of coordination — with Hugging Face or a central place — where agents can find that benchmark resources have problems.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
RecommendationBenchmarks need version control, pull requests, and issue filing.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
Method · MoneyNacho Martinez describes experimenting (with Risman) with plugging into external third-party skill and tool hubs: pre-compute embeddings of a collection of MCPs, then run vector search against that third-party group of skills or tools.
Method, not a verified protocol.
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyNacho Martinez describes one of the easiest ways to do skill distillation: create a skill MD file and put it into the same skill box that held everything before; skill distillation is one of the techniques to improve the learning of a model over time.
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyUse hybrid search — a combination of semantic search (embeddings) plus keyword search — for skill retrieval. Hybrid search combines keyword search and semantic search so that results in the combined plot are more relevant than either method alone. RRF (Reciprocal Rank Fusion) is a type of hybrid retrieval that performs much better than either search method alone. Combining keyword and semantic search costs a little more compute but yields better retrieval quality.
creator's recommendation; compute-cost tradeoff noted
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
MethodWhile dynamic workflows run, you can watch what's happening in real time via a single event stream and check agent by agent what events are happening. Set budgets on your sessions before kicking them off so you don't accidentally burn substantial amounts of tokens — token hunger is a real problem when running many agents. If you don't like the direction agents are going, you can steer them in real time by interrupting via the API surface.
Method, not a verified protocol.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
MethodIf you installed Claude Code starting Sunday night, run 'claude API managed agents onboard with Watchlist Scanner' to spin up a template and manage agents directly from Claude Code. Then tell Claude: 'I want you to use a workflow, one ticker per agent, and go out and do this scan. This is what I'm looking for' — letting Claude take control over building the initial flows, then fine-tune afterwards.
Provenance flags a possible numeric error on one atom.
Method, not a verified protocol.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic