Recommendation · MoneyDavis recommends using intent-based auth to help users ensure their agents buy the right item within their personal budget, capturing purchase intent data to understand why a purchase is being made and whether an agent should be trusted to visit a merchant's site. He also recommends extending the know-your-customer concept to 'know your agent', and coupling the agent identity with the intent of the user, operator, and processor for financial services.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyDavis recommends that enterprises onboard agents like they onboard developers: don't give full commit access day one; add levels of capability as the agent matures.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Recommendation · MoneyLaurie Voss lists code eval possibilities: check the output parses as JSON, is under a length limit, avoids phrases like 'as an AI language model', query your database to verify state, verify tool calls did what they should, and call an API to double-check reported stock prices. Anything that gives the same answer for the same input every time can become a code eval.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyThe same test suite can be run in GitHub Actions, which starts Phoenix as a service container.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Anecdote · MoneyA judge with a knowledge cutoff earlier than the agent's data will fail correct reports — Claude's cutoff around January made it flag October stock prices as future hallucinations, scoring all 16 reports incorrect. When an eval scores everything the same way across the board (e.g., all zeros), treat it as a signal that the eval is probably the wrong choice — pick a question the judge can actually answer.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Recommendation · MoneyLaurie Voss recommends marking the data clearly as part three of a good rubric: wrap each piece of data in its own XML tag. Claude was trained on billions and billions of XML tags and is really good at working with them. Wrapping data in XML tags creates clear boundaries between input, output, and instructions — this helps against prompt injection and also helps the agent from getting confused. If you include examples in your rubric, mark the example as an example and not the data — otherwise it will get very confused.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyLaurie Voss describes running faithfulness with the same pattern as before: create the evaluator, evaluate every run, and log the results. He notes faithfulness evals are slower than correctness because the judge is reading every search.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyThe implementation: query Phoenix for project spans filtered to parent_id null (top-level agent run spans), use the getAgentRuns helper to fetch each run's span ID, input, output, and research turns; wrap an ordinary function with createEvaluator from the Phoenix evals package, marking it a code eval so no LLM is involved; the function checks each requested ticker appears in the report and returns a label and score, naming the missing ticker so the failure explains itself; run it on every agent run and log span annotations back to Phoenix. Run code evals with npm run eval code.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyHave the experiment task hand back the report and the research, then give each eval a small wrapper that picks out the fields it needs (e.g., faithfulness gets the research as context). Run the experiment script with 'npm run experiment'; it takes about four minutes to complete.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyMap every prompt change to a specific eval finding: judge said 'makes no recommendation' so ask for an explicit recommendation; 'mentions risks with no evidence' so ask for risks with supporting data; faithfulness evals find invented numbers so instruct the agent to use only numbers that appear in its research; trace reading finds the agent trying to save files constantly so change the prompt to tell it to reply with the report instead.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyWire up a classification evaluator by giving it a name, a model, a template, and its output options (actionable with score 1, not actionable with score 0).
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Recommendation · MoneyIf you're getting golden examples of how the data should look, a good argument is: why would your process give you something different to that? Why don't you just check with your golden examples? Golden examples should be checked against the process output — if your process gives something different, check why against the golden examples.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Anecdote · MoneyThe speaker showed a colorful example to senior leadership; when asked where a number and assumption came from, they couldn't answer ('I don't know where that came from'), which damaged trust in their harness and its results.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Method · MoneyTo start a harness session, describe the problem you want to achieve in the chat window — the chat window on the right-hand side starts essentially empty and is the place to start with a financial worry or analysis you want to do. Walk from the goal you want to achieve and try your best to give an explanation to the agent, and you can keep adding clarifying information to the chat as the planning conversation continues. If you want to run the harness for a problem you care about, start by describing that problem in the chat; if you pick your own direction, you can enter things like planning your own wedding, a trip, or a new investment in your company or personal life — anything relating to financial life where the same principles apply.
Method, not a verified protocol.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Method · MoneyYou can open a branch on a tagged challenge — a separate agent tasked with reviewing the plan, reviewing the assumptions, and doing some healthy challenge: does this read reasonable? Click the tagged challenge open, and you will see how Felipe did — he will join back into the trunk and change the plan, so there is a consequence. You can try opening another branch and just clicking on dismiss — some conversations don't actually change anything or aren't material to your process, and you will know better.
Method, not a verified protocol.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Anecdote · MoneyA person using a shared master file didn't know its embedded assumptions had changed; they did nothing wrong themselves, but their results became entirely wrong. Traceability for changes in shared files prevents this.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Recommendation · BodyAdvice Grubbs received and passes on: strength coaches should 'wear their suit and walk around' — meaning what you're saying is what you're actually doing. She practices this in hydration: wake up, wait to drink caffeine, and drink water first.
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
Recommendation · BodyGrubbs tells people they can get into the field knowing no one — she did, with parents who knew nothing about sports or coaching. But be strategic and tactful about where you pick to do internships, and consider the 'projectability' of an internship: where it will take you next (e.g., the Clemson tree, then Texas or wherever). Projectability is super important.
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
Recommendation · BodyDon't treat NordBord, Force Frame, and 1080 as testing-only tools: use Force Frame for safe arm care isometrics at 30-50% of max, use the 1080 for plane-specific work like frontal plane missions or linear sprints, and use the 1080 to measure limb-to-limb asymmetry during acceleration; force-velocity profiling with varied loads can reveal asymmetry.
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
MethodClark Barrett describes a workflow: with VerisForge, you enter a specification in English of what you would like to build, and it builds the code and does the formal proof. VerisForge (and the CryptoProver agent) generates a structured specification, generates Rust code and a Veris specification, then verifies it with the Veris tool and checks the spec is unchanged, producing fully automatically verified code. Somebody still has to write the specifications — someone has to say what properties the code should have.
Method, not a verified protocol.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club