Creator perspective · MoneyNick Nisi describes relying on CI and the tests running in CI to validate agent work before shipping, plus automated code review tools — his team uses Greptile, and he names CodeRabbit and Codo as alternatives. He treats a 5-out-of-5 Greptile score plus all CI checks passing as the signal that a change is safe to ship, giving him extreme confidence he won't break production. At the end of the day he reviews every lane/branch started that day and confirms each is correctly done.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyZack Proser recommends getting to the point where your changes and your PRs automatically flex that eval, so you know if changes are safe to land, whether they're improving things, making things worse, or not changing anything at all.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyNick Nisi advises thinking about how you would reproduce a given bug and, if you can break that down, encoding it for the agent. As a concrete example, for sign-on flows you verify the cookie is properly set, the expiration is set, and the claims in the JWT are correct. He also notes that error messages matter more than ever: when the app throws an error, an agent can interpret and run with that error really well, so poor error messages block self-healing — and observable state (like ball coordinates in a canvas game) should be surfaced to the model so it can debug what it cannot see with its own eyes.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyNick Nisi walks through the possible failure outcomes of the verification game: the agent names the bug with output but has no evidence, claims it works when it didn't, claims it works when it did, or blames a bug that wasn't there. He highlights the 'overpromised' case where a screenshot or judge check reveals the agent claimed a pass that didn't happen, and the case where code lies — after a page reload the app says 'waitlisted' while the screenshot shows the seat confirmed, and the mismatch is itself a failure. He also cautions that a successful screenshot is not proof: it may show a state you didn't expect, cause side effects, or take a UI path that isn't actually possible.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Report · MoneyWisedocs' system is set up so that a human always makes the final decision, with AI surfacing insights for human review rather than acting autonomously.
AI Engineer · Insurance Claims Are AI's Messiest Data Problem — Denys Linkov, Wisedocs
Creator perspective · MoneyDick explains that OAuth 2 added scopes, but before that it was just 'do anything', and it required preregistration and expected a browser with a redirect to be involved. In the agentic world, you don't know ahead of time what the agent needs to do; the authorization decision must be made over time, and different agent instances need different kinds of access. Related framing: making a payment is not a scope; it is whether you can do this one thing in this particular context.
None beyond the speaker's own framing.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Report · MoneyLaurie Voss suggests that if you don't have users, you can make some up: get an LLM to generate lots of test queries and vary them on purpose — e.g., 'Research Tesla's financial performance', 'what's going on with Tesla stock', and 'yo, is Tesla buy right now'. Then get a human to look at the LLM-generated query list, because LLM-generated queries are a useful way of seeding your initial data but are not perfect. LLM-generated queries are relatively conservative and don't go as far afield as human users tend to actually go — human users send wildly off-domain queries, like asking a financial analyst agent 'tell me what's wrong with my child's health', and your agent has to have something to do with that.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss explains that adding Phoenix tracing to an agent project is minimal: call register from the Phoenix OTEL package, which sets up tracing and points it at Phoenix. Frameworks already have instrumentation built in, so the register command just turns it on and points it at your Phoenix instance (e.g., your laptop).
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyAsking a machine to follow all instructions and make no mistakes will probably work 99% of the time, but the 1% edge cases are the real risk. Determinism via back-end code covers these out of the box.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyWhen a calculation needs to be repeated reliably, convert it from agent-driven to a script. Letting the agent decide categorization, aggregates, refunds, and expenses yields slightly different results every time and wastes money and time.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyWhen reviewing agent work, review the set of tasks the agent plans rather than approving every individual change; let the agent do its own thing and review the results at the end. Harness design should be intentional about this.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyIf you don't have a way to validate, you end up checking everything, checking nothing, or having no sense of what about-right looks like — and if you don't have that point of reference, your harness won't have it either. You need to build external validation points for your harness. For verification, add notes in the decision — an external point of reference added to some of the stuff the model thought.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneySubmitted documents like bank statements are facts; agent reasoning or information pulled from memory must be treated as assumptions; unstructured thoughts in chat must be labeled as interpretations.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyThe boundary between a skill and a separate tool is when the use case becomes critical enough that the trade-offs of the non-deterministic approach are not acceptable. Felippe describes the prompt-based (skill) approach as: 'Hey, please follow all of this and make no mistakes. Please, please, please.' Prompt-based approaches work because agents are good enough nowadays and will probably perform your instructions 99% of the time — but the question is what about that 1%.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyBecause you have code that is tested, you have a trail you can follow backwards for evidence. Tests run in your environment with no agent thinking involved, and skipping them sabotages the analysis.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyBuild a harness when you feel the use case is worth enough — when you want reproducibility, or when the results you are getting using pre-built harnesses are very frequently getting it wrong. There's no clear answer to how generic or specific a harness should be or how often to build one — it comes down to use case value, number of users, and problem criticality. The decision on whether to build a harness comes down to how many users you believe will go through that use case and how critical the problem is that you're seeing.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyEvidence goes into the code itself: track the provenance and the method behind every number, not just its source. Gather as much evidence as you can and assign the right level of importance to each piece, the same way you would curate a dataset for training.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyTreating all information the same without distinguishing facts from estimates leads to fatal errors in financial analysis, because a single wrong assumption can invalidate the whole model. Enforce the hard-truths-vs-estimations distinction through harness design.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · BodyGrubbs argues that in professional baseball, if players don't lift on game day they effectively never train — skipping game-day lifting across a 162-game season means losing size and building no strength foundation. She frames game-day lifting as something done 'literally based on necessity.'
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
PrincipleMichael explains the Swiss cheese model from aviation safety: multiple layers of defense each have vulnerabilities (holes), and safety requires ensuring the holes don't line up so a critical scenario cannot pass through all layers. He details the aviation layers (strategic separation, air traffic control, collision avoidance system) and applies the model to LLMs (training data control, runtime monitoring, chain-of-thought monitors).
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club