417 results · Most relevant
Approving every calculation or reviewing every PR line by line that agents ship doesn't feel great — it's the other side of the spectrum from full automation and leads to a lot of toil and unnecessary work if not designed for the use case.
When presenting analysis to your firm, attribute your decision to the body of evidence you had rather than just stating the conclusion; show where your numbers came from — something that bites data professionals weekly.
Building on the shared domain principle, the agent understands that pieces of the plan that are just code or a calculation will always give a consistent, deterministic result. Based on identifying deterministic elements of the plan, you need to implement code — something that gives the same result every time.
In an AI-first world, we sometimes forget that AI is only a means to an end, not the point itself. At Monzo, they approach AI from a highly impact-based point of view: not 'we should use AI because AI can X', but 'what's the outcome we're looking for here?' Define how you measure success for an AI initiative — e.g., are you looking for less back-and-forth with a BI team, or to validate a new dashboard — so you can track whether AI supports that outcome. Track metrics like fewer internal ask tickets (because a number looks wrong) to see if AI supports generating that outcome. The way you achieve an AI outcome doesn't really matter to a point, because AI will be a tool that gets you to the result probably faster, with better guardrails.
If you don't have traceability even for changes you make in your deterministic code, you leave yourself vulnerable to blind spots when the process changes, even if the inputs are the same.
Francesco Galletta argues the goal is a financial analyst harness you can actually trust, rather than ad-hoc chatting with spreadsheets, and that design principles for achieving that trust matter more than specific technologies or product readmes.
Thin slicing is a concept where you have thin slices of your anecdotal experience, and you know what things feel right or feel wrong based on your experience in your domain, whatever analysis you're doing. The model will have an average of everyone else's experiences. When you're building your harness, you need to make sure that you can have a feedback mechanism so that you can verify from what the model thinks it knows all the way to what you think you know.
The gates you design need to be fit for purpose — you need to know the use case really well to put gates where it matters. Adding a gate before every decision asking 'ask me' makes it very low signal again.
Giving an agent a bunch of instructions that say the right things but still rely on the agent not making mistakes is the non-deterministic way and produces a non-deterministic result every time. The value of building a harness (rather than plugging principles into skills/instructions for an existing harness like Codex or Claude Code) is mixing non-determinism with deterministic behavior you want to control when relevant. The principles of mixing determinism and non-determinism are the same regardless of implementation details like abstractions, methods, or actual code.
The design of your harnesses needs to be intentional to increase trust. AI results will likely not be accurate unless you design all the mechanisms so that whatever process runs, you know you can trust the results to a degree. Don't assume that because something is running on AI it's going to be '200% accurate' — it will likely not be.
If you're getting golden examples of how the data should look, check with your golden examples rather than letting your process give you something different.
If there's a non-business-critical test you're solving, you're probably fine to have a human in the loop reviewing the outputs and steering the agent, even if that takes a few extra turns. If you're putting something in the hands of an internal user — for instance, 50 credit analysts using something to support key company decisions — you absolutely don't want a lot of variability on that.
Bad decisions carry forward into your other decisions. The gates you design need to be fit for purpose; you need to know the use case really well to put the gates where it matters. Adding a blanket 'before any decision ask me' gate makes the mechanism very low signal again.
The presenters recommend building verification into the system with a separate point of view or another perspective; in data and finance you often need a separate point of reference — a counterfactual — to check against, and you should verify things are going in the right direction, not just that things look correct. Ignoring verification leads to running things on your own biases, so results will always be skewed.
For code, you don't necessarily need to ask before decisions because the tests will speak for themselves; check that everything is passing and you will have a lot of confidence in the output. You can be hand-wavy with reviews prior to functional tests passing, but never push anything into production that hasn't passed all your tests.
When the model assumes a decision, verify with it by asking whether that's the number it wants to use. You are well within your right to correct information and add more context after a decision has been made.
The trust users have in your product, your agent, and its output is what will drive usage, adoption, and improvement through the loops mentioned — tracing and evolutions. When building a harness, make sure you can make trust happen for customers so they trust the results the agent gives them.
When a data person says 'trust me, I ran the numbers' without showing their work, the correct response is to not trust it.
Evidence is about the receipts — being able to show that this is where this came from, this is correct because of, and doing that causality relationship. It builds on the second principle: since you have code that is tested, as well as everything else, you have a trail that you can follow backwards — a way to substantiate the decisions you are making.
When you have a few possible values for something, ask the agent to do three scenarios and review them side by side before making any decision.
No published protocol matches this selection.
The creators behind most of what cleared.
344 cleared insights match
71 cleared insights match
Choose sources → to get a library of your own and a daily brief.
