Explore

The harness guides code creation and test running; don't rely on remembering to do it yourself

As you accept the plan, a graph is built and you get a prompt at the top asking 'do you want me to build the code?' and it will do it for you — it's all guided. The point of a harness is that it guides things. In the wedding example you can click on step one, then code, then the test, and run the tests now. You shouldn't rely on yourself remembering to run the test, or remembering that something sounds like it can be created as code to solve it every time. The agent should guide you through running tests and creating code — one of the principles baked into building this harness.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

A 'looks about right' challenge result can still miss that the company cares about a 5% difference

When a challenge branch result looks about right, it may still miss that your company actually cares about a 5% difference — and when that's the case, the harness must reflect that threshold.

endorsed; numeric figure flagged as possible transcription error

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Agents increasingly do aggressive tool calling for deterministic calculations; waiting for agents to improve creates in-flight issues

Over the last year or so, most agents have started to do more aggressive tool calling and building their own tools to solve calculations in a more deterministic way. But waiting for agents to improve so all problems are solved will create issues on the go, in flight — relying on agents to generate code on the fly may produce the right code at one point in time, but by run 100 it may invent a new piece of code that didn't need to be reinvented.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

The evidence-gathering and verification recipe is geared to financial use cases

The evidence-gathering and verification recipe is specifically geared towards financial use cases and may not be as useful for other use cases; for AI-generated social media content, evidence gathering and verification steps are probably not needed.

endorsed

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

'Trust me, I ran the numbers' isn't enough; a trace to follow back increases human confidence

When a data person or analyst says 'no, no, no, here's the result. Trust me, I ran the numbers,' that is not something to trust. Nobody in the audience would want to see all the logs and the math behind a single number, but having a trace to follow back a decision increases confidence for humans quite a bit.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Simple research tasks can run on plain agent runs; nuanced financial planning needs something robust

For simple research tasks like finding the top five photographers in Madrid, a new web search with refined tooling worked like a charm with nothing else needed; but for nuanced use cases needing control and predictability, like financially related planning, you need to build something robust rather than relying on ad-hoc agent runs.

anecdotal

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

When money is involved, small error rates are expensive — models still must earn trust

When talking about money, 1% wrong is sometimes a lot of money. Models are going to get better — the last six to nine months have been insane, giving even a bit of whiplash with how fast things are going — but they still have to earn the trust; it's not 100% right.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Tangential domains reveal nonsensical numbers without re-checking calculations

When you look at a slightly tangential domain, you may find out that a number is nonsensical or wrong just by looking at the other thing next to it — without re-checking your calculations.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

The harness guides question rigor and triage before implementing anything

The harness guides the sort of questions you get asked and the rigor the plan needs before implementing anything. Most harnesses today come with a notion that triages that you're trying to plan for a problem and will try to come up with a reasonable idea and a set of questions for more clarity.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Aligning user domain knowledge with the agent takes most of the time; find the Venn diagram superposition

Aligning the user's domain knowledge with the agent's knowledge takes a lot of back-and-forth and is often where most of the time is spent; users should try to find the Venn diagram superposition between their domain knowledge and the agent's knowledge as much as they can.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Trust in the UI comes from aggregating provenance across all steps

To demonstrate trust in a UI, the intent was aggregation of everything — all steps in the framework should convey 'I know where this is coming from, I know this is running SQL, I know this is tested'.

described design intent

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

A good harness guides you to a good plan; a yes-man harness gives variable results

If you design a really good harness, it will guide you by the hand in coming up with a good plan, implementing things well, and knowing which components matter. If your harness is basically 'say yes to everything I say and do your best to implement it', you will get variable results.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Nuanced, control-sensitive use cases need something robust, not recalculation

For more nuanced use cases where more control and predictability are needed (linked to their financial nature), Francesco found he needed to build something robust rather than recalculate repeatedly.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Harness demo: agent interviews user, then breaks problem into a decision tree

In the Monzo demo, the 'plan accepted' indicator means the presenter already spent around 10 minutes of back-and-forth; the agent then breaks the problem into steps represented as decision points in a decision tree, and the interview behavior itself comes from instructions inside the harness.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Leaving implementation decisions entirely to the agent yields hit-or-miss results

If you just leave it to the agent to decide how to implement things and when to do it in code versus building something on the side, you're going to have a big miss ratio — hit or miss.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Play with the agent manually before switching to automode

When you want to see what's happening with an agent before enabling automode, play with it manually first and then switch to automode.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Tools supporting key company decisions for many users must minimize variability

When putting a tool in the hands of many internal users supporting key company decisions, you don't want a lot of variability.

endorsed

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

The spectrum: embed everything in skills vs building every tool from the ground up

One end of the spectrum is to use Claude Code or Codex for everything and just embed everything into skills, instructions, and so on. The other end of the spectrum is building every small new thing from the ground up — which gets tiring because you spend more time creating your tools than actually using the tools sometimes.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Figure out what you care about: functional requirements and intent per step

It all goes about figuring out what you actually care about and what the functional requirements for a functioning harness are. When building the parts that make up the behavior of your harness, the objectives of each of the steps should have an intent — e.g., the intent of one of the first components is gathering as much information as possible to come up with a plan, which is written in the harness instructions.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Uncaught example fixation means unexplainable numbers in front of leadership

The hosts recount showing some colorful example to senior leadership, being asked where a number and assumption came from, and not knowing — which hurts the trust you have in your harness and the results it produces.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo