417 results · Most relevant
When designing a harness, include on purpose those gates so you can still be accountable for what is being produced. Make sure any important judgment call or disambiguation — not the trivial ones — you can still get hold of. Without accountability gates, the only answer to 'what happens?' is 'the stochastic machine said it was fine.'
The principles build on top of each other and form a logically sensible order: build a shared domain (plan mode) first, then set up consistency, then evidence, then accountability, then finally verification.
Treating every input as a fact when some are estimates can invalidate a whole financial model. The model cannot tell them apart on its own, so you must enforce the distinction through harness design.
Include review gates on purpose when designing a harness so you can remain accountable for what is produced; without them, your only answer is 'the stochastic machine said it was fine.' Capture important judgment calls and disambiguations — not trivial ones — at those gates.
The main takeaway for designing accountability mechanisms is to use few gates with very high signal. Breaking tasks down in a way that you can reason about the issues and how they connect is a good example of a high-signal gate. You can be a little hand-wavy with reviews before functional tests pass, but you wouldn't push anything into production that hasn't passed all your tests.
If you don't make sure calculations run in code, the arithmetic is in the model's head and is gone once you start a new session. Make the harness strict so the agent doesn't do arithmetic by parsing tokens.
Francesco recommends first building a shared domain with the agent — calling it plan mode — before anything else, then setting up consistency.
Design a harness with a branching structure where decisions branch out and join back in, so the model can verify assumptions with you or figure out how things fit into the overall system.
endorsed advice
When evaluating AI adoption, first define what outcome you're looking for and how you measure success; e.g. if the goal is less back-and-forth with a BI team, validating a new dashboard, or fewer internal ask tickets because a number looks wrong, track that metric and see if AI can support generating that outcome.
endorsed advice
Move from a skill-based prompt approach to a separate tool when the use case becomes critical enough that the trade-offs of the non-deterministic approach are not acceptable.
endorsed advice
Write the calculation once and then run it every time you need it. The model can pick tools, but your harness must make it very strict that it will not do any arithmetic or try to solve these problems by parsing tokens. If calculations aren't in code, the arithmetic is in the model's head and gone once you start a new session, and new code gets generated for every new question or session until it becomes a mess. Asking the same question and getting different answers is non-determinism — you're asking a stochastic machine to produce a consistent output, so you're likely not going to get it.
Francesco Galletta recommends sharing a domain with the model and agreeing on a plan so you and the model talk in the same terms, noting that your definitions likely differ from what the model knows because its training differs from your context, and that going back and forth with the model really makes a difference in getting a shared understanding.
Francesco Galletta explains the harness should not go into yes-man mode and implement an unclear idea; instead it is guardrailed to force the user through every single data point needed to come up with a plan, helping them drill down into what actually needs to be done and challenging assumptions they may not have known they had.
When presenting analysis or doing financial analysis on an investment, you need to show where your numbers came from and attribute your decision to the body of evidence you had. 'Trust me, I ran the numbers' without a trace should not be trusted.
In data and finance you need a separate point of reference, a counterfactual you can check against, without redoing your plan entirely. Have something review and give you a rough answer or challenge if something doesn't look right.
Start from the outcome you want and how you'll measure success, rather than adopting AI because it can do something. The specific tool matters less than the result.
Open a branch on a tagged challenge: a separate agent tasked with reviewing the plan, reviewing the assumptions, and doing healthy challenge.
endorsed advice
Look for external points of reference outside your own data sources to enhance signal and avoid the feedback loop where only your own information and golden examples exist and everything looks right. Use the internet when a source claims a number, and get creative about other references.
endorsed advice
Francesco recommends that in data and finance work you have a separate point of reference — a counterfactual — that you can check against.
The hosts recommend that when you're building a harness, you make sure you can make the evidence trail happen for your customers using your agent, so that they trust the results it gives them.
No published protocol matches this selection.
The creators behind most of what cleared.
344 cleared insights match
71 cleared insights match
Choose sources → to get a library of your own and a daily brief.
