417 results · Most relevant
If you don't have a point of reference for what 'about right' looks like, your harness won't have it either, so build external validation points into it.
endorsed advice
Francesco's main takeaway for accountability mechanisms is to design few gates with very high signal, putting some gates in place but making sure those gates are very high signal.
Francesco recommends including on-purpose gates in any harness you design so you can still be accountable for what is produced.
Francesco recommends designing harnesses so you review tasks instead of individual changes, noting the design is intentional. He describes his own shift: you have a set of tasks, you review the task, let it do its own thing, and then review at the end.
The hosts recommend that when presenting a piece of analysis or doing financial analysis on a potential investment, you need to show where your numbers came from and be able to attribute your decision to the body of evidence you had.
The hosts recommend data labeling and making sure you have a curated data set for either training something or running any process, calling it paramount, and say you need to assign the right level of importance to the evidence you're gathering and gather as much of it as you can.
Francesco recommends having evidence to back up where decisions are coming from, what information is coming through, and whether things are being tested, and putting that front and center.
Francesco recommends designing a harness that gives you accountability by making sure you can trace all the decisions that have been made and who made them.
Asking a machine to follow all instructions and make no mistakes will probably work 99% of the time, but the 1% edge cases are the real risk. Determinism via back-end code covers these out of the box.
When a calculation needs to be repeated reliably, convert it from agent-driven to a script. Letting the agent decide categorization, aggregates, refunds, and expenses yields slightly different results every time and wastes money and time.
When reviewing agent work, review the set of tasks the agent plans rather than approving every individual change; let the agent do its own thing and review the results at the end. Harness design should be intentional about this.
If you don't have a way to validate, you end up checking everything, checking nothing, or having no sense of what about-right looks like — and if you don't have that point of reference, your harness won't have it either. You need to build external validation points for your harness. For verification, add notes in the decision — an external point of reference added to some of the stuff the model thought.
Submitted documents like bank statements are facts; agent reasoning or information pulled from memory must be treated as assumptions; unstructured thoughts in chat must be labeled as interpretations.
The boundary between a skill and a separate tool is when the use case becomes critical enough that the trade-offs of the non-deterministic approach are not acceptable. Felippe describes the prompt-based (skill) approach as: 'Hey, please follow all of this and make no mistakes. Please, please, please.' Prompt-based approaches work because agents are good enough nowadays and will probably perform your instructions 99% of the time — but the question is what about that 1%.
Because you have code that is tested, you have a trail you can follow backwards for evidence. Tests run in your environment with no agent thinking involved, and skipping them sabotages the analysis.
Build a harness when you feel the use case is worth enough — when you want reproducibility, or when the results you are getting using pre-built harnesses are very frequently getting it wrong. There's no clear answer to how generic or specific a harness should be or how often to build one — it comes down to use case value, number of users, and problem criticality. The decision on whether to build a harness comes down to how many users you believe will go through that use case and how critical the problem is that you're seeing.
Evidence goes into the code itself: track the provenance and the method behind every number, not just its source. Gather as much evidence as you can and assign the right level of importance to each piece, the same way you would curate a dataset for training.
Treating all information the same without distinguishing facts from estimates leads to fatal errors in financial analysis, because a single wrong assumption can invalidate the whole model. Enforce the hard-truths-vs-estimations distinction through harness design.
When chatting with an agent for financial purposes, you may reason about something after a decision has been made, and you are well in your right to correct information and add more context. When creating a harness, build not only gates for validating things and staying accountable, but also for correcting the course and making sure you can do so safely. Without course-correction support, typing one sentence in the chat can wipe out your whole analysis — the speaker says this has happened many times, including while dictating to their phone.
A good way to showcase that you need to design a harness that gives you accountability is making sure you can trace all the decisions that have been made, who made them, and whether it was an agent assumption you let carry on or something you provided a revised input to.
No published protocol matches this selection.
The creators behind most of what cleared.
344 cleared insights match
71 cleared insights match
Choose sources → to get a library of your own and a daily brief.
