417 results · Most relevant
The speakers generalize the master-file failure: if you don't have traceability even for changes you make in your own code, you leave yourself vulnerable to blind spots when the process changes. The same failure mode applies to deterministic code built with your harness.
The hosts explain that when you have code that is tested, you have a trail that you can follow backwards to substantiate the decisions you're making.
In agentic building it is very easy to assume something or treat something as a guesstimate, and the model cannot tell what is a hard truth and what is an estimation. You need to enforce the distinction between hard truths and estimations through the design of your harness.
When you give an agent an example alongside a modification request, the model fixates on the example so hard that it puts it everywhere in the output — which you may not catch in conversation.
The hosts observe that in agent building it's very easy to assume something or treat something as a guesstimate, and the model can't tell what is a hard truth and what is an estimation.
Users can add a curveball to the agent by typing in the chat; the harness traces that a changed input came from the user correcting a fact. Clicking on an underlined figure that changed shows all the way back where it came from and why it matters for the rest of the calculation.
The hosts warn that if you treat everything as a fact, you will make mistakes because you won't know where the information came from.
Francesco argues that ignoring verification and running things without a counterfactual leads to you running things on your own biases, so your results are always going to be skewed.
You should build trust with your agent when building a harness like this by making sure the information you care about regarding its implementation is in front of you. One example in the wild: giving the agent versions of the code — your code is more likely versioned. If anything changed with the version and you're trying to run the same process, the agent should pick it up and surface that the code changed; even though the function is the same, if the code version changed you probably shouldn't trust what's coming out of it because the calculation is different now.
A lack of a trace of consequence can really hurt the trust you have in your harness and the results it produces. Accountability builds on evidence, because you only make a decision when you have some facts and some point of reference.
You want to reduce ambiguity, not remove it. Many people remember fondly the days prior to automode when they had to approve every single change by tapping enter. The speaker still reviews changes manually because if it breaks, he could lose it. An audience member used to pre-approve a lot but now reviews a set of tasks and lets the agent do its own thing, reviewing at the end. The harness design is intentional: you review tasks instead of individual changes.
If you have golden examples of how the data should look, check with them rather than letting your process give you something different.
endorsed advice
Set up a feedback loop where you tell the model from your experience if its output looks entirely off or looks reasonable.
endorsed advice
Work with SMEs, the client team, or domain experts to understand the best set of tools and information for agents to get to the best answer.
endorsed advice
Francesco recommends designing with the intent that some things really need to be an aside — like having too many stupid questions when chatting with AI. He ends up opening a separate chat when he doesn't trust that a question won't contaminate what he's doing, saying 'hey, this is a stupid question. Please don't use it for anything else.'
Clone the workshop repo via QR code or link and follow setup instructions. If setup succeeds you should see the UI open and 'success' in the terminal. Running the second command opens a UI for doing stuff with the harness. To switch from the wedding example to your own use case, delete the database in the my/var folder and reopen the UI with the previous command.
Francesco gives breaking tasks down in a way that you can reason about the issue and the way they connect as an example of good gate design.
When creating a harness, make sure you can correct the course safely so that typing one sentence doesn't wipe out your whole analysis.
endorsed advice
Start with a chat describing the problem: a wedding with three payments, fixed costs like a DJ and decorations, unknown attendee count, and per-seat dinner costs. Add clarifications like no savings, money from pocket, mom may give 5K, and price per person around $150. When you accept the plan, you'll get a prompt asking 'do you want me to build the code?' For the wedding example, click on step one, then code, then the test, and run the tests now.
Francesco demonstrates continuing the prompt with more context — no savings, money coming from his pocket, his mom possibly giving 5K, and a price of around $150 per person — and recommends keeping adding clarifying information to the chat as you think of it.
No published protocol matches this selection.
The creators behind most of what cleared.
344 cleared insights match
71 cleared insights match
Choose sources → to get a library of your own and a daily brief.
