417 results · Most relevant
An edge case can make things not work when distributing prompt-based approaches. If you distribute a prompt-based agent to internal users, a user's instruction in their user-level claude.md file can go against what you're telling the agent to do. When you start distributing agents to a number of users and use cases, things can get messy in places you are not able to predict.
Give the agent versions of the code so it can surface when the code changed.
A model's judgment that something 'looks about right' may look nothing like yours — and it turns out your company does care about that 5% difference, which makes a world of a difference. When you're building a harness for your team, be aware that a 5% difference may be a deal breaker for the whole team.
For non-business-critical use cases, you're probably fine having a human in the loop reviewing outputs and steering the agent even with extra turns.
endorsed with hedging
When you don't trust that a question won't contaminate what you're doing, open a separate chat and say 'hey, this is a stupid question. Please don't use it for anything else.'
When distributing agents to internal users, a user's own cloud.md file may contain instructions that go against what you are telling the agent to do, causing unpredictable failures. Teams frequently write instructions trying to cover edge cases.
If you click on the underlined figure that changed, it should show you all the way back where this came from and why it matters for the rest of your calculation; you can also click to see which parts of the flow are proposed by the agent.
Typing a sentence in the chat should not cause your whole analysis to disappear — the speaker recounts this happening many times, including while dictating to a phone when distracted, where one mid-task sentence wipes out the analysis and can't be taken back.
The demo UI annotates whether plan elements came from the user, from the agent, or are still open points requiring a decision. Users can click a step to get a breakdown of where information came from, why the model thinks a decision is needed there, and whether it is assuming things from the user's language — and can go back into a step, tell the agent its assumptions are incorrect, and iterate. The presenter showed that his vague language led the agent to make an assumption from a keyword.
Having accountability mechanisms in harnesses greatly reduces the stress of wondering whether the agent made a wrong assumption; keeping accountability high requires being able to see inside the black box.
The evidence trail builds on the testing principle — since you have code that is tested, you have a trail you can follow backwards, which substantiates the decisions you're making.
Verification is just extra information from a different source to avoid biases and avoid getting stuck on your own thinking, but it doesn't remove any of the responsibility or accountability.
In spec-driven development for coding agents, you have something that is the source of definition for the thing being built — the same principle applies to the harness.
When building a harness for a specific use case, the code is a means to an end — the agent helps analyze and solve a problem, unlike coding assistants where you build a product.
Asking an AI to build something big without knowing the principles and decisions behind it leaves you relying on the randomness of something good happening by accident.
If there's no feedback loop, the financial decision (or any decision) doesn't improve at all.
Trust the agent to the same degree you trust a dashboard, metric, or visualization — that is when you've reached alignment with it. There won't be a single thing that earns user trust; the process covers a multitude of possibilities.
Premises, source data, and context will change — the harness should be able to adapt to that rather than having to prompt again and go back to the process every time. It can continue guiding you into alignment with the agent, keeping it true over time.
Relying on an ever-changing spec means relying on the non-determinism of the agent to interpret it, which can be fragile. Metric and dimension definitions always change, so plan for it.
You can easily fall victim to doing a lot of axe sharpening (building harnesses/tools) without actually using the axe.
No published protocol matches this selection.
The creators behind most of what cleared.
344 cleared insights match
71 cleared insights match
Choose sources → to get a library of your own and a daily brief.
