Check against golden examples rather than trusting the process
If you're getting golden examples of how the data should look, check with your golden examples rather than letting your process give you something different.
35,866 results · Most relevant
If you're getting golden examples of how the data should look, check with your golden examples rather than letting your process give you something different.
If there's a non-business-critical test you're solving, you're probably fine to have a human in the loop reviewing the outputs and steering the agent, even if that takes a few extra turns. If you're putting something in the hands of an internal user — for instance, 50 credit analysts using something to support key company decisions — you absolutely don't want a lot of variability on that.
Bad decisions carry forward into your other decisions. The gates you design need to be fit for purpose; you need to know the use case really well to put the gates where it matters. Adding a blanket 'before any decision ask me' gate makes the mechanism very low signal again.
The presenters recommend building verification into the system with a separate point of view or another perspective; in data and finance you often need a separate point of reference — a counterfactual — to check against, and you should verify things are going in the right direction, not just that things look correct. Ignoring verification leads to running things on your own biases, so results will always be skewed.
For code, you don't necessarily need to ask before decisions because the tests will speak for themselves; check that everything is passing and you will have a lot of confidence in the output. You can be hand-wavy with reviews prior to functional tests passing, but never push anything into production that hasn't passed all your tests.
When the model assumes a decision, verify with it by asking whether that's the number it wants to use. You are well within your right to correct information and add more context after a decision has been made.
The trust users have in your product, your agent, and its output is what will drive usage, adoption, and improvement through the loops mentioned — tracing and evolutions. When building a harness, make sure you can make trust happen for customers so they trust the results the agent gives them.
When a data person says 'trust me, I ran the numbers' without showing their work, the correct response is to not trust it.
Evidence is about the receipts — being able to show that this is where this came from, this is correct because of, and doing that causality relationship. It builds on the second principle: since you have code that is tested, as well as everything else, you have a trail that you can follow backwards — a way to substantiate the decisions you are making.
When you have a few possible values for something, ask the agent to do three scenarios and review them side by side before making any decision.
When you ask an agent where a number came from, ask specifically for the paper trail rather than accepting a verbal explanation; make 'where did this number come from?' a standard check in every conversation.
Open a spreadsheet and look at a few cells you know will give you a ballpark total. If it doesn't feel right, say so. When something looks plausible but doesn't feel right, look into the calculations behind it.
If a decision influences the final decision, it definitely comes with accountability. Design a harness that gives you accountability by making sure you can trace all the decisions that have been made and who made them.
Grubbs frames sprinting as a pyramid with a baseline of strength at the bottom: athletes need a certain strength level and rate of force development to sprint well. For elite athletes, specific transfer exercises like speed bounds and scissor runs come into play.
her coaching model, not a formal study
Grubbs raises the concept of 'how strong is strong enough' — for a track sprinter and a baseball pitcher alike, there's a point where more strength stops helping. For a pitcher, there's a limit to how much upper body strength is useful before it becomes a deficit. She advises against hard numeric caps (e.g., 'you can only dumbbell bench 100 pounds'), instead using the coaching eye to judge when strength is enough.
her coaching judgment, explicitly not a numeric rule
Because professional baseball teams play every day, players must lift on game day — otherwise they would never lift across the 162-game season.
In professional baseball, every day is game day — there are no practice days like other sports. Lifting on game day is out of necessity: if you don't lift on game day, you won't lift for 162 games.
We won't be able to reason about massive large language models; our best option is securing the boundaries — a better option than trying to prove a 500 billion parameter model won't do something crazy.
Clark explains that proving code memory safe makes buffer overflow attacks 100% impossible, and that memory safety is a universal spec that everyone writing C code should prove. He notes buffer overflow remains the number one security vulnerability.
Turning everything over to computers with nobody in the loop means that when something weird happens, it can go off the rails — as in the movie WarGames. The guest concludes that we do need humans in the loop for AI systems.
Choose sources → to get a library of your own and a daily brief.