Talk to non-technical stakeholders to define 'good'
Engineers have biases about what makes a good agent and won't always agree with domain experts; talk to non-technical stakeholders because they know what good looks like.
40,505 results · Most relevant
Engineers have biases about what makes a good agent and won't always agree with domain experts; talk to non-technical stakeholders because they know what good looks like.
Laurie Voss warns that running your whole agent on every commit wastes a lot of time and money, and that alerting on every single eval failure produces noise from one-off failures.
Design eval tests to be very flexible because agents may solve problems in unexpected ways.
Laurie Voss describes several things code evals can do: query your database to verify state is correct, call an API to double-check that the stock price the agent thinks is happening is really the stock price, and make sure the tool calls did what the tool calls were supposed to do.
Prioritize off-season training as the window to build a strength foundation before the season starts. If athletes do nothing all off-season, they build no foundation of strength — off-season training is critical even though practitioners don't have access to them then. During the season, focus on educating athletes repeatedly on what's important since you can't reach them in the off-season.
Offer professional athletes menu options for their lifting layout (total body vs upper-lower split, before or after game) rather than one prescribed plan. Have everyday position players get one total body lift in per series, and stack all training stress on one day when an off day follows — lift after a Sunday day game knowing Monday is off.
Just as AI systems should be continuously monitored, benchmarks should be continuously monitored too. There are quantitative methods for identifying distribution shift, and monitoring shift in language is something the field needs to do.
Open source models are happy for you to run and distill on them, so you can use the best open source models for dataset generation.
A good way around the reviewer bottleneck is to instill better testing practices so output can be automated and filtered out.
When deciding how many agents to spin up, use evals and lower defaults to determine the appropriate scale; spin up a higher level of agent parallelization only when pushing on harder problems like Navier-Stokes.
Bring your own system prompts per agent, or have Claude spin them up, to enforce totally separate personas.
Share code layers with the team to collaborate and try out new ideas; teammates' exploration layers on the canvas show other directions being worked on.
When the agent generates variations, evaluate each against the original ask; treat off-mark ones as fun explorations but return to the one you were happy with and riff on it by duplicating its layer and exploring different ideas from that copy.
Build a plugin by first asking the agent to make it and then editing with the agent to make it better; AI credits are only used to make the plugin, not to run it afterwards.
Modes can be set up as different brands. Modes can be set up as typographic scales for light mode and dark mode. Set up two or three different font pairings as separate modes. Create three different modes to experiment with different typefaces before making a decision. Use different modes to make a type style more flexible. Typically you might experiment on the canvas before you go making the variables, or build one set and test another.
Sometimes you really do want local development for quick iterative back-and-forth with an agent while looking at your local copy of the codebase. Sometimes you really just want to fire off events that let agents run autonomously in the background for a few hours — then run agents in Kubernetes or a VM. Pick the right sandbox environment for the job and move between different environment types as needs change.
Hormozi advises ignoring people who don't have what you want telling you how to get something they've never gotten.
In multi-tenant CMA deployments, implement data segregation per workspace.
For long-running tasks, define a rubric and let Claude mercilessly iterate until the output matches the rubric.
Recognize people-pleasing bias in models: the model agrees to everything even when it's wrong, because it's trained to be nice, and sometimes makes things up. Maintain security consciousness yourself — the AI won't do it for you, so ensure every prompt includes the safety measures covered.
Choose sources → to get a library of your own and a daily brief.