35,595 results · Most relevant
A method for AI-assisted agent development: work with coding agents to automate instrumenting, finding failure modes, running evals, and generating many experiment candidates (e.g., 20 instead of one), but with structured evals rather than open-ended 'fix everything' prompts — and install the Langfuse skill (markdown agent skill files) as the starting point.
Annabell recommends that for some things you should get your taste in — get your domain experts in the room to build out the datasets, and to curate what actually is the problem that your application should be solving and which direction you want it to go.
Doneyli recommends keeping domain experts in the loop working with development teams to define datasets and quality criteria, because human judgment remains essential to agent evaluation.
You cannot control what your users input into the system, so you must catch PII (like credit card numbers in clear text) during instrumentation and eval building. Enable PII identification so any data deemed PII is automatically masked before it gets persisted into the observability system, and use a masking function that automatically masks any defined PII so it is not persisted in plain text. In the demo, a user inputting a credit card number in clear text shows the card redacted in the trace before persistence.
Teams shipping agents must continuously mine for signals where things are going wrong and make sure they don't happen again. Monitor aggregate metrics on what is going on and detect signals where things are going wrong. Monitoring should include finding the interesting traces you want to dig deeper into, not just aggregate metrics. Track quality signals such as compliance guardrails, truthfulness, and how many users are frustrated (e.g., writing all-caps messages).
endorsed
Annabell recommends being in favor of working with your coding agents to continuously improve your applications — but warns against two extremes. Don't just ask your coding agent to 'fix my agent, make no mistakes', which produces slop rather than anything helpful; if you fully automate agent improvement and just let the agent go for it, you will end up with a low-quality system and unsatisfied customers. But also don't go the full manual way anymore — don't instrument yourself, catch all failure modes yourself, or review all the data yourself; these can be automated.
Run the process manually in parallel with the agent (shadow runs) to compare how the agent handled it, measure accuracy, and build ground truth. Start with a fully human-in-the-loop system, then over time slice the data into parts you're confident to automate versus things where you always want a human in the loop.
Evan Kormos recommends defining what good looks like with domain subject matter experts before moving an agent toward unsupervised operation.
Diamond advises: do what you enjoy (perhaps the main advice, since most people don't), but it's got to be broader than that; if people tell you that you've got to live your life their way — iron your shirts, do this, this, this — tell them to go to hell.
When someone makes a serious mistake that could end their career, forgive them if you were forgiven for your own close calls; blaming others harshly ignores that you have made plenty of mistakes and close calls yourself. When you make a career-threatening mistake, recognize it as a close call and extend the same forgiveness to others; when a mentor forgives you, use the second chance to develop your career fully rather than repeating the error.
Ask peers and cross-functional leaders 'what am I not thinking of?' to surface hidden blockers like obscure procurement requirements before they kill your deal. Learn about infosec reviews and other cross-functional requirements proactively when moving from SMB to commercial/enterprise — ask internal partners 'why won't this deal get done?'
When your champion clearly lacks political clout, acknowledge their frustration, suggest the decision-maker would have better answers, and propose adding that person and others to the next call. Use the champion's 'I don't know' rate as a temperature check for whether they have real influence or will just pass everything to their boss. When a weak champion just relays everything to their boss, you end up with a closed-lost deal and the champion telling you to go back higher — so get the real decision-maker in early.
When a buyer mentions taking the deal to a board, ask whether they will be presenting, asking, or telling — Sam McKenna says this is a key differentiator in how the deal moves forward, and it opens the conversation for the buyer to be honest about what is really going on.
Naidu's recommendation to the audience: instead of selling an AI agent from outside or building a firm from scratch, consider partnering far more closely with incredible businesses that already exist — through acquisition.
The biggest opportunity for most SaaS founders is to move up into labor markets, because most businesses spend significantly more money on salaries and people than they do on software.
The speakers walk through a verification sequence: run the doctor command, add an email, check storage, reload, then snapshot to verify persistence state — the tool loop can confirm before reporting back.
Consequential creator advice requires independent verification before Today can promote it.
For their midnight snack page, the proof command uses Playwright to check an email was actually saved to the waitlist: check no emails, fill an email, click the join waitlist button, then expect the status element to contain 'waitlisted' and the email address.
An agent cannot verify anything from a terminal alone; it needs tools to check the network tab, read page contents, interact with the application, check API response codes, and verify local storage persistence.
Methods that look fine on five or six conversations diverge significantly at millions of conversations, so Martone recommends testing AI systems at production scale and building evals to understand the impact of decisions across millions of daily conversations.
endorsed advice
Laurie Voss recommends choosing a judge model that is more capable than the agent itself, and even better, a model from a different provider entirely — e.g., having OpenAI judge the work of Claude.
Choose sources → to get a library of your own and a daily brief.