33,353 results · Most relevant
Kent advises not turning feature requests directly into implementation — including not building an app that takes a user's request, sends it to an agent, and automatically implements it, which he says would produce a terrible product with no vision, taste, or judgment. Each feature request requires your product judgment, and requests shouldn't be nailed onto the product you already have.
Most real applications use a collection of code evals, LLM-as-a-judge, and human evaluation together. Use code evals when the answer is deterministic (format validation, length limits, forbidden phrases, required fields), LLM judges when you need semantic understanding, and keep humans in the loop for unseen failure modes and to verify judges are judging correctly.
Before writing any eval, read your traces and categorize what went wrong. Figure out what to measure first. Categorize eval explanations into types of failures and count up the categories — this is called coding. If you see the same explanation across 50 different traces, you have a systematic problem, not a one-off edge case, which makes it the first thing you should fix.
The speaker advised reading a dozen or more traces end-to-end — starting with the ones where something went wrong — before writing a single evaluator, asking what the input was, what the output was, and what specifically is broken.
Use a code evaluator when looking for a specific string in output (no LLM needed to check whether the ticker was mentioned). Use code evals for production-critical checks like output-is-JSON, length limits, and avoiding forbidden phrases such as 'as an AI language model'.
A demonstrated workflow: connect the MCP server to your agent and create a skill that tells it when to store memories; store the conversation in a named data pack and record incident facts as searchable memories; kill the session and resume it later from a brand-new chat window without being inside a project or enabling a special memory mode; ask the same question to verify the agent pulls prior context back out of memory. When resuming, the agent searches memory first instead of creating a new conversation, and the MCP server instructs it to create a new conversation only for storing new content.
endorsed
Harald Kirschner recommends getting your codebase as agent-ready — a no-brainer that everybody should already be doing. Create agents.md files that are lightweight enough and give agents a good map of the codebase — where to look at, mapping that out. These are living documents that should evolve as the agent makes mistakes and works. Teams that invested in developer experience gain a lot because good documentation and onboarding for a codebase is read by the agent as well and helps the agent as much.
Harald Kirschner recommends creating skills that encode expert knowledge — like VS Code infusing all accessibility best practices into a skill everybody uses. Previously accessibility required one person to get pulled in or give feedback; now that skill is reviewed and maintained by the area owner of accessibility to bring that expertise into everybody's hands. Think how you can encode your expert knowledge to unlock the few users who might always be asked for this input into your skill.
Harald Kirschner recommends that when shipping an agentic product, it's important to have your own evals of the product always at hand and easy to extend. VSC bench is VS Code's own eval harness for their agentic product. When a GitHub issue is filed to add a new eval scenario, an agent picks it up from a template and bootstraps the scenario for you. Bring developer scenarios from issues and customer conversations back into your evals so you can start hill climbing against them. Confirm improvements offline first, then run online experimentation — a classic product lifecycle.
When the authentication workflow is stable and reusable across business operations on the same portal, pull that logic out from the responsibility of the model and reuse the same authentication mechanism for many business operations. Leaving authentication in the model's responsibility forces it to re-derive a never-changing workflow on every run, adding unnecessary steps and preventing reuse.
: none
Meegan recommends giving the agent an opportunity to verify its work in real time via an OCR tool that deterministically extracts entities from the document and compares them against the system of record — a deterministic mechanism so the agent knows authoritatively whether it did its job right. Deploying without such a mechanism leaves the agent unable to know whether its trajectory succeeded or failed.
: none
When a task involves programmatically interacting with the page — e.g., downloading a file and retrieving it from storage into the agent's runtime — encapsulate the operation into a single tool call that performs download and retrieval in one transaction, instead of the model making decisions on the fly. Leaving such decisions to the model increases steps, raising unit costs and reducing performance and maintainability.
: none
Meegan recommends drafting a skill, structured like a standard operating procedure for humans but for an agent, that outlines the critical path. Introducing the critical path makes the agent more likely to take the correct next step (e.g., knowing it's at step two and that step three is next) because ambiguity is removed from its decision process. Running without a skill leaves ambiguity, making the correct next step less likely.
: none
Meegan recommends measuring agent correctness on a per-transaction basis rather than per-run, permitting retries (e.g., up to four). A hypothetical agent with a 50% per-run success rate and up to four retries climbs to a 94% per-transaction success rate. Ground the metric in the customer: customers don't care how many retries you perform, only that the workflow completed successfully and reliably.
: retry count is an example; 94% figure is a hypothetical calculation
Invest in clear documentation, modular code, reliable CI, and non-flaky tests. What is good for humans is also good for AI agents. Assess platform readiness to improve token efficiency and context provision.
Anthony demonstrates a step-by-step procedure: create a new MakeFile, open settings, select 'Create a Kit', choose an NPM package, click 'Start Installing', optionally add custom configurations, then test and publish the kit.
Anthony demonstrates testing a MakeKit by typing a test prompt, then publishing it by clicking 'Publish Kit' and providing a name, scope, and version.
Anthony recommends a procedure for writing guidelines: explain the setup assuming the AI has no knowledge, create categories to avoid overwhelming the AI, include actual examples, and call out common mistakes.
Anthony recommends a procedure for building templates: lock down critical elements as hard-coded constants, constrain the edit surface with predefined prompts, add complexity incrementally, and calibrate against the model.
Stop using Salesforce as your primary database for pipeline intelligence; push all data into a data warehouse like Snowflake with a semantic layer so it is agent-accessible. Clean your CRM data before layering on AI and put governance around accounts, contacts, and opportunities so AI systems analyze reality instead of CRM slop.
Choose sources → to get a library of your own and a daily brief.