Method · MoneyJesse Hall lays out a benchmarking method: take a complete working agent with a real backend as the test subject, put simulated callers with goals against it who can change their mind and add background noise, define scenarios as what the caller wants, what actions the agent should take, and what the outcomes should be, score on whether the correct outcomes were met (e.g. the correct reservation hit the database, or the correct authorized amount got charged in a financial benchmark), verify the agent took the right steps (not 10 steps when 2 would do, and no hallucinated function calls), swap the models in the same agent with the same scenarios and run the loop again, ensure every model sees identical input byte for byte, and run the loop continuously to build a personalized benchmark for
Method, not a verified protocol.
AI Engineer · How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
Method · MoneyCreate two slides: slide one restates the customer's situation ('here's what I hear: you're trying to do X because of Y, you've considered options but they're lacking') and asks 'do I have that roughly right?'; slide two offers how you can help and asks if it fits. Look for 'oh my God, yes' on slide one and 'how do we move forward?' on slide two.
Method, not a verified protocol.
Lenny's Podcast · How to find product-market fit (but, actually) | Rob Snyder (Author, The Power of Pull)
Method · MoneySnyder structures the case study factory as three steps: pipeline (how people talk to us), sales (how potential customers become customers), and delivery (how customers become weird not to churn). He applies the theory of constraints from manufacturing to identify the single bottleneck to growth and solve only that.
Method, not a verified protocol.
Lenny's Podcast · How to find product-market fit (but, actually) | Rob Snyder (Author, The Power of Pull)
Method · MoneyIf you're just starting the eval process, create a blank eval and start writing your template. Turn your written criteria into a prompt — much of it can be pasted in as the bullets you wrote. Start the prompt with an intro like 'You are an eval agent, or an eval LLM, looking at the output accuracy for your application.' The speaker likes to start evals with an example of what the LLM is supposed to do and always centers it in its job. After the intro, provide the data — this is really important. Include the check-in particulars, such as the three bullet points for pass and fail. Example instruction: mark only correct if the output is consistent with the input; incorrect if there's a wrong destination or property type.
Method, not a verified protocol.
AI Engineer · How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI
Method · MoneyLoad in the dataset of saved spans so you have inputs and outputs to test the eval against. If needed, map the dataset's inputs and outputs to different variables in the eval template. If you created an eval in the eval hub, you can add it in the playground from the dropdown and it will pop up. Run the eval in the playground (e.g., with 4o1 mini) to see the output of the eval right there. The playground is a great spot to iterate on your eval template: look at the response, agree or disagree, adjust the prompt, and continue running it over time. Save the eval once the template is in a good spot, and it becomes available to use across the platform for your demo project or any application you bring into Arize.
Method, not a verified protocol.
AI Engineer · How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI
Method · MoneyWhen jotting down eval ideas, write down the task: what you want this eval to be going off of. Define the eval task first: what the eval judges, what success looks like, and what failure looks like. Write down what the eval is actually supposed to judge. Write three bullets on what pass would look like and what fail would look like. Sit down and think about what your success criteria is when building evals for your application. The notes you write will be used to actually build the eval in the next step.
Method, not a verified protocol.
AI Engineer · How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI
Method · MoneyThe overall first-pass eval workflow: filter the spans you're judging, look through traces, identify what span you need to evaluate, build your filter, and add those to a dataset. To evaluate input versus itinerary, save all of the root spans to a dataset. To test whether a tool call returns the right information, add all of those tool calls to your dataset. Filters are the best way to identify data: to filter for root spans, use parent ID equals null or go off the default traces; to filter for a specific tool call, filter by the tool's name. Then select spans from the span view and add them to a new dataset.
Method, not a verified protocol.
AI Engineer · How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI
Method · MoneySofia Jakovcevic describes the change workflow: evaluate production traces to get scores before implementing a change (prompt, model, tool, skill, or policy change), then test and iterate on changes with the experiments workflow before shipping. If a change passes, ship it; if it is negative, go back, test, and iterate. If a change fails, bring the failed cases back to the observability layer so new production traces that failed become new cases in the regression testing suite, building regression test suites that constantly build on themselves for a more robust method of continuously iterating and improving.
Method, not a verified protocol.
AI Engineer · Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI
Method · MoneySofia Jakovcevic lays out a five-step regression loop for agents: start with production traces, build evals, run experiments, and close it all up as an automatic gate at CI. The loop uses observability to build a robust regression process that feeds automatically into CI/CD.
Method, not a verified protocol.
AI Engineer · Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI
Method · MoneyFarza describes his company's standing question of 'what are we launching this week?' and working backwards from the launch. When building a feature, they discuss the launch copy — the Instagram caption and the video hook — as part of building it, not after, because how you communicate the product is just as important as what you're building.
Method, not a verified protocol.
Y Combinator · This Is How You Should Actually Make Launch Videos
Method · MoneyFarza Majeed shares his consistent launch video formula: this is who I am, this is what's going on, this is what I'm working on, let me show you.
Method, not a verified protocol.
Y Combinator · This Is How You Should Actually Make Launch Videos
Method · MoneySet up agents by creating a blueprint that acts as a ceiling of access for each agent (e.g., one for the read-only agent, one for the fund agent). When a user invokes an agent, the scopes in the issued agent token should be the intersection of the user's permissions and the scopes in the agent blueprint's ceiling. The agent then calls the application's API using a scoped agent token.
Method, not a verified protocol.
AI Engineer · How to Let AI Agents Sign Up for Your App Safely — Isabelle Williams, WorkOS
Method · MoneyIsabelle Williams explains that if the agent's provider has already verified the user's identity, the agent provider can tell your application who the user is instead of requiring manual signup. When an agent asks the user for consent to share their identity and email with your application, the agent requests an ID-JAG from the agent provider. The agent provider verifies the user's identity (e.g., verified email) before issuing the ID-JAG, so the application can trust the assertion. Your authorization server verifies the ID-JAG signature before issuing an access token to the agent, and your application decides whether to create an account for the ID-JAG user, link it to an existing account, and what scopes to grant — it is completely in control.
Method, not a verified protocol.
AI Engineer · How to Let AI Agents Sign Up for Your App Safely — Isabelle Williams, WorkOS
Method · MoneyIsabelle Williams recommends hosting a markdown file called auth.md with your application that describes how agents can register for accounts, discover how to get started, create an account, and call your API. The file starts with standard OAuth parlance: finding the protected resource, the API, the MCP server, and the OAuth authorization server. She emphasizes that the application itself, not the auth.md file, is what enforces auth at the API.
Method, not a verified protocol.
AI Engineer · How to Let AI Agents Sign Up for Your App Safely — Isabelle Williams, WorkOS
Method · MoneyThe agent discovers how to authenticate with your API using standard OAuth parlance, asks the user for consent to share their identity and email with the application, and if the user consents, goes to the agent provider to get an IDJAG. The agent sends the IDJAG to your API's authorization server, which verifies that it is legitimate. Your app is completely in control of deciding whether to create an account, link to an existing account, and what access to give back to the agent.
Method, not a verified protocol.
AI Engineer · How to Let AI Agents Sign Up for Your App Safely — Isabelle Williams, WorkOS
Method · MoneyIn the user claim flow, an agent can take anonymous actions at your API and receive an access token plus a claim token. The agent gives the user a link and a one-time code; the user signs up, signs in, and enters the code to claim the agent's actions under their account. Using OAuth, your authorization server can then issue an access token on behalf of that user to the agent.
Method, not a verified protocol.
AI Engineer · How to Let AI Agents Sign Up for Your App Safely — Isabelle Williams, WorkOS
Method · MoneyThe team gives each LLM judge around a thousand words of examples, instructions, and directions per metric, plus the script to analyze, their own script as guidance, and the brief and research — a 14,000-word prompt total. The judge first categorizes between six categories, then gives a rating within that category, and must return a score with a justification and a specific supporting example. The judge analyzes writing quality only (not length) and doesn't know which model wrote the script.
Method, not a verified protocol.
AI Engineer · Why AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI
Method · MoneyBouchard's team measures five things for each generated script: tone (does it sound like you and adapt to a specific voice), pure writing quality (hooks, emotional transitions, story aspects), instruction-following (expected length, format, images in the right place), research accuracy (hallucination check, correct use of research), and slop (EQBench slop score plus their own measurements).
Method, not a verified protocol.
AI Engineer · Why AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI
Method · MoneyGary Vaynerchuk describes his '1585' system: involve yourself in the first 15% of a project to ideate and contribute strategy, let the team completely own the next 80%, then come back in for the final 5% to add a finishing touch.
Method, not a verified protocol.
GaryVee · You Might Be Making Content Completely Wrong
Method · MoneyWhen a GitHub issue comes in, write a Prose program for how the agent will try to address it — as complex or simple as you want, including multi-agent workflows, specific skills, or specific models. Use pull requests for event-driven adversarial review of agent-built code, letting agents receive back pressure from the app itself. From the trigger you triage workflows, so you can build out code factories from creating standing jobs of the Prose programs. The agent running cannot consider its job complete until its developer agents have taken the reviewer feedback into account.
Method, not a verified protocol.
AI Engineer · Stop Telling Agents How — Dan Barrett, Raymond Weitekamp & Jose Montes de Oca, OpenProse
