36,915 results · Most relevant
Kent C. Dodds warns that shipping reverse-category features causes users to hate your product — it's the sort of thing you just really don't want in your product at all. He advises pushing back on stupid product ideas that are making users' lives awful and taking a stand and refusing to build features that make users' lives worse, adding that you should have a little respect for yourself even when you're not self-employed.
Delighters are like experiments — even if you think something is core to your identity, a delighter is not a basic, and you need to make sure you're covering your bases on the basics. For each experiment, decide how much you want to invest in making it solid and reliable, given there is only so much time and so many tokens. For delighter experiments, take a little less commitment from an architectural standpoint. Instead of integrating an experiment into the existing system, build it in a separate system where you can do a lot of learning, then find a way to bring it back into the full system once you have learned its shape.
Kent C. Dodds advises that even shaped requests must be filtered, because every accepted feature impacts the system design you must maintain long term and slows down more important work. A request may solve a problem that seemed good to one person but wasn't generally applicable — verify broad applicability before building.
Kent suggests tracking how often sessions start on time as a metric for whether the check-in solution solves the ultimate problem.
Indifference-category features are things the user decides don't make a difference — whether you have them or not, they don't care. Do not make an indifference-category feature a permanent part of your product. Get indifference-category features out of your product as quickly as possible because they are dead weight you have to maintain as an engineer. Maintaining an unused feature wastes engineering time — e.g., spending two weeks maintaining a feature nobody uses. Go to your boss or product manager and ask whether anybody is using a feature before spending time migrating or maintaining it. You need to understand the Kano model so you can make the case to your boss or product manager about which features are worth maintaining.
Kent relays Jack Ryan's suggestion to set up a Slack channel or similar mechanism to collect all user feedback in one place — for example a channel with high-value customers whose decisions you really respect, or one that aggregates Reddit threads, X, and other social media into one place. You don't have to answer everything in it; that's not necessarily your job.
Install the open inference instrumentation Claude-agent SDK package to auto-capture traces from Claude Agents without changing your application code. AX is compatible with OpenAI's agent SDK and whole agent frameworks like CrewAI, LangChain, and Mastra — no matter what framework you built your agent in, it is already instrumented and you can just turn on logging. You can also use OpenAI instead of Claude or some framework — packages exist for all of those and you just need to swap them in.
Laurie Voss recommends writing one code eval next, since code evals are easier, faster, and cheaper than LLM evals, and making it for the thing that matters most: correctness, faithfulness, whatever fits your use case. Then run your evals, look at the results, see what patterns emerge, and build from there.
You don't have to do all of it at once: start by adding two lines of instrumentation and reading your traces for immediate insight, then write one code eval for the thing that matters most.
Laurie Voss advises that you don't have to do all of it at once; you can start very, very small. Start by reading your traces: add those two lines of code and start looking at them — that immediately gives you information about what your agent is up to that you didn't know. 15 minutes of reading real outputs will teach you more about your application than an hour of building dashboards.
Vary the phrasing, complexity, and level of specificity in synthetic test queries — 'research Tesla financial report performance', 'What's going on with Tesla stock', and 'Yo, is Tesla a buy right now' are the same intent with very different looks.
The speaker advised generating multiple phrasings of the same intent and varying complexity and specificity — e.g., 'research Tesla financial report performance', 'What's going on with Tesla stock', 'Yo, is Tesla a buy right now' — to match the range of what users will actually ask.
Laurie Voss notes that LLM judges cost money and at production scale those costs add up, so you don't need to run evals on every production trace — sample 10% or 1% of traffic to get a directional signal without running an expensive LLM judge on every instance.
If you only test positive cases (does it search the web), you end up with an agent that always performs the behavior — also test 'did it not search the web for this very obvious answer where it didn't need to'.
Failing to label examples with XML tags causes the LLM to treat the example as the thing to write and spit it back verbatim. Telling the judge to spit out a specific string like 'correct' or 'incorrect' is more likely to confuse it and make the eval fail — the judge instead calls a tool with correct or incorrect when done.
Laurie Voss recommends telling the coding agent to find themes and not chase individual failures: it's tempting for Claude Code to write a whole bunch of code to fix one particular failed trace when there are actually 10 failing traces with the same cause it should focus on instead.
Laurie Voss recommends using one model provider (e.g., OpenAI) for the agent and a different one (e.g., Anthropic) for judging, so the judge is less likely to agree with the generator due to self-preference bias. He notes he used Sonnet as a judge for Haiku-generated output for this reason, and that using a different lab's model entirely works even better.
Laurie Voss recommends running evals at the span level to evaluate a particular LLM or tool call every time, or at the trace level where the evaluator sees the full tree from start to finish — pick the scope that matches the question you're asking.
Hand a whole pile of eval explanations to a coding agent and ask it what it could do to improve — it will improve the application for the next run. You can also get your coding agent to do everything you do by hand today using the Arize Skills.
Trace voice agents and see each individual span with live audio playable in line from the trace. Attach metrics to voice sessions: latency, time to first audio, interruptions, and sentiment analysis. When debugging, look at inputs and outputs including the transcript plus specific metrics like time to first token, audio token costs, and interruption events. Don't view trace data as a flat list — look at the linkage between spans and events, and stop looking at disparate events from distant systems; look at the entire trace tree for everything that happened in the session.
Choose sources → to get a library of your own and a daily brief.