35,800 results · Most relevant
Charlie Guo argues that assuming a human must be in the loop for every step (like paging for an API key) blocks automation of the whole pipeline. He recommends identifying which steps genuinely need humans and automating everything else to unblock the process.
Charlie Guo recommends setting up a Codex Goal with success criteria that are as verifiable as possible, stressing that being really specific with the criteria is one of the biggest things he tells first-time users. He warns that vague criteria lead to a monkey's-paw outcome where the model technically completes the task but not in the way you expected, and suggests treating Goal criteria like a genie's wish and wording them carefully.
Based on personal experience.
Gabriel Chua describes a feedback-loop design: run a second automation that cross-validates drafted email replies against the eventual reply you actually sent (the ground truth), and have it update a Markdown file with best practices and tips so the email drafting automation improves over time.
Drawing on the FBI ANOM case, Lyon advises not assuming a 'secure' messaging app is safe just because it's encrypted or exclusive — verify who owns and operates it, since it could be a state-run honeypot.
Kent C. Dodds recommends closing the loop with user feedback to know whether your solution actually solves the problem effectively, and getting a prototype to early users — noting that early adopters are totally fine even if the solution has gaping holes.
Kent recommends that builders validating ideas find out what users do instead — the workaround — and how much it costs them. He also advises finding out how often the problem happens to them and how much effort they're already putting into solving it. These questions reveal whether the problem is worth solving.
When asked whether he talks to random girls often, Coach Kyle avoids a defensive denial and instead answers honestly about his circumstances (just arrived, exploring, doesn't know many people) and tells a playful backstory — his first concern being that she might not speak English — to display romantic intent and pass the test.
In AX, click annotate span on a span to add or remove annotation configs and set it as actionable or not actionable. Use the annotation UI's arrows to flip through all traces and label each as actionable or not actionable. Adding human annotations builds a golden data set, which is how you measure whether your evaluator is actually doing its job.
Take your eval explanations and categorize them into types of failures, then count up the categories — this is called coding.
A rubric is your set of rules that you put in your prompts about what counts as good. This is how you define quality for LLM judges.
Filter eval results in AX with expressions like eval.actionability.label equals not actionable to find the failing set. Turn the failing eval set into a data set and run new prompts and experiments against it.
Use code evals when the answer is deterministic — such as format validation (does the output parse as JSON?), length limits (is the output less than 500 tokens?), forbidden phrases, and required fields (did it mention the thing you asked about?).
Treat eval prompts like code: they are highly sensitive to wording, so version them and test them on examples where you know the right answer. Small shifts in eval prompt wording can lead to dramatic changes in eval results. Use the AX prompt playground to iterate on eval rubrics without touching your code.
Label your rubric data with XML tags, especially if you are using Anthropic. Put a user query tag around the input and a financial report tag around the output to create clear boundaries. XML boundaries reduce the chance that the judge confuses the question with the answer.
The speaker advised using synthetic data when you have no real traffic — having an LLM generate diverse queries across expected categories — and switching to production data as soon as it exists, since test data should look like production data, not what you wish production looked like.
Ask the judge to reason before it scores and have it explain its thinking first — chain of thought for judges. Chain of thought measurably improves judge accuracy. When the judge has to articulate why something is actionable or correct, it makes fewer mistakes than when it just picks a label without thinking first. The judge's explanation is also very helpful for you, in addition to helping the judge think better.
Use built-in evals as a starting point because they give an immediate signal without prompt engineering; when no built-in template checks your use case (e.g., actionability of recommendations), build a custom eval specific to your domain.
Laurie Voss explains that when running evals, you pull the input and output attributes out of the spans and feed them to the evaluators: the input becomes the evaluator's input and the output is what gets graded.
Test whether your LLM judges are judging correctly through meta-evaluation, and calibrate them against human judgment because they can be wrong.
In AX you can create an online evaluator and create a new LLM-as-a-judge via the UI. For an LLM-as-a-judge in the UI, you have to set up an LLM for it to use, which means giving it your Anthropic key again. Pick your inputs and outputs via the single trace method, then save the evaluator. You can set online evals to run continuously against new incoming traces — something you will almost certainly do in production.
Choose sources → to get a library of your own and a daily brief.