Turn hard-won workflows into skills or plugins for reuse
At the end of a difficult Codex conversation, ask it to make the workflow a skill or plugin for reuse: 'hey, make this a skill, or even better, make this a plug-in.'
40,353 results · Most relevant
At the end of a difficult Codex conversation, ask it to make the workflow a skill or plugin for reuse: 'hey, make this a skill, or even better, make this a plug-in.'
Charlie Guo shows how to customize subagents: you can configure one as a code reviewer with a specific prompt and set of developer instructions, point it at a lightweight model so it runs quickly or a more intelligent model so it's very thorough (in its own subagents.toml file), and name subagents so you can instruct the model to check with them — 'go check with the code reviewer, go check with the docs researcher in order to implement.'
endorsed
Charlie Guo describes his personal automations: one that checks the latest updates from all sources that might need his attention (Slack, email, Linear notifications); a local Obsidian vault that an automation constantly updates with daily notes and notes on different work streams, which he can later query ('what's going on with X since the last time I checked in?'); a weekly automation recapping what all of his direct reports have been doing; and an automation to review all analytics from a system for a one-time look together, then archive the thread.
When Codex is cooking in the background and you're away, check in from your phone: 'Hey, Codex, is everything okay? Is there anything I can do to help unblock you?' — acting like a manager.
Lyon advises asking whose servers a VPN runs on; if leased, find out who the lessor is and what their data policies are, because the underbelly of bare-metal-as-a-service is where data leaks.
Lyon advises that truly private communication means not using anything with a battery or electrical signal; if you must use digital tools, use point-to-point encrypted mesh apps with no phone numbers, emails, or names attached.
Lyon advises checking a privacy company's corporate governance and jurisdiction, citing DoxNet's US base with a separate Swiss company for European operations to create legal separation between regions.
Lyon's key lesson of PKI: when onboarding users into a cryptographic system, abstract the complexity away — don't tell them it's PKI. Make the invite flow look like a normal 'invite friend' button while it performs a cryptographic key exchange in the background.
Kent C. Dodds advises that when a feature request arrives without validation of why it exists, the product engineer should go back to the product manager and ask to understand more about why it exists, rather than implementing it blindly.
Extract the research context from the traces (the output of the first research step) and attach it to all spans as the context for the judge to work on. Faithfulness evals can be done online as well as offline, and there is a faithfulness built-in evaluator you can pull up and give your input, output, and context to. Turn on suppressed tracing when running evaluate_dataframe so the evaluation doesn't send a bunch of extra traces to AX — wrap evaluation runs in suppressed tracing because the evaluation is itself a call to the agent SDK and would otherwise generate traces of your tracing.
For each eval task, create a reference solution — a known working output that passes all of your graders — to prove the task is solvable.
Label the data in your eval rubric with XML tags — especially with Anthropic models, which handle XML well. Tags around input and output reduce the chance the judge confuses the question with the answer (e.g., a user query tag around input, a financial report tag around output).
Laurie Voss describes passing in your requirements (what you were trying to get done) plus an improvement prompt telling the coding agent what it should be working on and how to get better at it. The meta-improvement loop is itself another LLM application — non-deterministic and another way it can go wrong. You could meta-evaluate your meta-evaluator of your meta-evaluator, but it gets too complicated and isn't done here. Good requirements keep Claude honest — you should spend time making sure your evaluator is as accurate as you can make it. Then feed the improved prompts (a better research prompt and a better write prompt) back into your agent.
Run the judge on examples and produce a list of where they agree and disagree. Re-export labeled spans into the notebook after annotating in the UI — human annotations appear as columns. Filter down to just the rows you labeled before comparing human labels to judge labels. With a tiny sample like 12 examples, precision and recall numbers will jump around a lot, but you can still see whether the judge is in the right ballpark or completely off. If you've made your tasks something only a human can do — or worse, something nobody could really do — you'll get a zero score.
Check the evaluations tab in AX to see a really great explanation about why the report is actionable or not actionable. Every span in AX has multiple eval scores — mention sticker, correctness, faithfulness, actionability — and you can filter and select for specific eval results. Treat several non-actionable eval scores as a capability eval and a hill that you can climb.
In the AX UI: create a new LLM-as-a-judge, pick a trace scope, set up an LLM with your Anthropic key, pick inputs and outputs via the single trace method, and save. Programmatically, create a classification evaluator with a name, an LLM, a template, and choices (actionable or not actionable, scores one or zero), then run evaluate dataframe to get results. Attach the output from the first research step to all spans as the context column before evaluating.
Use different model providers for generation and judging — e.g., OpenAI for the agent and Anthropic for the judge. It works even better if you use a different lab's model entirely rather than just a different model from the same lab.
If your evals are expensive or you have traffic, don't run them on all of your traffic — set a sampling rate to evaluate only a percentage. Sampling 1% of traffic will still give you a whole bunch of eval data without breaking your budget. Set a sampling rate to serve evals to only a percentage of traffic (e.g., 1%) when evals are expensive or traffic is high, to get eval data without breaking the budget.
Use the open source, free, OTEL-compliant Open Inference semantic conventions to map audio agent behavior; traces can be sent to a Grafana backend at no cost.
To test observability, deliberately ask the agent a question you know it can't answer, then check the traces to see the failure; the speaker asked the app 'Show me all the toys you've got under $6' as the deliberate failure test, and also ran test queries like 'What about colorful toys for toddlers?' and 'Any toys for seven-year-olds?' against the Wonder Toys agentic shopping app.
Choose sources → to get a library of your own and a daily brief.