Recommendation · MoneyRunning agents locally on your own system causes interference and is not recommended. Use cloud sandbox providers like Daytona for agent evaluation sandboxes. Install the sandbox provider with 'uv tool install harbor[daytona]' and configure it via a YAML file with your API key.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyUse concurrency to run multiple trials at the same time instead of waiting serially for all 89 tasks. High concurrency with per-sandbox memory limits (e.g. 8GB each) can exceed your allowed resource quota. Running a benchmark without pinning the provider can split tasks across different providers mid-run.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyUse Harbor as the evaluation harness framework for TerminalBench. Alternatively, pick the evaluation runner first and then choose one of the benchmarks it supports as a valid approach.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyWhen benchmark scores between two models are very close, run 10 or more runs to be very sure before declaring a winner. But at some point you have to make a choice, pick one model, and go with it.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyPick a model and a provider for the evaluation, e.g. GPT 5.5 with OpenAI or Claude with Anthropic. Set the context window size deliberately because providers often deliver a smaller size than the model's maximum for performance reasons. A 'helpful assistant' system prompt produces very different benchmark results than an 'autonomous agent' prompt. A model that tries to ask the user for help in a benchmark gets a bad result because there is no user; telling it it cannot ask the user improves the score.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyBefore benchmarking, decide which metrics you are interested in: average, worst and best scores, solve rate, consistency, and cost. Cost is something not everybody takes into account. A cheaper and faster model can take longer and cost more than a bigger model because it uses so many tokens.
stated
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyWolfram Ravenwolf recommends setting the context window size deliberately because providers deliver different effective sizes for performance reasons. He personally thinks 256K is the sweet spot for best quality, price-performance, and speed, which is why he recommended it to the inference team.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyTwo fixes discussed for the demo agent: have it ask more questions before making recommendations, and add a guardrail so it only responds to certain questions and doesn't give financial advice where it's not warranted. When you add a guardrail, set up an evaluator on top of that guardrail to verify it is doing its job — such as prohibiting certain information from leaking.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyWhen the agent hallucinates on tool failure (reporting success but giving no actual response, or fabricating answers), build both an eval to detect tool failure and make prompt-based changes so the agent discloses failures instead of hallucinating. Also fix numeric groundedness so responses are always grounded to an actual reference point and don't make up numbers.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyWhen a tool call is wrong, diagnose whether the LLM prompt, guardrails, or the tool itself needs changing. For example, when a user asks 'where's my refund' and the agent responds with order status (refund issued) instead of explaining why the refund hasn't arrived, that is calling the wrong tool.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyThe team set up generic evals for known potential limitation areas in agents working in this workflow — such as 'is the agent healthy' and 'are tool calls correct' — alongside trace-specific evals. These generic checks are what produced the bad-summary flag in the demo.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyWhen fixing an issue, Duggal recommends testing the fix against a dataset built from the spans that caused the failure initially.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyPrompt the agent with a skill to pull down traces from a specific agent, find issues from evals and annotations, do its own analysis, and suggest future evals to prevent recurring issues. Then instruct it to build the recommended evals and run experiments against the spans that caused the initial failures, using the Arize dataset, evaluator, and experiment skills.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneySet up the Arize CLI by copying a single line of install code, create a profile connecting it to where your data is stored, then set up skills in your coding agent (method varies by agent, e.g., Marketplace in Claude Code). Get your code instrumented so all agent traces flow into the observability platform.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyJosh Woodward recommends thinking about planning as almost like a rolling six months — a short, continuously updated horizon suited to fast-moving AI work.
Lenny's Podcast · Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
Recommendation · MoneyGet outside the building and show people prototypes. When talking to someone about a product, you know if they're having a good time, if they're into it, or if they're just bored out of their mind. A way someone else described the 'eyes light up' signal: their pupils dilate.
Lenny's Podcast · Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
Recommendation · MoneyWhen an agent spirals out of control without reaching an answer, an AI gateway lets you point it back at what it was doing and ask what happened and why it didn't work. Cheaper models are usually not great at stopping themselves but are great at analyzing where they went wrong after the fact.
AI Engineer · An AI Future Without the Lock-In — Remy Guercio, Tailscale
Recommendation · MoneyRemy advises matching the AI interface layer to the team: engineering teams likely won't code via Slack threads, but marketing teams might work entirely in Slack asking data questions.
AI Engineer · An AI Future Without the Lock-In — Remy Guercio, Tailscale
Recommendation · MoneyRemy recommends that when a model provider offers OAuth, such as Anthropic OAuth or Bedrock Codex, you configure it on your AI gateway alongside standard API keys.
AI Engineer · An AI Future Without the Lock-In — Remy Guercio, Tailscale
Recommendation · MoneyRemy recommends that when an agent spirals out of control and doesn't reach an answer, you point it back at its own logs and ask it to analyze where it went wrong.
AI Engineer · An AI Future Without the Lock-In — Remy Guercio, Tailscale