Recommendation · MoneyYoni Michael recommends hooking the code graph in as an MCP to the tools you're already using — e.g., having the code graph in addition to Claude Code and asking Claude Code to call the graph — and then writing skills so the agent uses the graph properly. To engineer an agent to use these things in actual coding, promote the prompt-engineering practices into skills and give those to your coding agent. He notes an engineer receiving such a PR for the first time can get a half-decent first pass from being more deterministically guided, and that the deterministic part is helpful because you can incrementally diff over what's changed and analyze it deterministically, getting a fast conclusion about what's happening as opposed to waiting for the agent to read everything.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Recommendation · MoneyThe host recommends using a classification layer in front of an agent to check for prompt injection. For such a layer, you can either use an LLM-based layer, or alternatively a semantic search layer (e.g., vector search via Quadrant or LanceDB), or an in-memory process.
AI Engineer · How Secrets Leak Through AI Agents (and How to Stop It) — Venice AI
Recommendation · MoneyTo decrease risk when protecting agent-handled data, the host recommends ensuring narrow agent scope: if you pull tools from an MCP server that may have 15-20 different tools, you don't really want to use all of them — you actually only really need one. He adds that tight context loops caused by narrow agent scope can also improve context window performance, because fewer tools means the agent may perform much better.
AI Engineer · How Secrets Leak Through AI Agents (and How to Stop It) — Venice AI
Recommendation · MoneyThe host says using sandboxes and VMs, for example Firecracker and AWS EC2 in a VPC, can significantly upgrade the security of your agent. However, sandboxing does not act as a cure-all for agentic systems: if your agent has internet access enabled, it can still accidentally exfiltrate data through external API access if you're not careful, and therefore monitoring permissions is essential.
AI Engineer · How Secrets Leak Through AI Agents (and How to Stop It) — Venice AI
Recommendation · MoneyFor an LLM call observation, attach data like duration, token cost, token usage, model used, and prompt version used — everything you can think of that will help you assess the quality of your LLM call. For tool calls, the attached data can be completely different; for example, a web search observation can include the query that was used.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
RecommendationOmar Alhait recommends that when layout is hard to read, such as in a construction blueprint, you spend a lot of time in the layout prompt tailoring it to understand how well your current extraction solution maps across all of your different documents.
AI Engineer · Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto
RecommendationOmar urges the audience to send the Epstein cover sheet to ChatGPT and try to get results out of it to see how accurate it is, noting the cover sheet is a very complex parsing problem with hard-to-understand layouts and somewhat subjective reading order.
AI Engineer · Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto
RecommendationOmar recommends regular schema generation for something quick to get up on screen and see results, and suggests parsing the documents first and then passing them in if you are interested in nuance. To use enhanced schema generation, click Enhance Schema Generation and the tool will run the documents first to understand what's in them so it can tailor schema descriptions.
AI Engineer · Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto
Recommendation · MoneyTreat the WolfBench leaderboard as a decision surface, not just a chart — load a snapshot, add your own runs, filter metrics and models, reorder bars, and export. The display can look overwhelming because there is so much information on the same bar, but you can filter to different information to make it a simple bar. When comparing two models, use the compare feature to isolate just those two and reduce the number of bars on screen. Use the WolfBench interactive page to create charts you can copy and paste elsewhere — mark a model, remove or reorder bars, and export for reports or social posts. Share the WolfBench chart URL so others see exactly the same view you had on your screen — the chart state is encoded in the URL.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyRunning agents locally on your own system causes interference and is not recommended. Use cloud sandbox providers like Daytona for agent evaluation sandboxes. Install the sandbox provider with 'uv tool install harbor[daytona]' and configure it via a YAML file with your API key.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyUse concurrency to run multiple trials at the same time instead of waiting serially for all 89 tasks. High concurrency with per-sandbox memory limits (e.g. 8GB each) can exceed your allowed resource quota. Running a benchmark without pinning the provider can split tasks across different providers mid-run.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyUse Harbor as the evaluation harness framework for TerminalBench. Alternatively, pick the evaluation runner first and then choose one of the benchmarks it supports as a valid approach.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyWhen benchmark scores between two models are very close, run 10 or more runs to be very sure before declaring a winner. But at some point you have to make a choice, pick one model, and go with it.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyPick a model and a provider for the evaluation, e.g. GPT 5.5 with OpenAI or Claude with Anthropic. Set the context window size deliberately because providers often deliver a smaller size than the model's maximum for performance reasons. A 'helpful assistant' system prompt produces very different benchmark results than an 'autonomous agent' prompt. A model that tries to ask the user for help in a benchmark gets a bad result because there is no user; telling it it cannot ask the user improves the score.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyBefore benchmarking, decide which metrics you are interested in: average, worst and best scores, solve rate, consistency, and cost. Cost is something not everybody takes into account. A cheaper and faster model can take longer and cost more than a bigger model because it uses so many tokens.
stated
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyWolfram Ravenwolf recommends setting the context window size deliberately because providers deliver different effective sizes for performance reasons. He personally thinks 256K is the sweet spot for best quality, price-performance, and speed, which is why he recommended it to the inference team.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyTwo fixes discussed for the demo agent: have it ask more questions before making recommendations, and add a guardrail so it only responds to certain questions and doesn't give financial advice where it's not warranted. When you add a guardrail, set up an evaluator on top of that guardrail to verify it is doing its job — such as prohibiting certain information from leaking.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyWhen the agent hallucinates on tool failure (reporting success but giving no actual response, or fabricating answers), build both an eval to detect tool failure and make prompt-based changes so the agent discloses failures instead of hallucinating. Also fix numeric groundedness so responses are always grounded to an actual reference point and don't make up numbers.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyWhen a tool call is wrong, diagnose whether the LLM prompt, guardrails, or the tool itself needs changing. For example, when a user asks 'where's my refund' and the agent responds with order status (refund issued) instead of explaining why the refund hasn't arrived, that is calling the wrong tool.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyThe team set up generic evals for known potential limitation areas in agents working in this workflow — such as 'is the agent healthy' and 'are tool calls correct' — alongside trace-specific evals. These generic checks are what produced the bad-summary flag in the demo.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI