Explore

Search agent training data: Wikidata hops as deterministic seeds, Tavily for execution

To create search-agent training data, use a deterministic algorithm to create seed data — run a seed collection algorithm repeatedly over Wikidata to generate many different hops, then insert those hops into the prompt as seed entities and ask the model to create a question using any or all of them. Design search riddle questions the model cannot readily answer from the internet. Execute the searches via a search tool — Tavily (Exa is an alternative provider, or you can write your own search tool). Tavily returns URLs with titles and content for each result.

AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA

Coding agent trace analysis: find issues in traces, including beyond configured evaluators

The presenters demonstrate asking the coding agent to use the Langfuse skill to find issues in traces; the agent accesses live traces to look at what questions are asked, what scores exist, and where all-caps frustration or user disagreement occurs, and also checks for issues beyond the configured evaluators — such as latency problems and tool call failures — to find where the system is silently failing in production.

AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse

Sending prototypes to Figma Make and prompting it

Select frames and use the 'Send to Figma Make' option under prototype mode; a finished prototype with interactions can be taken into Figma Make; when sending, it takes the interaction there plus whatever is prompted; a prompt like 'Make this interaction into a section of a website. Use the intro animation and keep things as is. Honor the transitions.' was used, along with instructing it to make no mistakes; a prototype can also be converted to a working site with a prompt like 'make this prototype into a working site include to build an animation in the first frame'.

Figma · Figma for Edu: Prototyping with animations in Figma

Signal outputs: datasets for evals, GitHub issues, and auto-opened PRs to approve

When Signal finds a problem, its report lets you add the failing trace to a dataset so you can run evals on it, or create an evaluator based on the problem. It creates a GitHub issue completely describing the problem — which trace went wrong, how to find it, and where to fix it. Connected to your repo, Signal can open PRs directly with suggested fixes; instead of manually reading traces or evals, you review a queue of auto-generated PRs and approve or reject them.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI

Closing the loop: ask the agent to fix the identified problem, reviewing permissions before approving

To close the loop from observability to automatic fixing, the speaker asked the agent to fix the specific identified problem. When the agent asked for broad permissions, hitting yes-to-all would upset his head of security, so the agent's proposed actions were reviewed before approval.

AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI

Change Failure Rate Volatility and Quality Risks

DORA change failure rate impact is volatile. AI has changed the amplitude of this volatility, with some companies seeing up to a 2% increase (potentially 50% more defects). This volatility exists without AI but is amplified by it, influenced by release pipelines and testing.

moderate

AI Engineer · The State of AI in Software Development: Data from 400+ Orgs — Justin Reock, DX

Build new products with a small, isolated team working from scratch for about a month

Roman recommends creating a very small team that goes off into a cave for about a month with the sole objective of building an amazing project, completely isolated from the rest of the company (private Slack channels, a separate part of the office), starting completely from scratch rather than adding to an existing product, and making many micro-decisions every day to move quickly.

Lenny's Podcast · How we built Grok Bot in a month | Roman Ugarte (SpaceXAI)

Use a balanced scorecard instead of a single vague metric like 'agent quality'

Irina advises against a single vague metric like 'agent quality', which hides interesting failures, and recommends a balanced scorecard instead. This is the measurement backbone for the scenario-based loop.

AI Engineer · Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati

For a real benchmark, run multiple seeds, put all models in the same condition, and require a statistical threshold for records

Elie Bakouch recommends that a real benchmark should run multiple seeds with all models in the same condition, and that validating a record should require passing a statistical threshold so it is not just seed optimization.

AI Engineer · We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect

Design benchmark tracks with three access levels: none, archive papers only, and full access

Elie Bakouch recommends designing benchmark tracks at three access levels: one with no external access to measure pure model capability from weights alone, one with only archive papers to measure literature-grounded research capability, and one with full access including the latest human records to measure best-possible performance.

AI Engineer · We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect

Use a lightweight accessibility skill: semantic HTML only, ARIA as needed, keyboard-only interactivity

Jemima Abu shares the lightweight version of the accessibility skill she uses: it tells agents to use semantic HTML only, reach for ARIA when needed, and ensure interactive elements work by keyboard alone.

AI Engineer · Teach Your AI Agent to Build Products Everyone Can Use - Jemima Abu, CAIS

Audit your existing code base for accessibility faults before introducing your agent

Jemima Abu recommends auditing your existing code base for accessibility faults before introducing your agent, because an agent introduced into a code base with poorly written code will take that bad code as the standard.

AI Engineer · Teach Your AI Agent to Build Products Everyone Can Use - Jemima Abu, CAIS

Auditing trick: fix one canceling error to expose a wrong model

Omar El Mghari recommends an auditing trick: if two errors are canceling out in a model, fix just one of the two and the model's performance will look worse, revealing that the model was wrong. When the 45% real figure replaced the AI-made-up 15%, the forecast error jumped from 1.7% to 28%, proving the original model was wrong despite its good performance.

AI Engineer · Right Answer, Wrong Reasons: AI Forecasts Costco - Hassan El Mghari & Omar El Mghari, Together AI

Distinguish a failed response from a failed operation

A failed response must be distinguished from a failed operation before retrying an external request.

AI Engineer · Your Coding Agent Can't Undo Its Own Mistakes - Tanmay Sah & Dolly Sah

Admit self-evolution changes only when capability and recoverability are both met

Self-evolution changes should be admitted only when both capability and recoverability requirements are met.

AI Engineer · Your Coding Agent Can't Undo Its Own Mistakes - Tanmay Sah & Dolly Sah

Due-diligence questions for AI tools: verify numbers, sources, pointers, citations, and hallucination handling

Vinoo Ganesh gives vendors and builders a set of due-diligence questions: ask whether a tool can actually verify the numbers it produces rather than make them up; ask if you can click on a number to see its source — whether it's a calculation, an extraction, or a combination — not just after-the-fact annotation; ask whether the model actually writes pointers; ask whether the tool creates a mechanism for verifying the output; ask whether every citation source is checked; ask whether arithmetic is source checked; ask whether there is a mechanism for stopping files that reach clients or customers that they shouldn't reach; and ask what happens to a number or fact that came from nowhere and how hallucinations are handled.

AI Engineer · AI Can't Make Up a Number It Can't Look Up - Vinoo Ganesh, Kepler

Vendors and builders should ask: does the model write pointers, verify output, check citations and arithmetic, stop misdirected files, and handle facts from nowhere

As a vendor or builder, ask: does the model actually write pointers? Does it actually create a mechanism for verifying the output? Is every citation source checked? Is arithmetic source checked? Do you have a mechanism for stopping files that reach clients or customers that they shouldn't reach? What happens to a number that came from nowhere, and what happens when a fact comes from nowhere? How do I actually handle these kinds of hallucinations?

AI Engineer · AI Can't Make Up a Number It Can't Look Up - Vinoo Ganesh, Kepler

Planning is the bottleneck on ProgramBench: allocate agents there

Nouha Dziri concludes that you need to allocate more agents at the software engineering stage that bottlenecks your task. On ProgramBench, extra agents pay off the most in planning: allocating more agents to planning achieves almost the same performance as the full PIV setup but at half the cost. Planning is the bottleneck because you need to figure out the binary's behavior and come up with a detailed plan of what to implement first.

AI Engineer · How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere

Consider five design choices when building a multi-agent team

When designing a multi-agent team, you should consider organization (who leads, at which stage workers come in), communication (direct vs. report-to-lead only), team size, model selection, and stage allocation across planning, implementation, and verification. Each one changes how well the team does and how much it costs.

AI Engineer · How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere

When you see a very complete answer, first ask which decision in it you haven't thought about yourself

Ryo Lu recommends that if you take one thing away, let it not be to use one more tool, but this: next time you see a very complete answer, first give yourself some time and ask which decision in it you haven't thought about yourself.

AI Engineer · When We Can Do Anything, Do We Still Know What We Want? — Ryo Lu, ex-Cursor | AI Engineer Shanghai