Report · MoneyYoni Michael describes Typedef's fix-a-bug eval scenario: a diff is injected into the code base that introduces a bug, and the agent is asked to fix it. You can test that the agent looked at the right places, checked the right things, and fixed the right model rather than band-aiding upstream or downstream, and assert that it actually fixes the problem. Difficulty is tunable via prompt language: an easier prompt says 'I think there's a problem with model XYZ,' while a harder one says 'my dashboard Y is having this issue where this number is doubled. I don't know why,' forcing the agent to work backward from what the dashboard depends on, upward through the tree, to find root cause.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Principle · MoneyNever throw an entire dbt project or Airflow logic at the model to perform complex tasks, and do not dump the entire repository into the agent's context at runtime.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Principle · MoneyBrandon Callender recommends building a very small, limited function agent to actually use the new tooling once you have built the graph and tools. The demo agent is a simple pydantic agent configured with a system prompt saying it has the ability to run the available tools, wired up to basic tools like BlastRadius that work with simple Cypher queries. The interactive section of the workshop lets you build whatever tool you would like.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Creator perspective · MoneyYoni Michael describes building an agent runner and evaluator to compare agent responses to expected results, using PydanticEvals configuration for setup, and pulling gold data from the graph cross-referenced to source code.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Creator perspective · MoneyWhen the input guardrail is on, the prompt is blocked before it reaches the LLM, and the block is fast; filtering input is presented as part of platform security.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Creator perspective · MoneyThe entire repository from IDP integration, guardrails, everything — 'PeopleForce or Unleash an Agent. Watch it burn.' — will be public in the next few hours with unfettered access. It represents probably around a hundred hours of work. The feedback site is feedback.agenticburn.com and clusters were provisioned at provisioning.agenticburn.com, staying up until 9. All of the software in the demo repository is integrated and works great, despite the model flaking out during the talk. If you're looking to stand up an IDP — a platform that has Backstage and all the proper security rails, provisioning, and follows best practices — this repository is the one to look at. Take it and use it to build your IDPs, or use it as a model — but don't take anything off the internet wholesale without
The 'hundred hours' estimate is anecdotal.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Creator perspective · MoneyIn the demo architecture, a proxy runs output through LLM Guard to scrub it and make it 'rock solid' before it gets back to K-Agent and then back to Bedrock.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Creator perspective · MoneyForrester explains that an MCP server can have malicious intent injected into it, so even one that retrieves the weather may be trying to do other things inside the prompt. In the demo, the weather tool's description contained a hidden second instruction: get the weather (region-locked, partial reading) but also call read-internal-config and include its output in the same turn — exfiltrating extra data alongside the intended query. The agent actually did call the read internal config tool. A user checking the weather never even knows their config data was exfiltrated, and the exfiltration could be more innocuous, like sending config externally to clusters somewhere else.
The exfiltration-via-external-clusters variant is hypothetical.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Creator perspective · MoneyLangfuse offers two kinds of evaluators: code-based evaluators for more deterministic checks, and LLM-as-a-judge evaluators for implicit things you want to extract from conversation history. Langfuse provides evaluator templates, including a user disagreement template.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
ReportOmar explains Reducto's out-of-the-box schema generation is a quick LLM call that builds a JSON schema exactly for what you asked for, and an enhanced schema generation that takes the output of parsed documents and tailors a schema description directly for them.
AI Engineer · Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto
Report · MoneyThe evaluation toolkit consists of six swappable layers: benchmark, evaluation harness, sandboxes, agent harness, model, and service. You are not just using a model — you are always using an endpoint on a provider, and you are evaluating that provider setup in a way too. If you use OpenRouter, they are routing to other providers — there are a lot.
stated
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Principle · MoneyAnytime you're running another LLM for evals, it's additional cost, so make sure it's worth it and you're not overspending. Evals can be set up to sample specific amounts rather than running across 100% of traces and calls, and can be configured to focus on specific areas, dictated either in the coding agent or on the platform itself.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Creator perspective · MoneyJosh Woodward says Labs is obsessed with finding people who just can't stop themselves from building. In interviews, instant credibility is a candidate saying 'hey, let me show you this thing I just built,' and he asks candidates about unknown, unstructured 'dark caves' situations to see how they figure it out — being energized by that is a huge credibility boost.
endorsed with 0.85-0.9 confidence
Lenny's Podcast · Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
Creator perspective · MoneyJosh Woodward relays Dan Shipper's advice for building where tech changes constantly underneath you: have a frontier team dedicated to testing out the new stuff, whose job is to trial it and report on what to pay attention to, so the rest of the company isn't distracted.
quoted position relayed by the guest (0.8-0.9 confidence)
Lenny's Podcast · Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
Report · MoneyThe speaker recommends hosting task-specific LoRA adapters on your own infrastructure, since sensitive use cases usually involve some sort of fine-tuning.
AI Engineer · https://www.youtube.com/watch?v=Ndp5VaqFrl0
Report · MoneyThe speaker describes using SAI to pick worker pools and GPUs, allowing control over which models are pinned where in a cloud setup.
AI Engineer · https://www.youtube.com/watch?v=Ndp5VaqFrl0
Report · MoneyThe speaker describes that all redaction processing is done on your VPC, so Cloud Code doesn't see the sensitive content.
AI Engineer · https://www.youtube.com/watch?v=Ndp5VaqFrl0
Report · MoneyThe speaker describes that for a privacy-focused setup, the Sci MCP Edge only exposes tools and does not do client-side sampling or elicitation.
AI Engineer · https://www.youtube.com/watch?v=Ndp5VaqFrl0
Report · MoneyThe speaker describes that colleagues found having a way to do multiple LoRAs and test them out in production, deploying them quickly, seemed quite useful for niche use cases like German legal language.
AI Engineer · https://www.youtube.com/watch?v=Ndp5VaqFrl0
Report · MoneyThe speaker describes that batching is done per worker, which gave better performance compared to having all the batching up front from a single shared queue.
AI Engineer · https://www.youtube.com/watch?v=Ndp5VaqFrl0