Recommendation · MoneyYoni Michael recommends designing eval questions whose answers can be deterministically scored, arguing LLM-as-judge is overused when deterministic scoring is possible. Deterministic scoring reduces variance and cost across runs and makes you happier long term. He also recommends creating questions where you can structure a gold answer and check the agent's answer against it, and building a deterministic check that verifies the agent cites all expected evidence.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Method · MoneyBrandon Callender describes building the context graph for data: ingest the dbt project, pull in all the SQL, parse it, and link everything together; query the data catalog to get information about the physical table corresponding to a dbt model and pull it into the graph as well. He emphasizes that they read information from the catalog to form the context graph — the catalog and context graph coexist rather than being redundant.
Method, not a verified protocol.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Method · MoneyYoni Michael lays out a method for measuring tool adoption: use two different prompts trying to answer the same question with the same tool set, write an eval that examines all tools in the trace to get adoption numbers, then change the prompt to increase the tool usage percentage (he reports going to 72% tool usage after changing his prompt).
Method, not a verified protocol.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Recommendation · MoneyAlways be as deterministic as possible first, and only use LLMs when resolving ambiguity or doing things models are actually really good at; use the LLM only for the human-readable parts — understanding business names, summarizing model utility, or resolving ambiguity from the deterministic layers.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Recommendation · MoneyMake business definitions first-class constraints for the agent — not just a comment in some YAML file — so the context layer catches definition changes before implementation; that is the line between a syntactic refactor and a change a human can actually trust.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Method · MoneyBrandon Callender describes an eval scenario: inject a diff into the code base that introduces a bug, then ask the agent to fix it, and test that the agent looked at the right places, checked the right things, and didn't fix the bug upstream or downstream with a band-aid. He also describes forcing the agent to work backward from a dashboard symptom upward through the dependency tree to find the root cause, then asserting it fixes that problem, and notes you can make eval tests easier or harder by changing the prompt language — e.g., 'I think there's a problem with model XYZ' versus 'my dashboard Y is having this issue where this number is doubled. I don't know why'.
Method, not a verified protocol.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Recommendation · MoneyYoni Michael recommends a recipe for building really solid, honest evals: mine real task candidates out of your corpus as the first step, build a deterministic, anchored score for your evals as much as you can, run your evals many times over rather than once, and put in examples of things your agent may not necessarily be good at. He adds that giving your agent very easy and simple evals does no one any service.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Method · MoneyIn the live demo, Dean runs 'flash dev', which launches a FastAPI localhost so he can toy around with his function; executing it provisions a serverless endpoint that appears in the RunPod console. He views the container logs to see the worker testing health checks and running the job. The demo's first function talks to Torch to get what kind of GPU you have, and it ran on an NVIDIA GeForce RTX 4090, showing GPU RAM and GPU count in the result. When he cancels the flash dev session, it deprovisions the endpoint so it doesn't lie around.
The GPU model comes from a live demo and the source flags a possible numeric error.
Method, not a verified protocol.
AI Engineer · AI Apps in a Flash: Ship to GPUs Without Docker — Dean Quiñanola, Runpod
Recommendation · MoneyForrester recommends that developers try to have all the guardrails at their application layer, while acknowledging things might get missed. He further recommends running guardrails at the infrastructure layer for every agent running in the cluster, providing an added level of protection against a broad sweep of everything coming in and out.
None stated.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Recommendation · MoneyAfter every run, go through all of the errors and mistakes and check whether there have been inference issues. You can detect an inference issue if the model output contains repeated characters, and detect a tokenizer or inference server issue if prompt template tokens leak into the actual response.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyManually reviewing all production agent traces to find issues is infeasible — the speaker says he couldn't go through all of them, but Claude, Cursor, or Codex can. One of the biggest superpowers you can give coding agents is the ability to analyze traces and find the actual issues for you; Claude was able to look at actual traces and understand what the real issues are today in the live production agent.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyShipping based on vibes — manually checking input and output to verify correctness — is a common starting approach, but you can't really ship on the feeling that 'it's correct': you won't know where things are broken in production. Set up actual evals to find out where things are breaking and how to directly fix them, and build evaluations to continuously monitor the use cases you test against.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyManually reviewing all production agent traces is not feasible at scale, so give coding agents like Claude, Cursor, or Codex access to production traces so they can automatically review them and identify issues you cannot find manually.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Method · MoneyNyah Macklin explains how to enable GenAI capabilities (Generative AI Assistance and Aura Agent toggles) in the Neo4j organization, open the instance in query mode, create an agent from scratch tied to the Aura instance, and name and describe it for the GraphRAG Fundamentals course.
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Method · MoneyA step-by-step workshop walkthrough: create a vector index in the Aura console, connect the Neo4j instance, generate a knowledge graph, then create and configure a GraphRAG agent tied to that instance.
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Method · MoneyNyah Macklin shows how to find the workshop solutions, run the knowledge graph structured builder (e.g., via VS Code), watch it populate the Aura instance live, and then query the graph with Cypher to inspect nodes and relationships.
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Recommendation · MoneyCreate a space where weird things can grow and where obsessed small groups can happen, be on alert to catch it, and funnel it forward. The best ideas get started in very small groups of people who are just obsessed and usually very curious about something and just won't stop. Sometimes it's less about the idea and more about who the interesting people are and their side projects — create chemistry around those. Build an environment and nurture both ideas and the people that can thrive there.
Lenny's Podcast · Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
Recommendation · MoneyEven if you don't use Aperture, Remy highly recommends starting to consider and look at an AI gateway. Gateways are good beyond just avoiding having three different providers configured or switching between them, and are great for understanding what agents are doing — even for individuals. Many people run their usage behind an AI gateway as individuals just to see what their agents are doing, e.g. people running something like OpenClaw on their tailnet in the background that did something it wasn't supposed to do. Using an AI or LLM gateway is important not just for lock-in prevention but for experimentation and understanding as a whole.
AI Engineer · An AI Future Without the Lock-In — Remy Guercio, Tailscale
Recommendation · MoneyRemy advises organizations against consolidating on one vendor (or maybe two) to get a handle on AI spending. In the 'panic zone' of having spent a lot without knowing how, consolidating feels natural, but with only two frontier models and their stack ecosystems, certain things become untenable and AI usage can't expand across the company.
AI Engineer · An AI Future Without the Lock-In — Remy Guercio, Tailscale
Recommendation · MoneyHide metadata fields like tenant ID from the LLM and validate output to prevent leaking internals. If the LLM outputs a hidden metadata field, wipe it out and inject your own metadata field underneath. Hidden fields in the schema, like ID or metadata fields, can leak internals if not sanitized.
AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked