34,068 results · Most relevant
Secure the critical path by running SBOM and AI BOM scans very regularly — every time the app changes — for all repos, and verify that packages exist as part of supply chain hardening.
Dan recommends defining the agent's mandate beyond the system prompt: what the agent should do, what is out of scope, which policies, boundaries, facts, documents, and connected systems it reasons over.
Dan recommends defining which systems the agent may engage with, restricting to specific endpoints rather than granting access to an MCP server with 4,000 tools where you don't know what it's doing.
Dan recommends defining the business impact and ROI target for the agent — e.g., expecting 70% contract renewal — and measuring it at the end to judge business success.
Dan recommends that the agent log everything — input, reasoning, decisions, and calls to integrations — as a runtime observability requirement.
Dan recommends creating success criteria together with each task so outcomes like a contract renewal can be judged as successful or not.
Dean runs 'flash deploy', which packages up the whole application, builds it, and sends it to RunPod's servers — no Docker involved. He describes this as instantaneous packaging of your application, sending it out to servers, and serving it. After deploying, he changes the endpoint ID in his prompt script to point at the new deployment, checks the workers log to see the job's printed output when it's done, and notes that once deployed the application is fully servable — customers can use your application that uses this backend, no longer on localhost.
Do not give an AI agent direct access to your cluster; you wouldn't let a person have direct access, so stick with your known flows. Some customers let AI agents have direct access to their cluster, which the speaker flags as a mistake.
Michael recommended a testing approach: run attacks against the unguarded bot without symptom prompting for a real-world test, keep that first cluster open, and reuse the exact same chats and prompts against the second, security-enabled cluster — what worked before should now be blocked. Specific challenges include getting the bot to deploy a public image, exfiltrate data (even just the menu) to a public endpoint, snoop the file system for the secret recipe, and comply despite explicit instructions not to share.
Lotte Verheyden describes the change workflow: run an experiment on your data sets, evaluate the results, and only deploy to production when results are better and no regressions were introduced.
The presenters recommend monitoring for out-of-scope requests in a production application to detect when users try to do something the system is not supposed to handle; when the evaluator fires, decide whether to expand the application's scope or fix user communication about what the application is for. They note this can reveal that users misunderstand how the system is supposed to be used, especially in larger-scope applications.
The hosts advise putting in the reps and closing the loop continuously so the dataset represents what users are actually doing: create good production examples and add them to offline evals, run experiments on maintained datasets and evaluate them, make eval runs part of CI/CD to avoid quality regressions, and deploy only when you can ship with confidence that quality has not regressed.
The presenters walk through building an out-of-scope request evaluator: set it up via the Langfuse evaluators page by selecting a template and mapping variables; target generations rather than traces because you also want to reference the system prompt; map the system prompt (the very first message sent in the interaction) and the last user message as variables using JSON paths so you can target specific parts of the interaction precisely; and in the evaluator prompt, always compare what the system is supposed to do with what the user is actually trying to do. Once happy with the test run, click 'execute' on the evaluator.
Most people will use a bunch of documents from their business, and you can use the LLM to help you figure out the right schema. Once you have a schema, using that schema will guide the LLM so it won't just make up new node and relationship types — it'll stick to that. If you don't provide a schema, the LLM will make up new node and relationship types.
Once the vector index is created, go into the agent section and access the prompt. Create the agent from scratch and configure it with GraphRAG and auto-populate. In the add tool section, create a similarity search tool that searches lesson content; the similarity search tool lets you gather insights from the lesson without having to take the lesson. Add a description to the similarity search tool and make sure you have that description too. Select the vector index named chunk embedding for the similarity search tool. Use about the top five results for the similarity search tool. Select OpenAI as the embedding provider for the similarity search tool. Use the text-embedding-3-small embedding model for the similarity search tool. The default dimensions for the embedding model is 1536. Make 9
Nyah Macklin details configuring the agent's similarity search tool: naming it 'search lesson content', selecting the chunk embedding vector index, setting top 5 results and 1536 embedding dimensions, enabling connected graph data return, and using a Cypher query that returns nodes and scores.
If a question can be solved with a search engine, vector-based RAG is usually sufficient. If a question requires multiple hops (at least three hops) through connected data, GraphRAG becomes imperative. GraphRAG is also helpful for tasks like source code analysis, supply chain questions, or anything requiring aggregation or concatenation of data from different documents.
Nyah Macklin explains that similarity search (vector search) must be added to the agent before creation, and walks through getting the sample 'create vector index' code from the GraphRAG for Python section, pasting it into the Aura console's Cypher query window, and running it, noting the 'already exists' message if run again.
Josh Woodward argues every company needs a set of people whose job is to be at the frontier — trialing new technology and reporting what the rest of the company should pay attention to. These members should hold subscriptions to all the major AI tools (roughly $200/month across everything), and the organization must be architected so that can exist. A frontier member who hasn't heard the latest hot thing viral on X in the last 24 hours is failing at their job.
Josh Woodward advises creating a space where weird things can grow — you can't just nestle the lab under an existing business unit. It needs to report to the CEO, or at enough organizational distance to maintain independence.
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Choose sources → to get a library of your own and a daily brief.
Map each implemented component to the original paper and explain 2 reduced-model failure modes without claiming full-scale reproduction.
Use calibration and shift measurements to justify a monitoring or abstention threshold, and give an explicit confounding counterexample to causal feature-attribution claims.
Defend a causal claim or refusal using the DAG, numerical estimates and a counterexample showing why predictive model fit cannot establish identification.
