Anecdote · MoneyIn his company, everyone using Claude Cowork gets a company-wide policy that protects against data leakage and harmful actions like deleting files or sending data outside. Company-wide policies should disable always-allow permissions even though it's a pain, because it protects against many harmful actions. But just putting a company-wide policy in place doesn't work — people will figure out workarounds, so you have to educate them about what the guardrails protect against. Guardrails protect against harmful agent actions like deleting your mailbox, spamming everyone in Slack, or sending a crypto wallet to a third-party website.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyJavier Garza recommends that whenever an AI agent gives you an answer, you ask it how it came up with that answer to surface hallucinations; that when asking an AI tool to do something important, you request four different ways of doing it and pick the best one; and that you don't trust everything an AI tells you, just as you shouldn't trust everything you read on social media.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyUse network segmentation with strict firewall rules so the AI cannot open ports that should only be internal.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyUse the OWASP Top 10 for LLMs to learn about today's attacks and how to defend against them. Develop AI coding policies for your company and publish them to everybody.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyDan recommends defining who is allowed to engage with the agent as part of its security configuration.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Recommendation · MoneyMake the search questions as simple and riddle-like as possible so they invoke the search behavior from the model.
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Recommendation · MoneyYou can add LLM-as-a-judge to score how your model is doing — e.g., another model judges the quality of the question the previous model created.
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Recommendation · MoneyCreate questions that are difficult so they invoke reasoning and tool calling from the model.
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Recommendation · MoneyAnnabell Schäfer advises leveraging coding agents to 10x yourself, but staying in the loop and not letting the coding agent do everything on its own.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Recommendation · MoneyLotte Verheyden advises treating your data sets as structured test cases, heavily inspired by what you see during monitoring.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Recommendation · MoneyAnnabell Schäfer advises deciding what is interesting to monitor for your specific application rather than monitoring generally.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Recommendation · MoneyRegister for a free Weights & Biases account to follow along with the workshop; the account comes with free inference credits you can use to run evaluations. If you prefer, use OpenRouter or a different provider instead of the default, and tell your AI agent to change something and make it work with your setup.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyWolfram Ravenwolf recommends being very clear what you are using when publishing your results, so others know what you actually measured.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyDo not use LLMs for the validation/sanitization part of the pipeline; write procedural code instead. Technically, the sanitization and validation implementation is not that difficult — you just have to write the code, and it's all procedural.
AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked
Recommendation · MoneyYou can express a structured query interface through MCP, even dynamically updating the tool every time the schema changes. You could also provide an MCP interface that takes natural language queries and translates them on the back end, or a direct interface to the database, depending on what the customer interface looks like.
partial
AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked
Recommendation · MoneyJesse Hall recommends testing the conversation rather than individual components, and continuing to test in production.
AI Engineer · Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit
Recommendation · MoneyPatterson advises that before adopting AI agents at a company, you need security approval from CISOs and buy-in from DevOps, since they're the ones who have to deploy it to teams. You should talk to security people about what the CISO is concerned about and how to put boundaries around agents.
AI Engineer · The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder
Recommendation · MoneyPatterson urges teams bringing AI into an organization to rethink their architecture to protect against agents dropping databases, deleting personal information, uploading customer information, or extracting company IP to the public internet. He also recommends setting up AI agent rollout in a repeatable, consistent, reliable, and secure way for everyone at the company, and going slow enough to think about architecture and know what infrastructure actually needs.
AI Engineer · The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder
Recommendation · MoneyPedro Lopez's guidance on choosing between MCP and CLI: MCPs are great for non-technical users to quickly prototype things, build one-off reports, and leverage the MCP ecosystem; the CLI is better for more technical users and longer-running tasks. Both have their place.
AI Engineer · Designing CLIs for Agents, Not Humans — Pedro Lopez, Airbyte
Recommendation · MoneyDerive eval scenarios from production rather than imagination. Creating synthetic data might be very important at the beginning of the process, but as soon as you start getting customer signals, make sure they feed your eval.
AI Engineer · Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab