Explore

Customize rule severity; configure action-required or high-severity issues to block merges until fixed

You can customize the level of severity of your rules; a rule does not have to be block-worthy, it just needs to be something that is addressed, and that is up to you. The deterministic part is that you can decide, based on the level of severity of the rule, that the branch or PR will be blocked. Configure in the Qodo portal that any action-required or high-severity issue blocks the merge until it is fixed, then Qodo runs again and verifies the fix.

AI Engineer · How to Build Quality Gates into Agentic Coding Workflows — Nnenna Ndukwe, Qodo AI

Harden the supply chain with regular SBOM and AI BOM runs

Javier Garza recommends supply chain hardening: secure the critical path, for example running SBOM and AI BOMs very regularly, every time there is a change in the app.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Use an AI BOM to trace zero-day impact quickly

Javier Garza recommends using an AI BOM — an inventory of all components in agentic applications — to quickly identify everything impacted by a zero-day in a specific model, dataset, or MCP tool and fix it.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Check MCP tool code and scan for malware before installing

Javier Garza recommends checking the code of any MCP tool you use — most people never read it, and it's easy to publish an MCP server with malware and wait for people to install it. Because MCP servers are long and hard to read manually, he recommends limiting what you install, only installing from sources you consider good, and using a tool that checks for malware and other issues in the code rather than reading it manually. Installing MCP servers with malicious code can get you into trouble: when an MCP tool asks you to do something and you accept, the MCP runs and may do something malicious, so verify the code before accepting.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Hook AI code generation to security scanning in a fix-until-clean loop

Javier Garza recommends creating an agent hook that scans AI-generated code for vulnerabilities and fixes them in a loop until clean: whenever Claude generates code, use a security tool to scan it, re-scan after fixes, and rinse and repeat until the fix the tool generates has no security issues. He also suggests creating a rule or hook for a tool of your choice to secure any code the tool generates, and integrating Snyk security scanning into Claude or other AI coding tools through a system prompt.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Never let AI agents touch production directly

Javier Garza says AI agents should never touch production directly, only through monitored CI/CD pipelines with safety measures in place.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Keep a human in the loop for high-risk approvals

Javier Garza recommends always putting a human in the loop for approvals for things like deployment to production or operations like deleting a database or making a user root.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Defend against prompt injections

To defend against prompt injections, sanitize input, put a good system prompt, require human approval for high-risk actions in agents, configure agents to always ask before doing anything like deleting emails or sending an email on your behalf, constrain the model behavior with a system prompt, and define and validate the output formats to ensure anything that should be treated as text and not as a command.

Contested by The Official SaaStr Podcast: SaaS | Founders | Investors

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Scan-fix-rescan loop for AI-generated code

The speaker recommends a workflow: create an agent hook so that whenever Claude generates code, a security tool scans it for vulnerabilities; if vulnerabilities are found, use another tool to fix them or have Claude generate new code, then re-scan; repeat until the new code has no security issues. This can be integrated into Claude or another AI coding tool through a system prompt.

Method, not a verified protocol.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Audit and monitor AI agents, including background log scanning

Javier Garza recommends putting auditing and monitoring on AI agents — something he says most people don't do. Without auditing, agents may do things you don't want without your attention. He suggests logging agent activity to a file and running a background tool that scans the log and flags malicious behavior that may be missed in the user interface, especially for background agent tasks which are prone to hiding malicious behavior.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Two conditions for applying hill climbing: replayable environment and verifiable outcome

To apply the hill-climbing technique to any domain you need (A) a replayable environment so loops can run, and (B) a verifiable outcome or quantifiable metric. As long as you have a verifiable outcome, you should be able to design a good loop. A replayable environment is under-discussed but extremely important.

AI Engineer · Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase

Run evals many times and measure pass^k, not just pass@k

Run evals multiple times over the same corpus to measure consistency; measure pass^k (correct on every run) not just pass@k (ever correct) — when you change the metric, the numbers completely flip.

AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

Design tools to match model priors: name them 'search', return ripgrep format

Rename graph query tools to 'search' so agents use them without training; return graph results in the same format as grep/ripgrep output because that's what the model's been trained on; lean into the model's prior experience rather than fighting it, and account for how the model will query the tool when designing it.

AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

Build graph-harnessing tools, including a blast radius tool that walks dependency edges

Build tools that specifically harness the graph's capabilities to give the agent something that makes its job easier. Build a 'blast radius' tool: pass in the qualified/canonical name of something, and it walks down the graph edges to figure out exactly what depends on it and what will break if it gets changed.

Method, not a verified protocol.

AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

Practice eval-driven development for agentic systems

Practice eval-driven development when building anything agentic — you need to be able to measure it; create a set of real production data engineering use cases, tasks, and workflows ranging in difficulty from easy to blind, with each task run multiple times on isolated clones of the actual data environment so there is no leakage.

AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

Prefer kernel-level prevention over detect-and-respond for runtime security, using KubeArmor, Falco, or Tetragon

The speakers recommended a runtime engine — Falco, KubeArmor, or Tetragon — to watch what happens at the Linux kernel level in production. They contrasted approaches: Falco watches everything and alerts (with Falco Talon able to kill the pod where an exec happened), while KubeArmor uses Linux security modules to stop forbidden commands from ever happening. They advised preferring prevention over response — wiring into the Linux kernel on the front end — noting Tetragon uses eBPF and LSM, that security doesn't always require fancy tooling (sometimes just a kernel change), and that a fork bomb not caught within about 200 milliseconds can spawn enormous numbers of process IDs and exhaust resources.

The 200ms fork-bomb figure is an approximate field observation.

AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog

Use an MCP gateway or registry with an allowlist instead of pulling tools directly off the internet

Use an MCP gateway or registry to provide a list of allowed tools that get pulled in, rather than pulling directly off the internet; in an enterprise you don't pull NPM or PyPI packages straight off the internet either.

AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog

Don't give an AI agent direct cluster access you wouldn't give a human; enforce policy with Kyverno admission control

The speakers discouraged letting an AI agent have direct access to your cluster — a mistake you wouldn't make with a person — and recommended Kyverno as an admission controller to validate or reject deployments based on cluster-level, organization-specific policy rules: requiring certain resource requests, allowing pulls only from a trusted internal registry while blocking external registries, and supporting mutating policies in addition to validating ones.

AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog

Test extraction accuracy on a small sample before scaling to a million documents

Omar recommends running a quick check on a small sample of documents to verify extraction accuracy matches expectations before scaling parsing up to a million documents.

AI Engineer · Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto

Choose models by high ceiling, stable average, and high solid base

When choosing a model, look for a very high ceiling, a very stable average, and a very high solid base. If two models have nearly the same average, pick the one with the higher solid base — a 6 percentage point difference is a noticeable gap that justifies choosing the higher-consistency model. Before picking, first decide whether you care about capability or reliability.

AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases