Design with the grain: help agents lean into what they want to do anyway
Design with the grain as much as possible — help agents lean into what they want to do anyway, and it will help you out quite a bit.
endorsed
35,595 results · Most relevant
Design with the grain as much as possible — help agents lean into what they want to do anyway, and it will help you out quite a bit.
endorsed
Yoni Michael explains that you can discover which data models relate to which business domains by examining the structure of the data itself: table-level lineage, column-level lineage, shared keys, and shared naming can be used to cluster data models into domains.
In an ideal world, your agent gets better and better at answering the same questions. Changing the questions based on a changing corpus doesn't actually help you evaluate if your agents are getting better — the best way to evaluate if your agents are getting better is having to do the exact same task again. For the purpose of evals, your goals should be constant against a constant corpus.
endorsed
Kube Armor or Falco watches the runtime — what's happening at the Linux kernel level — it doesn't know about your Kubernetes. Falco watches everything, and if something goes wrong it can send an alert if it sees suspicious commands. Falco can send alerts to Falco Talon, which takes action on the alert — for example killing the pod where the exec happened to stop it from happening again. Kube Armor uses Linux security modules, so it can stop a forbidden command from ever happening in the first place — if an LS command is attempted, it fails. With Falco, if an LS command is attempted, it goes through briefly and then Talon kills the pod or whatever you have it do based on your policy. Tetragon is a sibling of Cilium and uses eBPF and LSM to get runtime enforcement done.
The output-sanitization technology demonstrated is LLM Guard, running as a proxy that scrubs responses before they return to K-Agent and then Bedrock. LLM Guard has an output_scrub field that ensures specified words never get seen in the output. Output guardrails can be applied while streaming: you can turn them on mid-stream and catch sensitive content as it streams through your system. With input guardrails off and output guardrails on, the agent's output was redacted for sensitive content.
You can set up code-based evaluators for more deterministic checks about your application, and LLM-as-a-judge evaluators for implicit things you want to extract from conversation history. Code-based evaluators support Python and TypeScript, and can run custom code checking for specific words such as brand names that should always be written the right way (e.g., catching 'Langfuse' misspelled with a capital F).
Many variables are relevant to benchmarks: the benchmark version (e.g., drop-in-bench 2.0 vs 2.1), the agent version, the model service and route (including updates to the inference endpoint), sandbox resources, sampling settings such as temperature, thinking defaults, and timeout and retry policies. If the agent runs out of sandbox resources it will fail, and you get a bad score for the model although it wasn't the model's fault. If you don't retry and there has been an infrastructure error, you would get a bad score.
Wolfram Ravenwolf describes temperature as a difficult setting for a benchmark because some model providers give a recommended temperature, some don't, some agents have a default, and some agents decide based on the model. He recommends deciding per model what the recommended temperature is, and if there is none, choosing which temperature to use. He also recommends making sure that people running the same benchmark are using the same temperature setting you chose, deciding on a top P value (whether to use 0.95, 1, or any other value), and deciding on max new tokens — how many tokens the model is allowed to generate — because there are provider recommendations and host limitations, and you usually have to configure it. He recommends setting a seed when you can, so the randomness is reduced
endorsed
After choosing one part for each of the six layers — TerminalBench for evaluation, Harbor at a specific version, the sandboxes, Terminus 2 as the agent, GLM 5.2, and specific settings — you have made your evaluation configuration. Wolfram Ravenwolf says that whatever framework and model you use for evaluation, you have to make these choices, and you should be very clear about what you are using. Being clear about your configuration is important when publishing results, so others know what you have actually used and measured. He also suggests downloading the full configuration via the QR code so you don't have to copy settings from the slides.
endorsed
Engineers often ask questions like 'PRs that I merged last week' where they know the starting point but not the destination, but they do know the relationships. Using RAG for such a query can return the wrong answer: a semantically related PR merged three years ago, because the temporal/structural aspect is missed. Traditional RAG cannot answer structural questions around state or temporality correctly.
Thorsten explains that choosing containers or VMs as the distributable unit can incur cold-start delays from several seconds up to multiple minutes, especially when scaling fast to meet unexpected demand. Over-provisioning can overcome cold starts, but keeping many unutilized containers or servers raises cloud spending and is not sustainable environmentally.
Patterson frames the talk's core argument: there are three architectural ways to protect an organization from an agent going crazy — cloud development environments, a model proxy, and an agent firewall. He also warns that having all three Lethal Trifecta layers present means a security incident is a matter of when, not if.
Share agents by making a marketplace: a repository with extra JSON in one of the directories marking it as a marketplace. A marketplace gives you light package management capabilities inside the repo: simultaneous agent updates, versioning, and security vulnerability notifications. There is also a capability to sync marketplaces with the local client device and push agents in and out.
Ray by default prefers to schedule and co-locate workers on the same nodes, which makes it easy to saturate one machine's resources (e.g., its NIC) before utilizing others.
Zero-copy retrieval and spread scheduling don't help individually at small scale, but together they have a synergistic effect at large scale. A config can do poorly at a small scale but perform really well at a larger scale, so you should always think about how to ablate your configs across your scaling ladders.
Feedback is what closes the loop for memory. When an agent retrieves a candidate memory, it will use it or reject it; if using it causes everything to go haywire and crash, that feedback needs to come back to the memory system. If a memory was harmful, its feedback score should go down; if it was useful, maybe promote it. Your memory layer needs to store whether memories were helpful, stale, irrelevant, harmful, et cetera — it's not just a repository.
Viren Baraiya frames the agent harness as the deterministic 'hands' that executes work, while the LLM is the non-deterministic 'brain' that reasons about what to do next. The harness is a combination of deterministic and non-deterministic parts: determinism is delivered by the harness, and non-determinism comes from the LLM's reasoning and thinking process.
Giving someone 10 bullets of scope and a few user journeys can result in a 20-page document that looks polished but requires hours to figure out whether there's any real thinking inside it. AI-generated work in the worst sense does not clarify the problem — it expands it, handing back ambiguity as a larger artifact for other stakeholders to resolve. In code, a vague task becomes a large PR, and a small feature becomes a pile of files.
Because efficiency in doing a task means it can be done more often (and translates into cost savings), the same evaluations can run at much higher frequency — at PR level and even on the developer's machine — catching problems earlier in the lifecycle.
One of the standout opportunities is bringing the semantic layer of what the business does into the context layer that coding agents take as input, including reasoning about why a data model exists, why a domain maps the way it does, and why one API should be chosen over another — which improves subsequent coding decisions.
Choose sources → to get a library of your own and a daily brief.