33,794 results · Most relevant
With agent experiments and the agent playground, deploy the fixed agent to dev, stream dataset inputs into it, and compare side by side what the dev agent with the fix does versus what the production agent did. Look at before-fix vs after-fix results, the judge, examples and queries, and operational metrics like latency and cost — then quantify the results of the experiment.
endorsed
To get an agent to use a graph tool, don't fight the model's prior experience — lean into it by making the graph feel like what the model already uses. Rename graph query tools to 'search' (e.g., call a tool 'query graph' just 'search') so the agent uses it naturally; if you name a tool 'search', the agent will almost always do multi-keyword search into it with no training. Meet the model halfway: name the tool 'search', account for how the model will query it, and adjust the tool's behavior accordingly. Return graph results to the model in the same tool structure and response format as ripgrep, because that's what the model has been trained on.
The recommended agent-debugging loop: first identify production failures, then gather a nice sample size of that failure information for experimentation. Determine what actually was the issue — whether the LLM prompt should change, guardrails should be added, or the tool was doing the wrong things. Bring in evaluations to see where things went wrong, make the changes, run the experiment, ship to production, and repeat the whole loop continuously.
The standard process: (1) identify production failures such as wrong tool calls, (2) gather a nice sample size of failure information for experimentation, (3) determine what actually was the issue — prompt change, guardrails, or a broken tool, (4) build out evaluations to continuously monitor the use case, (5) make the changes, run the experiment, ship to production, and repeat the loop continuously. It works but is long, manual, and tedious.
LiveKit built the benchmark they wished existed: they take a complete working agent (the hotel receptionist that takes real bookings with a real backend) and it's open source. They run hundreds of simulated callers against the agent; the simulator interrupts, mumbles, and changes its mind just like a real caller would. They don't score on vibes; they score outcomes defined the way they would for a human: did the right reservation land in the database, did a caller get cut off, are the names and dates correct, and were there any invented tool calls or leaked personal data. Completion alone isn't enough as a benchmark metric because completion can be gamed; a model can guess its way to complete. They penalize what a completion score can't see, like asking questions that didn't need to beaskd
End-of-turn detection fires from audio before the transcript lands, so the language model is already running on a provisional transcript — if the final words match, you just saved a half a second. TTS should start on the first complete sentence, not the last token, because the first audible byte is when the human stops waiting. Every model you choose shows up in the gap between the caller's last syllable and the agent's first audible byte, and this window has a fixed size.
In-season training should work to maintain 85-95% of absolute strength — the first job across three sessions. Assign one back squat session per week, plus a heavy unilateral lower-leg exercise such as a front foot split squat or Bulgarian split squat. For athletes with weekend games, stack the back squat session at the start of the week (probably Monday morning) assuming the athlete is fairly fresh. Maintain upper body strength as well, though it likely has less negative influence on gameplay. The secondary goal is transference into speed and power: clean pulls from rack, power cleans, and faster, higher-speed, slightly lower-force exercises. Accessory exercises go in the latter half of the session and are predominantly prehab/rehab.
Patterson lays out a coherent security method: (1) control data access by running agents in a secure, preferably virtual environment rather than on your laptop — this blocks private file access and prevents destructive actions on your machine; (2) restrict external communication by putting the agent on a VM isolated from your network, only able to reach domains you want, on your VPN/VPC, through a firewall; (3) govern all agent traffic on the box so you can see everything it does; (4) set up alerting so admins are notified if an agent breaches the perimeter and can kill the agent; (5) lock down content, contact, and access with restricted permissions and tightly sculpted credentials.
endorsed
Patterson recommends using cloud development environments as a remote execution layer: access them via URL, RDP, or SSH rather than working locally; run agents on cloud infrastructure that is isolated with right-scoped permissions; configure the environment so agents cannot communicate with anything unless explicitly allowed, including the public internet; and pre-configure environments so agents can start developing immediately without downloading anything. In an isolated cloud environment, an agent can only work on the code, credentials, and data you purposely put into it.
endorsed
Patterson recommends using a model proxy between the agent/model on your machine and the LLM provider: route agent traffic through a proxy server or gateway that sniffs all traffic and logs everything the agent does; the proxy can sanitize traffic and strip prompt injections, and can strip sensitive data like accidentally included credit card numbers before prompts reach the LLM provider.
endorsed
When a query comes in, first check whether there is a skill or memory match; if so, execute that skill. If the skill does not need external data, just generate a response and send it right back. When there is no skill or memory match (or the skill needs external data), go to data selection, starting by checking the cache or synced data store. Use cached context only if the data is fresh — within your SLAs and viewed as fresh enough; otherwise call the live API. Loop through data selection a few times until you build up the context you need. Once you have enough context, you might want to store a memory of whatever you just discovered or figured out, then generate the response and send it back.
Good memories are very intentional: decide the life cycle, the scope, the different relationships, and associated metadata. When an agent runtime decides to create a memory, start first with redaction — strip out PII before doing embeddings or enrichment, because letting secrets or irrelevant, weird information slip into the memory layer is a recipe for disaster. After redaction, enrich the memory: determine what the memory is related to and its metadata. Apply scope — which agent worked on it, which team, which organization — so sessions don't get memories from other agents or workflows, which adds confusion. After enrichment, do the embedding — create the vector of the content. After embedding, form the episode — the slices of transcripts that created the memory — which is good for prove
Gradient descent is summarized as: initialize weights randomly, then repeat until convergence — compute the gradients, then take a small step in the opposite direction of the gradients to update the weights. The gradient tells you the direction that increases (maximizes) the loss, so to minimize loss you step in the opposite direction of the gradient. Start by picking an initial weight value anywhere on the loss landscape, then repeat computing gradients at the new point and stepping in the opposite direction until converging to a local minimum of the loss landscape.
None stated.
Will Lyon lays out a skill distillation flow: first identify the scope for distillation (a subgraph around a certain entity, specific conversations, or a class in the ontology — an entire workspace is often not what you want), decompose the skill representation so each component is grounded in underlying data, kick off distillation in the distillation queue which fetches the data and runs a seven-stage pipeline that constructs the skill as a graph, then package it with a skill.md file and references following the progressive disclosure standard.
The agent has to surface its own uncertainty: when it hits two ways to read a field or a document type it hasn't seen, it shouldn't guess and shouldn't just pause exactly where it is. The agent should ask: 'I can read this two ways' and tell the user, 'What did you mean?' In Reducto's evals, they started failing the agent for building any shaky assumptions. Give the harness prompt guidance to use the ask-user-question tools that Claude Code and other frontier harnesses offer, instead of guessing. Reward asking instead of guessing.
Liemandt's faster way to teach: give a worked example showing the solution first, then progressively remove parts of the example so the learner takes on more of the problem until they can solve it with no scaffolding. This beats handing kids a problem with no instruction ('good luck').
Marc Klingen describes a backtesting loop for agent improvements: reproduce the known issue with the v1 baseline implementation of the agent first, so you have a baseline to compare against; backtest a new agent implementation on the updated dataset to see whether the new implementation would do better than the old implementation; and verify that the v2 candidate maintained the same performance on form, compliance, and accuracy as before, but improved on user-facing language, showing a positive delta.
The first step of The Algorithm is to question the requirements: why must these things happen in this order? Why are we doing all these requirements? Why must this take this long? A great way to know something is taking too long is to compare total time on task against total delivery time — everything outside actual time on task is waste, because it's uncaptured potential value.
The Algorithm framework has five steps: (1) question the requirements, (2) delete — delete so much you have to add stuff back, (3) simplify/optimize — get the most for the least, (4) accelerate — pull up timelines, and (5) automate — get the machines to do it. The speaker first heard the process two years ago from an interview Elon did before the book came out.
The third step of The Algorithm is simplify — another way of saying optimize: get the most for the least. Even if a step gives a return, ask whether it's worth it — that's a separate question from whether it works. If color grading took 4 of 10 total hours, it would need to be 40% of the output's quality to be worth its cost — and it isn't. Opportunity cost matters: if the team has other videos to edit, it makes more sense to do two videos at 95% quality than one at 100%. If the team had nothing else to edit, taking the extra time would make sense.
Choose sources → to get a library of your own and a daily brief.