35,848 results · Most relevant
We're sometimes using our judgment to give agents access to our own permissions, like our own email inbox or WhatsApp account. Granting agents access to personal permissions leads to things going awry — a wrong message being sent or received — and a blast radius for all of these that expands and expands, exposing more over time.
The creator frames the LLM as a high-performance engine (like a V8) that isn't very useful by itself: to make it effective you must drop it into a full system — chassis, transmission, drivetrain — what people call the harness. If models get better but you don't have a good harness, the system won't be effective, so adapt your harnesses as models improve.
The source lays out what an agent needs to execute successfully: a safe, secure, performant runtime/sandbox; tools to take actions in the world (an agent that can't act isn't useful); context about the system and goals (like the smartest person you know dropped into a company where they know nothing); feedback — a way to run and validate its work and keep going if it's not correct; and human review (code review, evals, or other checks) to ensure alignment with your goals.
Production-grade auto-research requires three properties: a tree-based search where the agent shows every attempt so ideas compound instead of starting from scratch; observability, where you can open any attempt and read the exact code change behind its score; and steerability, where you can inject your own ideas and knowledge while the loop is running so it tries new directions when stuck.
With a normal coding agent you steer the agent every single step, but with AutoResearch the agent runs in a loop on its own. This makes AutoResearch a much bigger multiplier on your prompt and setup: point it to the right eval and it will boil the ocean for you; otherwise you're just burning through all of your LLM tokens.
Shubh's core framing: agent reliability and performance can be improved through skills and tooling around the model, without touching model weights.
Dan describes a scorecard framework built on six pillars split into two groups (configured and runtime), with governance spanning all pillars, used to evaluate agents.
Platform capabilities measure whether agents work together and share memory, human handoff, and orchestration of skills, so all agents behave the same rather than reinventing the wheel. Handoff should be centralized: don't have each agent handle its own handoff; use a central action center where people engage with all agents in the organization, and evaluate whether each agent integrates with it. Skills should be evaluated on how granular and discoverable they are, how they are orchestrated, and how thin or large the prompt is when combining multiple skills. Also evaluate whether skills come from a central repository, are built on the spot, or are human coded, and what tools are available. At final judgment, the performing agent is not the one with the largest number of tools but the one
Ang Li describes a code-first design principle followed across all of Simular's systems and even inside every component: if you want everything to be reliable, always go to the code first, because code can guarantee 100%. When the code is confusing, go to the flexible agent, and then come back to the code. The guardrail design is also code first — lots of rules; if a rule doesn't capture the case, it goes to the agent, then to the human for verification, and then comes back.
Thor Schaeff demonstrates that users can barge in and interrupt the Gemini Live model mid-response when it talks too long, and that developers can give the model access to external tools during a live conversation.
Thor Schaeff argues that because intelligence is baked into the audio model, Gemini Live switches fluidly between languages, unlike cascading voice agent pipelines where language switching is less fluid.
Fuad Ali explains that LLM-as-a-judge is essentially a fixed rubric, so it excels at catching known issues when you have a specific use case and eval you want to score on.
Your agent works in experimentation with the evals you set up, but you don't know all the evals you need before shipping — otherwise you would have written them. So you must observe what the agent actually does and adjust what you test as it goes.
described
The detect-investigate-fix-verify loop is the through line of how to get started with Arize and create self-improving agents. Verify with a Harness-as-a-judge and measure with agent experiments.
endorsed
When you need to measure progress in the foundation model community, use a benchmark — a standardized test set without the answers. BrowseComp is a really difficult benchmark for search; its questions are complicated multi-constraint queries (e.g., which teams played in a soccer match between 1990 and 1994 with a Brazilian referee and four yellow cards, two for each team — the answer is Ireland versus Romania). Even putting the full blob of benchmark text on the web wouldn't yield the answer directly.
Yoni Michael states evals are the only way to understand if agents are using intended tools, and that the eval was built not to prove the tool is good but to prove it's not going unused. Measuring tool adoption numbers is what gets you the last part of the way.
Yoni Michael recommends taking tribal knowledge that lives in someone's head or in documentation elsewhere, computing it, and making it queryable for the agent.
When producing facts for agents, always be as deterministic as possible first, and only use LLMs to resolve ambiguity or do things models are actually good at — like understanding business names, summarizing the utility of models, and resolving ambiguity from the first two layers. Facts are computed and checked against your data so the LLM only names the data assets at the end on the graph being built. This separation matters because it keeps the core facts tied to the actual implementation and to the data itself.
Even with amazing tools built, the agent will mostly ignore them unless you help it use them; getting the agent to actually use the tool is where the work really is.
Be deterministic whenever possible when building agent systems over code or data; ground understanding in as deterministic a base as possible.
Choose sources → to get a library of your own and a daily brief.