35,848 results · Most relevant
One score is not enough because performance is a distribution, not a point. Leaderboard averages and aggregates hide important information, and a good leaderboard score might just be a lucky run rather than a genuinely good model.
stated
Wolfram Ravenwolf recommends retrying errors caused by sandbox crashes rather than the model, because failing to retry a crashed-sandbox task gives the model a bad score that wasn't its fault. He advises not retrying errors caused by the model itself crashing by doing something stupid — count those as a fail you want to measure.
endorsed
Account for all variables relevant to benchmarks when comparing results: benchmark version (Terminal-Bench 2.0, 2.1), agent version, model service and route, inference endpoint updates, settings, and sandbox resources. Sampling, temperature, thinking defaults, timeout policy, and retry policy are also variables to consider. You must decide on a temperature setting and ensure others running the same benchmark use the same setting.
The system prompt is a very important evaluation factor even when you cannot change or inspect it. A system prompt of 'you are a helpful assistant' versus 'you are an autonomous agent' produces very different benchmark results. Agents like OpenClaw or Hermes agent have a sole MD file where you can put a lot of instructions.
uncertain
Manually checking inputs and outputs ('vibes') gets development going, but shipping based on that feeling means you won't know where things break in production; set up evals (LLM-as-judge, code-based) before shipping.
Duggal endorses the principle that evals should be built on top of traces, since traces are the foundation for understanding agent behavior.
Unlike traditional software with a deterministic flow you can follow, you won't know what your agent is doing — whether it called a tool, did analysis, or internal thinking — until you see the traces. Arize traces show the pathway the agent took and where it messed up, which is why evals should be built on top of traces.
In a normal (basic) RAG pipeline, no knowledge graph is used: take a few PDFs, chunk them, index the chunks with a text embedding model, and at query time use the same embedding model to fetch the top-k relevant documents. The fetched top-k documents are passed into an LLM, which gives back the final answer from those documents. GraphRAG is fundamentally RAG, but where the retrieval path actually includes a knowledge graph. The architecture is just slightly different from normal RAG: a knowledge graph is incorporated into the retrieval space. Vector-only RAG and vector-only databases can fall short on tasks requiring node traversal and historical context, though vector-based RAG is incredibly powerful for most use cases. When asked when to use RAG versus GraphRAG, Nyah Macklin usually 9
The host's thesis that every company needs to start thinking and operating like a labs team, with a dedicated team whose job is to be at the frontier of new technology.
Josh Woodward argues every company needs a set of people whose job is to be at the frontier, and that architecting the organization so such people — with many paid subscriptions and constant exposure to new tools — can exist is just as important as the team itself.
endorsed with 0.85-0.9 confidence
The speaker asserts that if your infrastructure and architecture is set up well, you get latency improvement and better performance end-to-end, even if per-model inference is not optimized.
Remy argues that consolidating on a single vendor while in a panic zone of overspending is a natural but flawed response, and that relying on only two frontier models and their stack ecosystem makes certain things untenable and blocks expanding AI usage into the company.
Any AI gateway (not just Tailscale's) helps you make choices at each layer of an internal AI deployment. It sits in the middle and lets you control and monitor what's happening — from security, cost, and learning perspectives — and functions as an almost universal translation layer across endpoints. To make the best choices across environments, LLMs, and data connectors, you need a place to see them all and bring them together.
All authentication in Aperture happens at the gateway layer, decoupled from individual tools. Tools like Cursor, Claude Code, Codex, or custom agent frameworks only need to hit the gateway endpoint — if on the tailnet, everything is authenticated automatically as the user, agent, or tagged device. Without centralized auth you have to re-authenticate into every new tool, which creates friction; small points of friction lead people to not try new use cases or experiment to make things cheaper or faster. Centralized auth lowers the switching cost of trying tools, workflows, and experiences.
Untrusted LLM-generated queries may use dangerous operators (e.g. the $function operator in Mongo) that execute JavaScript directly on the server, which you probably don't want. Allowing the where or function clauses creates a prompt injection point because it can run JavaScript on the server.
Mongo is schemaless, so there is no schema to read (unlike Postgres where you could read table definitions); you must sample documents to infer the schema. To infer a schema dynamically, sample 300 documents, collapse them into a single tree, and skip nulls because missing fields won't constrain the type.
Jesse Hall advises that any operation taking over a second should be handled asynchronously so the conversation never stops.
An agent can keep talking and do things asynchronously; the first update hands the mic back to the agent while it keeps working in the background. The conversation and the tool stop sharing a thread.
A human receptionist doesn't freeze while the booking system loads; they buy time and listen while they keep working, saying 'Give me a sec. I'm looking up your account.' Buying time and listening while continuing to work is the human-like behavior to aim for in voice agents.
For most athletes in season, the main decay is in absolute strength — the absolute strength number falls off more than anything else. Conditioning values will increase or at least be maintained, and speed and power numbers tend to improve throughout the season. There is also some fall-off in range of motion and strength numbers.
Choose sources → to get a library of your own and a daily brief.