You are benchmarking a specific configuration, not just a model
Wolfram Ravenwolf emphasizes that when you benchmark, you are actually benchmarking a specific configuration rather than just a model.
42,308 results · Most relevant
Wolfram Ravenwolf emphasizes that when you benchmark, you are actually benchmarking a specific configuration rather than just a model.
In the demo financial agent, about 44% of interactions produced a bad summary, and the agent also produced many fabricated answers, which the speaker flags as equally serious.
The 44% figure is publisher-reported from the demo; the fabricated-answer volume is qualitative.
Bostrom argues it would be presumptuous to think humanity has achieved the level of enlightenment where all possible values are transparent to us — we might be very naive. There should be humility and hope that something much better is possible in principle.
Having biased models as judges leads to optimizing something that becomes a target the models hack rather than a proper evaluation.
Peter Werry's core thesis: semantic similarity search (RAG) covers only content questions, while the other half of retrieval is translating natural language into structured database queries. The two approaches are complementary, not competing.
A context engine takes scattered context (code, PRs, docs) as input and outputs grounded context that relates to each other, tailored to the task at hand. It must care about structural relationships connecting the data, not just semantic relationships.
Use procedural tricks (deterministic code) for schema/enum extraction rather than passing everything through an LLM, because LLM extraction is non-deterministic. If you do use an LLM for extraction, always wrap it in a structural harness running procedural code to validate extracted items or to do the extraction itself.
Jesse Hall argues that a voice agent should be understood as a real-time system that senses and responds the way a person would, rather than a chatbot with a speaker stacked onto it. He frames this as the single most important takeaway from the talk.
Thorsten argues that traditional region-bound architectures fall short when addressing a global audience: depending on deployment location, an MCP or agent may be fast for a subset of users while others see much higher latency and a worse experience, with latency being one of the biggest concerns.
More data is not always better. You want fresh, high-quality data, and you want to pull only what you need. Pulling more data than needed degrades AI answer quality.
Pedro Lopez argues that CLIs should be designed with agents in mind, because features that help human users can actually block agents from using the tool.
Kohli argues AlphaFold's utility came from the fact that replicating PDB structures happened to be useful, and that this does not mean protein dynamics is understood. He and John Jumper had discussed what they were actually trying to solve: the true ground state and structural distribution of proteins.
explicit
Teams optimize for public or synthetic benchmarks, but customers encounter unexpected problems in production that break the product. The root cause is that most teams work on evals that are static, so benchmark performance fails to predict real-world behavior.
Viren Baraiya stresses that harnesses run inside your infrastructure stack, where things go up and down and network failures and partitioning happen. Without durability, infrastructure failures like network partitions or cloud outages cause unrecoverable losses, so the harness must be able to recover.
Viren Baraiya argues that when building an agent system, the harness is the application and vice versa. Treating an agent as just a component rather than an application means it alone doesn't deliver the whole premise. He also notes that shipping a microservice that does everything stops being a microservice.
Viren Baraiya warns that leaving execution guarantees to an LLM means hallucinations can cause unintended actions. He also cautions against relying on a single agent for multiple responsibilities, since hallucinations can compound across tasks.
We're all human and driven by incentive — if incentives drive people to get PRs in, we're in for a rude awakening. External pressures from stakeholders, coupled with pressures of competing with slopmeisters, begets mucho slopo. Engineers are supposed to be moving at the rate of token production, creating pressure to produce more, faster.
Lines of code is a garbage metric for productivity, and so is the engineer who burns the most tokens. No human can fully understand all the important bits of a multi-thousand line PR.
Lines of code are the cost of building — AI makes it easier to spend on files, abstractions, and complexity, but you own the bill. That bill shows up as confusion, brittle flows, and features nobody wants to touch because they don't fully understand them.
Garrett Galow describes the failure modes: relying on engineers to recode and deploy a non-engineer's prototype burns significant engineering time; non-engineers lack Vercel or Cloudflare accounts; even engineers don't want to set up deployment, auth, and security from scratch each time; and infra provisioning plus security approval slow shipping. A CEO without a Lovable account couldn't even use an app built in Lovable.
Choose sources → to get a library of your own and a daily brief.