35,595 results · Most relevant
LLM-as-a-judge only reads input/output against a fixed rubric and catches known issues. Harness-as-a-judge moves evaluation to an agentic flow that can call tools, connect to external systems, and catch unknown-unknowns via trace analysis.
endorsed
Nathawani warns that continually prompting an LLM in a loop (e.g., 'give me a product, a review, and a customer profile') leads to repetition after 10-20 tries and lacks diversity, correlations, multimodality, steerability, and validation.
The speaker advises asking many small questions rather than one big question when building auditable semantic analysis on top of deterministic analysis. Throwing one gigantic prompt at a model with a large SQL file causes hallucination and invented information. Instead, parse the SQL and ask about one column at a time with its type and comments, and weigh a semantic question by giving more prompt input to that single focus question.
Yoni Michael describes a safety practice: before writing SQL to change a model's grain, walk downstream in the graph and ask each downstream dependency node whether the change breaks it.
Yoni Michael describes how a business analyst with a wrong dashboard number should be able to trace the issue back through multiple layers of the data pipeline to find the root cause — for example looking back three layers to find a wrong select, tracing back to a specific commit or deployment that landed recently to identify when and why it broke, and checking for errorful runs that populated the data to understand when the problem occurred.
The first area where coding agents fail when applied to data is impact analysis ('what breaks if I make this change?'); the second is join recommendations — whether a join is actually safe; the third is safe transformation logic changes — whether an edit preserves the meaning of the metric.
Yoni Michael argues you must meet the model halfway: adjust tool design to match its training patterns, use the same tool structure and response format as ripgrep for graph tool results, and name tools carefully (a tool named 'search' triggers multi-keyword search with no training).
A data context layer can silently replicate every single row when a join goes from a one-to-n relationship, so the agent must understand which join scenarios will result before acting; choosing a join path based on column names alone can cause a fan-out, a 7X inflation in rows, or silently drop 12% of rows downstream.
Changing a model's grain from monthly to weekly causes a monthly ARR measure to be repeated across weekly rows, so downstream ARR sums show approximately 4X the accurate value; before writing SQL to change grain, an agent should walk downstream in the graph and ask each dependency whether the change breaks the grain represented there.
When an agent tidies up a metric and removes a predicate, the SQL still compiles but the predicate may be a critical part of the business definition — e.g., removing a 'paid' predicate silently redefines the metric to include trial churn across all downstream dashboards.
Omar argues that parsing complex documents like the Epstein cover sheet with a general-purpose LLM (ChatGPT) yields inaccurate results because layouts are hard to understand and reading order is subjective.
People need to realize you're not ranking a model — you are benchmarking a specific configuration. If you change the dataset, it's a new benchmark. Behind the model sits the inference engine and whatever happens on the endpoint, which you often can't control. If the provider updates the inference software, results can change again, and you have no way to see that unless it's your own inference on your own system. The agent harness version matters too: testing Claude Code or Codex next week after an update is a different version than today. Even sandbox providers can change something in their backend without you noticing, and it could affect your evaluation results. Change any part of the configuration — dataset, model, endpoint, harness, sandbox, or settings — and it's a different.
stated
Remy describes an internal AI deployment as four components: the LLM(s), the data connector layer (MCP for many, but really CLI, APIs, or any way of getting data, even reading a file), the interface, and the sandbox/environment where the agent runs. It's important to be able to separate these components and make a choice at any given layer at any given time, because the right choice differs across teams — an engineering team won't work via a Slack thread, but a marketing team might.
Peter Werry explains that behind the scenes at Unblocked they use a lot of static schemas because they want correctness absolutely nailed down; the schema is static, tied to their code base, and they control it. For a completely unknown schema, as documents get added the schema could update, so your choice is either to rerun schema generation on a sampling interval or use a static approach.
Peter Werry argues that full-text search identifies keywords as one component of a hybrid retrieval system, but you still need a mechanism for semantic similarity search because ideas and sentences are encoded that aren't necessarily textual matches.
Peter Werry describes an anti-pattern where generated queries using where or function clauses can literally run JavaScript on the server, which means this is now a prompt injection point.
If you can stay within a second and a half end to end, the conversation still holds, but you're right on the edge. If you can get closer to 600 milliseconds, the conversation starts to feel really human. The question is never 'how fast can I go' but 'what is the most human-like agent that I can fit inside this window' — and the question 'what models should I pick' was never actually answerable.
The 50-calorie and 100-calorie air bike tests are useful conditioning benchmarks for field sports. Around three minutes for 100 calories indicates good conditioning depending on the sport's demands; about a minute for 50 calories means you are a beast. Nobody reaches the three-minute 100-cal mark without significant conditioning time — it cannot be faked. The 50-cal times converge more across sports than 100-cal times, and benchmarks differ by sport (a front row rugby player will not match a jiu-jitsu or judo athlete). The test is low impact the day after, so you can still squat the next day.
Publisher-reported benchmarks; sport-dependent
The host uses A-skips and B-skips as a proxy drill to assess and develop sprint coordination, bounciness, and body awareness. If A-skips become bouncier and springy and B-skips become more cyclical and smooth, sprint quality will very likely improve down the line. Conversely, a B-skip that looks like the hamstring is about to tear off signals poor sprint technique. He notes A/B skips benefit athletes for roughly their first six years of training, are a great tool for teaching middle school kids rhythm and timing, and are used by sprinters like Noah Lyles for warm-ups, while acknowledging ongoing debate about their value in track and field specifically.
Host's confident heuristic; he acknowledges debate around A/B skip drills in track and field
Patterson identifies the three architectural ways to protect an organization from an agent going crazy: cloud development environments, a model proxy, and an agent firewall.
endorsed
Choose sources → to get a library of your own and a daily brief.