Don't go back to vibes: compare before and after structurally
If you fix your agent by changing the prompt and then just run it on a couple of examples and eyeball it, you're back to vibes — you need a structured way to compare before and after.
35,848 results · Most relevant
If you fix your agent by changing the prompt and then just run it on a couple of examples and eyeball it, you're back to vibes — you need a structured way to compare before and after.
The ticker check catches really basic failures but catches them really cheaply — that is why it is important. If your financial analyst agent writes a beautiful report about Microsoft when the user asked about Tesla, no amount of eloquence matters because the answer is wrong. You can catch basic agent failures like wrong-company reports with five lines of Python.
Code evals are simple deterministic checks similar to unit tests — fast, cheap, and totally reproducible, giving the same answer every time. LLM-as-a-judge evals use a second LLM to judge the output of the first, handling questions code evals can't answer like factual accuracy, faithfulness, and tone. Human evaluation is the gold standard for quality when slow, expensive review is acceptable.
Treat your LLM judge as a classifier: it takes an input, makes a prediction that can be right or wrong, and you can measure its performance against ground truth (human judgment).
Laurie Voss steps back to describe the whole shape: production produces traces, online evals grade them and produce explanations, a coding agent reads the explanations and proposes fixes, experiments verify the fixes work, then you ship. The production traffic that comes back gets graded by the same online evals that started the loop. He calls this the software development life cycle closing in on itself — your AI software helping you improve your AI software.
The Swiss cheese model from safety engineering: imagine each layer of defense as a slice of Swiss cheese with holes — gaps where problems can slip through — but if you stack enough slices, the holes don't line up, so what gets through one layer gets caught by the next. Your code eval catches format issues but misses semantic problems; your LLM judge catches reasoning gaps but misses subtle hallucinations; your human review catches the subtle stuff but can't scale to every trace. No single eval catches everything — the combination of evals is what gives you coverage, so stack the cheese.
Benchmark your LLM judge against human performance, not perfection, because humans get judgment wrong quite a lot of the time. If your judge achieves higher consistency than humans, it is doing better than humans — a judge that sometimes disagrees with you is not by itself a reason to distrust it.
Multi-agent systems add handoffs where one agent passes work to another, requiring evaluation of whether the triage agent routed correctly and the specialist agent handled the handoff gracefully. Non-determinism means you need more evals because errors can cascade. Cascading agent failures are worse than obvious failures because no unit test would have caught them.
Traces record what your AI did at runtime: every agent call, tool call, and LLM invocation with inputs and outputs at each step. Read each span's input, output, timing, token counts, and metadata to understand what happened at that step. Without traces, all you see is 'I gave it a prompt and I got a report'; with traces, you see every step.
The core abstraction of the workshop: a task is a Python function that takes an example from a dataset, runs anything (full agent, single component, or API call), and returns an output. AX doesn't care what happens internally — as long as input goes in and output comes out, you can run experiments on it.
Code evals are easier, faster, and cheaper than LLM evals — write one first for whatever matters most (correctness, faithfulness, etc.) — but they're brittle because two perfectly correct answers can be worded completely differently.
You can close the software development loop without doing anything manually once you've defined what good looks like, and you can use any observability platform to do this, not just Arize.
When agents read your agents, you can detect signals in your traces — patterns inside eval failures en masse.
The prefill/decode arithmetic-intensity difference is the fundamental constraint and also the operating space for configuring an engine and adding features or adjusting engine behavior.
Inference is stateless if you don't care about performance; it is stateful once you do care about performance, which is pretty much every case.
Charles Frye says the latency budget (SLO) coming from the application layer is the primary constraint you operate under when deploying an inference engine. Different application archetypes have different budgets: chatbot-plus workloads have short decodes (tens to hundreds of output tokens) and very tight latency tolerance, since a person wants to see something in a few hundred milliseconds at the very most or they get bored, get angry, and turn away. Background agent workloads have looser latency budgets, so milliseconds of network overhead matter less. Data processing workloads demand aggregate throughput over latency because they rip through database backfill type jobs. He recommends mapping workload metrics back onto your application archetype (chatbot plus, background agent, data处理器)
The scheduler process is the bottleneck to the GPU component — it is the gatekeeper to the GPU and the other place where the majority of performance engineering and care needs to go; despite not doing as many operations as the GPU work, it can become the bottleneck because it gates access to the GPU.
Frye recommends CUDA graph capture as a key technique to avoid per-kernel CPU launch overhead. It works by tracking all kernels in the model forward path and turning them into one big data structure — a DAG of operations encoding dependencies (e.g., once this kernel mutates data at this pointer, the next kernel can run). Per-kernel CPU work (deciding what to run, how, and which pointer) collapses into a single CPU-side decision: which graph to launch. This also relaxes constraints on the model forward pass component of the engine.
Drawing on the Systems Bible, Blaine argues that when choosing between building your own fast-but-free system versus integrating with existing slow-and-terrible systems, the fact is both are going to be hard: integrating just makes a bigger failure in a bigger system, and the bigger things get, the more likely you won't be able to ensure their success. A system will eventually break down and stop working, probably quickly. So it's on you to be vigilant about keeping a system down — making sure it works beautifully at a small scale or solves one thing absolutely perfectly — and then very cautiously scale up a little bit and make sure it doesn't fail. He notes it's very easy to make a great design system that doesn't extend into new features, and that the grandness is in how deep you're愿意 to
medium
The first time you figure something out with an LLM you burn lots of tokens to reach the end goal. A subsequent agent session can retrieve the result from memory with far fewer tokens, saving token spend.
described
Choose sources → to get a library of your own and a daily brief.