Principle · MoneyEvery LLM-as-a-judge has three parts: a judge model (the LLM doing the grading), a prompt template (the criteria/rules of what defines good), and the data (the examples being evaluated). AX keeps these three things separate so you can mix and match — same criteria with different judge models, or same judge model with different criteria. The judge reads the input and output from each span and grades it against the rubric. A slow, complicated judge agent can judge whether your quick, fast worker agent was doing a good job.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyDebugging from transcripts alone fails to capture what is actually happening under the hood — dead air, interruptions, mishearings, and tone problems are invisible.
AI Engineer · The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize
Principle · MoneyThe team enumerated a schema of all possible failure modes and how to assign credit for each, and the universal verifier adheres to that schema.
AI Engineer · Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase
Principle · MoneyTokens per query (input and output, aka pre-fill and decode) is under the control of both users and the model, since when to finish processing a query is not under deterministic control; because it has two sources of non-determinism, you need to measure and benchmark it rather than compute it.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Report · MoneyIn an inference request, input tokens are processed in the pre-fill phase and output tokens are generated in the decode phase: one pre-fill run does a forward pass through the model, then multiple decode runs produce one or a few tokens at a time; afterwards tokens are returned to the user or a tool call is generated.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Report · MoneyThe primary state accumulated over time for requests is the KV cache of previously computed things, attached to some notion of session; an individual engine replica can track the KV cache, but handling sessions lives outside the inference server/engine in general — the wrapping layer (e.g., NVIDIA Dynamo or LLMD) defines what a session means, and clients define the sessions in that architecture.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Report · MoneyThe decode phase does far less arithmetic per user per decode than the prefill phase, which has much higher arithmetic intensity (more math per request).
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Report · MoneyCharles Frye recommends measuring time-to-first-token (how quickly you finish the pre-fill phase and have your first tokens ready to send to users) and time-per-output-token / inter-token latency (how quickly you generate each set of tokens during decodes and get them back to users) as your key latency metrics. He also suggests thinking of queries per second at the engine level as how many queries per second you can handle per replica of your inference engine, rather than trying to control the aggregate demand coming from users. Tokens per query (input and output, aka pre-fill and decode) is under the control of both users and the model, so you need to measure and benchmark it rather than set it deterministically.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Report · MoneyIntermediate artifacts produced during processing are cached in the KV cache, and the degree to which that cache is reusable is a key component of the definition of a workload; depending on the workload there are relatively fixed patterns of prefix reuse.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Report · MoneyA single application workload actually contains two semi-independent sub-workloads inside the engine — prefill (processing input prompts) and decode (producing outputs) — which often follow different code paths.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Report · MoneyInference engine workloads break down into three categories: chatbot plus, background agents, and data processors.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Report · MoneyInference engines are generally architected as a set of communicating processes: one process handles IO/HTTP/RPC serving to the outside world, pre- and post-processing with tokenizers and detokenizers wraps the core, and the core has a scheduler plus the GPU code.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Principle · MoneyBlaine Billingsley argues a design principle is a working tool — a jig — and is only useful if its opposite isn't obvious; vague principles like 'be clear' or 'be useful' don't help make decisions. As an application, when writing an automated email, agree with your PM on a specific tone principle (e.g., 'authoritative about X, friendly about Y') before drafting.
Design Better · Blaine Billingsley: OpenAI designer on the collaboration problem AI hasn't solved yet
Principle · MoneyThe source explains that when you label a stakeholder's concern out loud, more often than not the tension drops and you start solving real problems together. Blaine calls this labeling, and it works as well on stakeholders as it does across a negotiating table. The technique comes from former FBI hostage negotiator Chris Voss and his class on the art of negotiation.
medium
Design Better · Blaine Billingsley: OpenAI designer on the collaboration problem AI hasn't solved yet
Principle · FuelIf you train naturally, the most important thing you can do is figure out a way to improve your body's ability to recover from hard workouts. Jeff adds that if you're trying to add new muscle and can get less damage to those new muscle cells, you're in business. Jesse echoes this, and Jeff also argues that avoiding fruit because it 'makes you fat' is a mistake, since a single blueberry has only about 0.8 calories, so why not indulge?
ATHLEAN-X™ · 17 Foods That Make it EASY To Get Lean (AND STAY THAT WAY!)
Creator perspectiveDr Ghayas Issa explains that menin inhibitors are small molecules that do not cross the blood-brain barrier and therefore would not prevent CNS leukemia; when not combined with high-dose cytarabine, it is very important to give prophylaxis with IT chemo to prevent CNS leukemia.
Consequential creator advice requires independent verification before Today can promote it.
Research To Practice · Acute Myeloid Leukemia — An Interview with Dr Ghayas Issa on the Role of Menin Inhibitors
Creator perspectiveDr Ghayas Issa explains that differentiation syndrome is caused by leukemia cells turning back into normal cells when the differentiation arrest is lifted by drugs such as IDH inhibitors, arsenic, FLT3 inhibitors, or menin inhibitors.
Research To Practice · Acute Myeloid Leukemia — An Interview with Dr Ghayas Issa on the Role of Menin Inhibitors
Creator perspectiveDifferentiation syndrome is caused by leukemia cells turning back into normal cells when the differentiation arrest is lifted by drugs (IDH inhibitors, arsenic, FLT3 inhibitors, or menin inhibitors). In some patients, for reasons not fully understood, this is associated with a cytokine imbalance causing capillary leak, cytokine storm, and infiltration of maturing cells into organs like lungs or kidneys.
mechanism partly not understood, per Dr Issa
Research To Practice · Acute Myeloid Leukemia — An Interview with Dr Ghayas Issa on the Role of Menin Inhibitors
Principle · MoneyIf a stakeholder fails to connect you to another person, they may not be the right person; involving decision-voicing stakeholders before investing in a POC tests real access and influence.
30 Minutes to President's Club | No-Nonsense Sales · #608 - How Mapping the "Shadow" Org Chart Got Me to 305% of Quota
Principle · MoneyThe key startup question is how to build a product that gets better as models get better but doesn't get cannibalized as models get better — that's the sweet spot. Model capability improvements over the past year-plus drove much of AI product growth, and products built on models explode in demand and utility when model capabilities improve: you build something great that may not do everything you want because model capabilities weren't quite there, then the models get better and your product explodes in demand. Value is moving back to the software layer now that capable models are widely accessible — build rich products that leverage the models rather than competing on model capability. In AI products, context and community become the asset; the experience layer, network, and ways of
Principles/arguments from the speaker.
The a16z Show · The Top 100 Consumer AI Apps: Who’s Actually Paying?