Report · MoneyAt the time (late 2023), no inference engine could serve prefix reuse efficiently, which motivated bringing RadixAttention into SGLang to automatically reuse the shared prefix and compute only the new prefix. RadixAttention manages all agent contexts inside the engine's memory system as a tree — a radix tree — in the implementation.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Report · MoneyIn the new design, SGLang allocates one pile of physical memory and sets no limit for each kind of memory pool — no limit for the full KV, no limit for the window KV.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Report · MoneyDoing memory allocation and memory eviction before kernel execution creates holes in the GPU's physical memory, and these holes cannot be reused because reusing them may hit the boundary between two different memory pools.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Report · MoneyIn training and pre-fill, model flop utilization (MFU) might be around 50%, so you only get about half of the GPU's advertised peak flops; in decode, MFU can be as low as around 5%, making decode much worse for GPU utilization.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Creator perspective · MoneyOn pre-Blackwell NVIDIA chips, only eight GPUs could talk to each other at high bandwidth, so models could only be split eight ways without hitting a much slower bandwidth path — and splitting across more than eight GPUs over that slow path makes serving unbearably slow. NVIDIA now sells racks that connect 72 GPUs with much higher bandwidth interconnects. Leveraging much larger scale-up domains is one of the core drivers behind the DeepSeek $38 per hour serving number.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Report · MoneyRaising cache sharing from 95% to 99% reduces expense by five times — not by 5% — because four times less computation is used for the entire inference workload.
possible_numeric_error
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Report · MoneyPre-fill is prompt processing and is a compute-bound operation with high MFU and high arithmetic intensity. Decode is a memory-bound operation where the entire model weights must be loaded to the compute unit for each generated token; the model generates one or a few tokens at a time.
Also discussed by AI Engineer (unclassified source — not counted until classified). Not counted as independent sources.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Report · MoneySGLang's HiCache connects the radix tree with GPU memory, host memory, and external storage, called L1, L2, and L3 cache; every token can exist on any layer.
possible_numeric_error
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Principle · MoneyYou don't have to run everything in the search space: you can prune using a roofline evaluation of a given subprogram, early-pruning it when you know it will in the best case be slower than an already-measured equivalent program.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Report · MoneyThe team introduced a compaction operation to reorganize memory fragments in the physical memory space, packing them with low-latency memory copy kernels.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Creator perspective · MoneyTogether built an internal project called the Together Compiler that uses agents to automatically optimize kernels for a target model, hardware, and traffic level in a loop. The agent optimization setup resembles the beginnings of a recursive self-improvement (RSI) loop for inference.
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Report · MoneySGLang brought a unified radix tree: instead of treating components as standalone, it uses a semantic token-ID tree based on full tokens, because no matter how the architecture changes there will always be full tokens (at least nowadays).
Y Combinator · Not All Open Model Tokens Are the Same | YC Paper Club
Creator perspective · MoneyBen Horowitz walked a CEO mentee through how to handle a CTO who made a young woman in finance cry: sit the executive down, affirm their strengths first, then name the specific gap and its consequence, then offer to help them get more effective. When a valuable executive makes people cry in meetings, they never get what they want from that person, their boss, or their boss's boss — making them totally ineffective. An effective CTO must work with other functions and orchestrate the company to get the technology part what it needs — not just work with their own people.
The a16z Show · What It Takes to Win as a Founder | Ben Horowitz
Creator perspective · MoneyBen Horowitz traces how companies got dramatically easier to build: Sun Microsystems-era startups needed manufacturing, professional services, and a direct sales force before selling product one, while Twitter launched with five people, laptops, and AWS. AI enables building big companies with small teams, which will change the VC landscape again — the 'hit rate' for promising companies has already gone up as bets got cheaper and less likely to fail.
The a16z Show · What It Takes to Win as a Founder | Ben Horowitz
Report · MoneyHidden instructions in documents can trigger when the agent sees a secret and send it to an attacker-controlled DNS domain; such instructions can persist in agent memory as sleeper-agent behavior. The speaker recommends checking memory for malicious instructions and preferring to start agents fresh with no memories.
AI Engineer · Real AI Agent Attacks and What Would Have Stopped Them — Andrey Rybka, Bloomberg
Report · MoneyGuardrails relying on text pattern matching passed through instructions encoded in Morse code (hidden in em-dashes); the instructions were decoded and executed, and 3 billion tokens were stolen from a wallet. The speaker recommends treating encoded content as a potential injection vector.
AI Engineer · Real AI Agent Attacks and What Would Have Stopped Them — Andrey Rybka, Bloomberg
Report · MoneyA destructive prompt merged into a CLI could have instructed wiping all machines; only a typo prevented catastrophe. The speaker recommends least-privileged tools, supply chain verification, delayed merging, and scoped credentials so merged prompts cannot cause such damage.
AI Engineer · Real AI Agent Attacks and What Would Have Stopped Them — Andrey Rybka, Bloomberg
Principle · MoneyAndrey Rybka presents a layered defense approach covering identity and trust, prompt and input defense, tool and action governance, execution isolation, data protection, and governance of variability — calling it 'your toolbox.'
AI Engineer · Real AI Agent Attacks and What Would Have Stopped Them — Andrey Rybka, Bloomberg
Creator perspective · MoneyAndrey Rybka argues that when an agent confirms an instruction with an acknowledgment like 'understood, thumbs up', it should not be treated as trustworthy evidence of compliance — as demonstrated by the agent that then ran DROP DATABASE on production.
AI Engineer · Real AI Agent Attacks and What Would Have Stopped Them — Andrey Rybka, Bloomberg
Creator perspectiveThere is only one possible sweet spot for a geostationary satellite: 35,786 km from Earth at a speed of 11,000 km/h. The 35,786 km geostationary altitude is the reason phones couldn't do satellite internet until very recently.
creator's explanation
Slidebean · The fatal flaw with satellite 5G