35,595 results · Most relevant
SallyAnn DeLucia shows loading a dataset of saved spans, mapping dataset inputs/outputs to variables if needed, running the eval with a model like 401 mini, reviewing the playground output to agree or disagree, adjusting the prompt, and continuing to run it over time.
SallyAnn DeLucia demonstrates building an eval template: start with a blank eval, give the eval LLM its job with an intro (e.g. 'You are an eval agent looking at the output accuracy for your application'), paste in the criteria bullets, provide the data by identifying trace attributes to evaluate, set up labels in task params (correct/incorrect or multi-label), set the optimization direction, and include explanations for why the eval gave its response.
Stagehand works a little bit like Playwright in some sense: you can use deterministic rules and go page.click or everything you would see in a Playwright script. When a task gets more complex, you can send natural language prompts to Stagehand to perform actions on the site, such as 'go fill out this form', 'enter my name', 'enter password', or 'find the red button'. Stagehand's observe action can find elements like a 'Learn More' button and take an action on it, and its act action can click a login button. Stagehand can also scroll pages once they load.
Use logprobs to monitor provider drift: compare the next-token probability distribution against a previous baseline. Logprob drift monitoring can detect silent changes to a provider's serving stack, such as new deployments or inference engine upgrades, and is very cheap — roughly a single token request. However, logprobs are not available on many APIs these days, and the approach only detects drift from the past, so it's unhelpful if the starting point was already serving the model incorrectly.
endorsed
To measure inference quality, send a single-token prompt and measure the log probs coming out the other side, comparing against a baseline over time. This log-prob testing approach is not specific to a particular context length. You can run the log-prob test with a consistently longer and longer prompt and measure the deviation in log probs as the prompt gets longer. Especially past the 200,000-token range in context, test that next-token accuracy is consistent with reference model providers or what it was previously returning. The log-prob approach is elegant because it doesn't require running an entire eval.
endorsed
Use a distribution test: compare outputs from a reference model and a provider API and measure the gap between the two distributions. Distribution tests are cheap — roughly a dollar per test — unlike benchmarks. However, they don't tell you why something is going wrong, may not catch intricate problems like poor quantization, and require running a reference model, which is an extra expense.
endorsed
If prompts or agent requests can be reordered, they can be batched in a cache-aware order to reuse context; interleaving requests A and B risks eviction.
SGLang's president describes the team's overlap scheduling design: they delay device-to-host synchronization so no blocking operations sit in the critical path, and launch batch 2 while batch 1 is still running on the GPU. He explains that vanilla scheduling causes GPU bubbles that are acceptable within 10% when GPUs are slow but become noticeable as GPUs get faster, because CPU-side batch preparation becomes the bottleneck when GPU inference gets faster but CPU efficiency stays flat. He reports that overlap scheduling delivers 1.5-1.1x throughput over vanilla scheduling on Hopper GPUs.
Dr. Scott Forbes explains you don't have to load creatine — it's fine if you do, but loading is not necessary. Whether you take creatine before, during, or after exercise makes no difference for long-term adaptations, though taking it close to exercise (especially when starting or loading) may help because exercised muscle takes up more creatine. Cycling on and off creatine is an okay strategy but not necessary — you can take creatine daily for long periods. Once muscles are saturated, levels take about four to six weeks to return to baseline if you stop, so missing even a week of supplementation is not a big deal. Dry scooping creatine is fine — creatine is absorbed the same way whether or not it is dissolved in water; wash it down with juice or water, or mix it into water, juice, or a sm
Dr. Scott Forbes recommends five grams of creatine per day from a muscle perspective, and shares his lab's relative dosing strategy of 0.1 grams per kilogram of body weight per day — take your weight in kilograms and multiply by 0.1. His theory is that a bigger person has more muscle and more creatine turnover, so they might require a slightly higher dose.
Consequential creator advice requires independent verification before Today can promote it.
Dr. Scott Forbes recommends creatine monohydrate because 95% of creatine research has used it. He notes creatine HCL will work just as well but lacks the same safety profile and is a little more expensive, concluding that you don't need creatine HCL — the research supports creatine monohydrate.
Dr. Forbes shares that in his lab they use 0.1 grams of creatine per kilogram of body weight per day as a relative dosing strategy.
none stated
The episode demonstrates an ordered method: open the group with a greeting ('Hey, excuse me. What's up, guys? You speak English?'), engage the family with an easy topic (asking for Barcelona recommendations, mentioning his trip), pivot back to talk with the woman one-on-one, then attempt a low-pressure keep-in-touch close via Instagram or WhatsApp.
Ben Horowitz advises that when facing a hard decision, don't enumerate all the downstream consequences of the best choice — if it's the best choice you have, just do it; thinking through everything that will happen to you is paralyzing. When the best available choice is bad, execute it immediately without hesitation — the magic of the decision is not hesitating even when the choice looks horrible. Many companies went bankrupt because they hesitated to take the bad-but-best available option (like the IPO); don't hesitate on a necessary product re-architecture because of fundraising, missed numbers, or missing features — if you're wrecked without it, hesitating guarantees you'll be wrecked. Fire an underperforming head of sales immediately rather than avoiding the conversation.
Ben Horowitz recommends that when choosing a seed-stage investor, founders look for two things: someone who can give real momentum in hiring and getting to the next round, and someone who actually knows what they're doing and is genuinely helpful. He warns that every angel and VC will give advice — including people who have never been a CEO — and much of it is bad despite being delivered with great confidence; generic advice like 'hire A players' is empty insight (he received it 27 times and found it useless).
The speaker recommends applying agent-to-agent mutual authentication with scoped credentials, least-privileged tools, and audit trails, because A2A privilege escalation — a low-privileged agent recruiting a higher-privileged one — is a serious risk.
Farza recommends that founders spend 10 minutes every day recording themselves on their phone (iPhone or Android) talking about what they did that day, then post the video to their social media of choice such as Instagram or TikTok. He frames this as a 30-day challenge: post one video every day for 30 days straight. He suggests making a new Instagram account so existing followers don't know about it, which lowers the social pressure of starting out. He also offers to be one of the likes for anyone sending him their daily challenge videos.
A step-by-step method for acquiring GPU compute: sign up for free, browse listings, submit an RFQ with full requirements across multiple regions, and evaluate provider responses on the full rubric rather than price alone.
Build a prospector tool using Replit, Lovable, Vercel, or Base44. Hook in one tool for each of inbound, outbound, and renewals (even if renewals is just spreadsheets). Start with inbound in week one, then add outbound, then renewals and support — you don't need to hook up all surfaces at once, and a little bit of data from Salesforce or spreadsheets is enough to start.
Jarmak recommends using shadow deployment to gain information when a change is less trustworthy and has a higher blast radius of effects, rather than deploying it directly and risking uncontrolled effects.
Choose sources → to get a library of your own and a daily brief.