34,513 results · Most relevant
Filter your traces down to just the failing ones and turn them into a dataset so you can run your changed prompt against just them — this is faster and cheaper. Also save a regression test set from the cases where your eval is NOT failing, and check it less frequently to make sure a rubric change hasn't broken previously-passing cases.
Define the judge's role first, give it context on what it's looking at and what you expect, list exactly what makes output good or bad with specific observable criteria, and avoid vague aspirations like 'a good response'. The 'you are an expert X evaluator' framing itself makes only a very small difference.
Include examples of good and bad output — it most improves quality. Label examples with XML tags, otherwise the LLM treats the example as the thing to write and spits it back. Examples give the judge a concrete reference for your labels, dramatically improving consistency especially on edge cases; if output is too non-deterministic, an example grounds it in what you expect. A good actionable example report contains percentage revenue growth, a PE ratio, a con backed by a number, and a specific recommendation; a non-actionable one is correct but has no specific data or risks and just says investors should consider various factors.
Build golden datasets to measure whether your evaluator is actually doing its job. Give yourself real concrete criteria and stick to them when labeling — don't just say this was good, this was bad; be specific the same way you told the LLM to be specific. Test in both directions: have cases where a behavior should occur and cases where it shouldn't — if you only test whether it searches the web, you'll end up with an agent that always searches the web even when it doesn't need to.
Split your labeled data into a development set (75%) and a test set (25%). Use the dev set to iterate on your eval — tweaking criteria and adjusting examples until the judge agrees with you. Hold out the remaining 25% to run the judge and see that it gets the scores you expect — that's how you know the judge is generalizing to new examples.
Laurie Voss recommends using a coding agent (e.g., Claude Code) to pull down your failing traces, modify your code, and improve your application instead of hand-writing prompt fixes yourself. Claude pulls down all of the failing traces, reads all of the explanations, and turns the explanations into actions it takes on your code base. Nothing in the automated improvement is guessed — every change the coding agent proposes traces back to a specific failure explanation. For example, if the judge said the output lacks specific recommendations, the coding agent will rewrite the prompt so that it does; if the judge said the output presents risks without supporting evidence, it will rewrite the prompt to fix that too. This is data-driven prompt engineering, and it can now be automated.
The verifier first generates a rubric with N criteria of what success looks like, ranks the most relevant screenshots in the trajectory for each criterion, uses the top-K evidence group to determine whether each criterion was met, and outputs a scored rubric with partial credit (process score) plus an outcome Boolean with an explanation.
Maintain a high index of suspicion for differentiation syndrome in any patient starting a menin inhibitor: for revumenib, onset is roughly within one to two weeks; for ziftomenib it can occur later, up to one to two months, because of its longer half-life. Patients should come in for evaluation immediately if they develop fever, joint aches, bone aches, chest pain, trouble breathing, or rash. Workup includes checking blood counts for rising white count and differentiating cells, evaluating kidney function and joint/bone aches, obtaining a chest x-ray for pleural or pericardial effusions, and performing an echocardiogram if a pericardial effusion is found. At any suspicion, give high-dose dexamethasone 10 mg twice daily for three days regardless of symptom improvement — stopping steroids
Consequential creator advice requires independent verification before Today can promote it.
Ship an AI application (e.g., a docs page with an AI chatbot), let customers use it, and real production logs start coming in. Sample a subset of production logs — maybe 10 to 20 rows — to create a dataset. From that sampled dataset, write your task, write your scoring system, and run evals; tweak your dataset, prompt, or score and run multiple different evals. After running a couple of evals and seeing which does best, you learn something about your system, make the code change, deploy it back into your application, and the whole flow repeats because those changes show up in your production logs.
Open with the 'heard the name tossed around' opener — 'Hey [name], we work with a few other [portfolio company type] companies. It's [name] from [company]. Have you heard our name tossed around?' — mentioning mutual connections or shared context before pitching. Then lead with the problem before the solution (a 'problem proposition' or 'reverse pitch'). For objections, use the 'Mr. Miyagi method': agree with the objection and sell a test drive before fully addressing it.
endorsed
Start with a 15-minute recruiter screen to filter out candidates who cannot compose a sentence or show basic professionalism, protecting the team's time. In the hiring manager interview, assess whether the candidate is 'smart' and 'cares' by asking why they want to get into sales and what they did to prepare. Then run a mock cold call twice: the first call will suck and that doesn't matter — evaluate whether the candidate listened to prep material and tried to apply it, and whether the gears are turning. Give no more than two points of feedback, coach them to lead with a problem statement instead of a pitch, then run a second call to see if they adapt. If they don't change anything on the second call, that's a massive red flag. After the mock calls you should know if you have a 'triple A'
endorsed
Run an SDR team on five weekly meetings: Monday morning meeting (energy + activity focus), Tuesday teardown (cold call/email/account review), Wednesday one-on-ones (individual coaching), Thursday teardown, and Friday team dials (energy + activity). Bookend the week with energy-focused meetings on Monday and Friday to keep the team dialing.
anecdotal
A concrete multi-step method: (1) go to the company's leadership page and manually add CEO, CFO, SVPs; (2) use LinkedIn and Sales Navigator to find additional CXO/SVP/VP-level people; (3) guess reporting lines (e.g., Scott reports into Jordan, COO Valentina reports into Jordan); (4) add people like Fatima and Nadia based on best guess; (5) make up an excuse for a one-on-one with the champion and walk them through the guessed chart so they correct it. Laura prefers doing this in a tool called 'Mirror' and has built charts down to six layers in some organizations.
A multi-step method for building multi-threaded executive relationships before a deal closes: map the customer's org chart and power base in a Miro board, seek to be corrected to learn the real org chart, get introductions from the champion to key people by framing why access is in the champion's interest, then play matchmaker by pairing peer levels (CEO to CEO, engineers to engineers, C-suite to C-suite) using the EA, a ghost-written message, and a prep doc.
The hosts present this as an endorsed playbook for enterprise deals; no outcome data is cited.
Run AM discovery in three steps: (1) recap results already achieved from the last project — start with the trust factor ('this is what you asked us to do and this is the result we got') rather than a hypothesis, unlike AE-style discovery; (2) run discovery for the next outcome; (3) come back with a next step to solve it. At the end, tell the customer you'll return in a week with a plan built with the CSM and product expert showing how they can reach the outcome using the product.
Commercial context: The creator presents this as their own offering and directs viewers to get it.
A coherent, ordered method from the talk: (1) Give the agent Markdown files — docs checked out locally, plus major website pages (changelog, pricing, features) converted to Markdown — so it understands everything about the business and stays perfectly in sync with shipped features; (2) provide all context as a large Claude MD instructions file (~30,000 lines) containing guardrails and operating advice, and ship an update to it whenever the agent gives a wrong answer or has a bad experience, so every future conversation improves; (3) show the agent the URL of its sources so it can cite them to customers for verification; (4) use a dedicated embeddings partner (Voyage) so the agent gets the most relevant documents up front and answers faster; (5) containerize the agent as a Docker file and (
Start with body weight single leg squat progression. Progress to overhead and goblet single leg squat variations. Progress to the Zurcher single leg squat variation. Progress to advanced VBT training in the single leg squat, pushing it as heavy as possible.
Roman Ugarte recommends using GrokBot as an infovore that consumes huge quantities of information, removing it from your cognitive load and coming to you with what's important. The V1 implementation: GrokBot sits on top of Slack and email, you tell it your role and what you care about, which cases warrant a direct notification, and which should just go into a daily roundup you read every day. He also gives bots a complete fire hose of information, having hooked his up to every mention of GrokBot on X plus internal context, and describes an always-on chief of staff entity that preserves your focus on what matters.
Commercial context: The creator presents this as their own offering and directs viewers to get it.
Roman Ugarte recommends setting up GrokBot to consume large quantities of information and remove that cognitive load, acting as an always-on chief of staff that preserves focus on important things. He suggests connecting it to Slack and email, telling it your role and what you care about so it notifies you only in important cases, including non-urgent items in a daily roundup instead of pinging directly, and only allowing it to page you for super urgent matters if you trust it has no false positives.
Aamir Shakir recommends giving the agent the choice between grep, normal search, or agentic search depending on query complexity.
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Choose sources → to get a library of your own and a daily brief.
Map each implemented component to the original paper and explain 2 reduced-model failure modes without claiming full-scale reproduction.
Use calibration and shift measurements to justify a monitoring or abstention threshold, and give an explicit confounding counterexample to causal feature-attribution claims.
Defend a causal claim or refusal using the DAG, numerical estimates and a counterexample showing why predictive model fit cannot establish identification.
