33,353 results · Most relevant
Pamela Fox recommends designing RAG test questions that include information NOT in the dataset to verify the model admits what it doesn't know, e.g., ask about honeybees when the dataset only has carpenter bee info. The RAG test question is: 'What are the key differences between how honeybees and carpenter bees build their nests?' against a carpenter-bee-only dataset.
Ask the buyer for a standard MSA and their non-negotiables (payment terms, litigation jurisdiction), then run everything through your own legal, procurement, rev ops, and sales ops in advance — because whatever the buyer wants, you ultimately won't agree to without internal review. Get on your own legal and procurement team's calendar early, before everyone else is rushing to do the same.
To multi-thread without being salesy, ask your contact specific questions they can't answer — questions only the decision-maker could answer — and they'll volunteer to bring that person in. Use decision-maker-level questions like: What are the priorities for this year? What are you forecasting? What's the growth? What's the roadmap? How are you gonna get there?
When a deal is blocked waiting on a board presentation, schedule the procurement meeting directly after that presentation date. There's no harm in getting the meeting on the calendar early — it gets you on their calendar before everybody else tries to book time.
Best reps parallel-process vendor review (procurement, legal, security) during the evaluation instead of after, because the three-month vendor review can slow the deal. Start as soon as you know the deal is moving forward (e.g., a 75-90% commit deal), rather than waiting for the decision to be final. By parallel processing paperwork during the evaluation, you'll have ~95% of the paperwork done by the time the decision-maker returns to sign the MSA.
Ask the champion: 'How can I help you prepare for that meeting? What questions come up? What objections come up?' — including whether they have one or two skeptical partners — and offer to prepare them for those questions so they look like a star. Reference the questions managing partners or executive committees typically asked at similar firms and ask if those questions matter to their firm. Deliver the questions in language that feels authentic to you and fits the personality of the person they're presenting to. The goal of champion prep is making the champion look as prepared as possible, which in turn gets your deal across the line.
Every time you hear the word 'they' from a rep or buyer, ask yourself: do you know who is in that 'they'? When a buyer says they'll take it back to the team and decide together, ask who makes up the buying committee — some may have been your clients, have mutual connections with your execs, or worked with your CS team. If you don't know who the competition is, you're aiming in the dark.
The speakers recommend getting into the typical TDD red-green-refactor loop with your agent, using it within the verification piece itself: verify, reproduce, show me, fix it, then show me the proof. How the agent shows proof of a fix is just as important as the fix itself, because proof is what allows you to trust the model to do the work alone — when the agent says it's done, you know it's done because it has proved it to you in some way.
The speakers recommend giving your existing end-to-end tests (Playwright, React Testing Library) to your agent as a tool so it can run them and prove things out itself.
The speakers recommend writing good error messages — more important than ever because an agent can interpret and run with an error really well — and making failures explicit about why they failed with as much detail as possible about how the app got into that state, so your agent can self-heal from the error. They add that this is very easy and very worthwhile.
The speakers recommend that the most important thing is to start: create an eval baseline, use it consistently, and evolve it together as a team. Get to the point where your changes and PRs automatically run the eval, so you know whether changes are safe to land, improving things, making things worse, or not changing anything. They note your team might score verification differently or have different criteria — the scoring approach is super flexible.
Within the pipeline, Wisedocs can flag conflicting diagnoses, deduplicate records, and search for missing records. When surfacing conflicting diagnoses to a human reviewer, the system points to the specific pages where the conflict occurs (e.g., 'on page two, on page 20, there's a difference of diagnosis between these two doctors. You should take a look').
Martone warns against using LLMs for every problem: they fail at determinism and calibrated probability estimates. With large amounts of data, traditional machine learning is the better answer, and problems need a mix of deterministic and probabilistic approaches.
endorsed advice
Give agents tokens that are both time-limited AND budget-limited (how much resource they can consume), so access ends when the token expires or the budget is exhausted. When an agent exhausts its budget, the supervisor decides whether to grant more budget — unlike traditional rate limiting which just cuts the agent off with no feedback. Provide continual feedback to agents about how much budget was consumed and how much remains, so the agent can operate within budget like a person treating a scarce resource carefully. When an agent's budget is close to being fully reached, withhold issuing another credential to stop the agent from using too much of a downstream resource.
None stated.
In the brains-and-hands architecture, keep agent credentials and access separate from the agent itself — no broad standing access for the agent. Put the agent's model (brain) in a sandbox containing only its tools, with an egress proxy outside the sandbox serving as the authorization boundary — this controls what the agent can access and prevents direct API access. The egress proxy boundary enables applying authorization to the agent system, translation between systems, and OAuth termination — while the sandboxed agent experiments freely inside. This sandboxed-brain-plus-deterministic-egress model is the predominant model that Anthropic, OpenAI, and many other labs are pursuing today.
None stated.
Ian says Keycard's broad view is to have broad-based deterministic policy that says what must never happen, or which things always require a human to be involved. There is a big gap between the policy a human or organization can write and what the policy should actually be for the specifics of a specific task, and the real challenge is layering over those two things. Related guidance: model both user-to-agent transactional intent and the business's intent for adopting the agent, then compute their intersection as the agent's bounding box of allowed actions; and derive access decisions from multiple intents from different areas of the organization that live independently but combine into a runtime authorization decision.
None stated.
Use faithfulness evals when the judge lacks fresh data: faithfulness asks whether the report makes sense given the data the agent was given. Run it with three inputs — the input (what the user asked), the output (the report), and the context (the search results and notes). Pull both the raw search results and the research notes in your getAgentRuns helper; a first version that only gave research notes caused the judge to flag real numbers as made up. Expect faithfulness to run slower than correctness because the judge reads every search result.
Query Phoenix for spans with parent ID null to get only top-level spans (the agent runs); the repo's getAgentRuns helper hands back each run's span ID, input, output, plus the research turns output. Use createEvaluator from the Phoenix evals package with an ordinary function, marking it as a code eval so no LLM is involved. Code eval functions receive input and output, check each requested ticker appears in the report, and return a label plus a score with the missing ticker named so the failure explains itself. Run them via npm run eval code — they execute almost instantly because code evals don't wait for an LLM.
Install the Phoenix plugin/skill for your coding agent, point it at your traces to look for weird things, have it read traces, come up with categories of failures, count them, and write each category back onto its trace in Phoenix as an annotation; let it write notes to disk if it asks.
Use all three kinds of evals at the same time — they are complementary, not competing approaches. Use code evals when the answer is deterministic: formats, lengths, required fields, banned phrases. Use an LLM judge when you need understanding: is it accurate, is it grounded, is it useful. Use humans for failures you haven't seen before and to check that your LLM judges are judging correctly.
Choose sources → to get a library of your own and a daily brief.