Explore

Calibrate a taste-matched critic using polarizing films

Rate seven highly polarizing films on your own personal taste, then find which critic's ratings best align with yours to identify a trusted critic for future recommendations. Use the nine films that top critics most disagree on as the calibration set, and watch the polarizing calibration films you haven't seen before rating them. When building the critic shortlist, exclude critics who don't publish numeric ratings (prose-only or video-review-only critics), since the comparison needs numbers. Accept that even a taste-matched critic will sometimes disagree with your ratings — that's expected and fine.

Method, not a verified protocol.

Slidebean · yeah... Netflix only wants you to watch THEIR movies

Level 1: prevent agents overstating what they did via explicit deliverables and verification commands

Level one of the review method prevents agents from overstating what they did. Before an agent starts a job, the job spells out exactly what it has to hand back: the file, where it goes, and a command that checks it. The job also points out shortcuts agents have taken before so the current agent doesn't repeat them. The agent sent the job runs the check itself, and the reviewing agent must end its review with either 'artifact verified' with a fingerprint of the file, or 'artifact missing' with the reason. Saying 'done' on its own is not a permissible answer.

Method, not a verified protocol.

AI Engineer · I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore

Level 3: prevent reasoning errors via three-stage adversarial review

Level three of the review method prevents agents from getting the reasoning wrong. When clarity of thought really matters, perform adversarial review in three stages. Stage one: the agent audits its own work, listing every claim and marking each as measured, checked against the source, inferred, or assumed; anything it can't back up gets tested or labelled as a guess. Stage two: reviewers with none of the original agent's context, each acting as a persona with reasons to find fault, read the work cold — each plays a real reader and gets the work and the sources, but no history, drafts, or conclusions to agree with; their findings get checked against the source before anything changes. Stage three: once the work is revised, brand new reviewers who've never seen it check the final version.

Method, not a verified protocol.

AI Engineer · I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore

Level 2: prevent factual errors via independent dual extraction and a source-anchored third agent

Level two of the review method prevents agents from getting the facts wrong. When correctness matters, first find something in the source that the answer has to agree with. Two agents get an identical set of instructions and do the same job independently, neither seeing the other's work; each notes where every fact came from and checks its results against the facts. A third agent with the original source in front of it compares the two agents' work line by line and settles every difference by going back to the source, never forcing a match.

Method, not a verified protocol.

AI Engineer · I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore

CVE remediation workflow: skill, Harbor task, and evidence

Al Chen describes a workflow for CVEs: create a skill that does the heavy lifting of CVE reporting and internal research, outputting a table of what to fix and explanations for bank security teams; create a Harbor task with instructions, Docker images, and tests; instruct the agent to write an exploit against the old version and a test to verify the new version remediates it; and run many Docker containers or Modal/Daytona sandboxes in parallel to get trials of agent performance. The point is that bumping a package version is not enough — you must provide evidence the exploit is gone.

Method, not a verified protocol.

AI Engineer · Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco

The AI engineering loop: trace, monitor, build datasets, run experiments, close the loop

Companies that successfully ship and continuously improve reliable agents follow a set of practices called the AI engineering loop. The foundation of the AI engineering loop is always tracing. Monitor aggregate metrics on what is going on and detect signals where things are going wrong. What you get from production through monitoring is one of the foundations you can use to build out your data sets. Run experiments by trying different variations of your system to check how it performs against your data sets. Continuously close the AI engineering loop: learn from production and feed it back into data sets.

endorsed

Method, not a verified protocol.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Rollout recipe: start supervised, refine data sets, move to sampling at a confidence threshold, then handle exceptions

Kormos lays out a phased rollout: on day one, start supervised — agents and humans work together as part of the process, and output is manually verified to ensure it is of high quality. During the supervised phase, refine data sets and ensure the right confidence mechanisms are in place, with domain subject matter experts defining what good looks like and beginning to verify it. When measured confidence reaches a policy threshold, move towards sampling: humans look at a slice of the output rather than all of it. During sampling, begin systematic evaluation through traces to ensure what the humans aren't seeing still meets the quality bar. Once the confidence threshold is cleared, humans handle more of the exceptions of the process but still act as participants in refining the data sets and

Method, not a verified protocol.

AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase

Backtesting workflow: gather historical data, extract features via Snapshot, and iterate one feature at a time

For traditional model backtesting, gather the data needed to backtest the model you already have, even if the live system doesn't use agentic web search. Use Snapshot to jump back to a known URL or website and extract whatever feature is needed (e.g., an array of numbers), even if Exa isn't involved in the prediction itself. Backtest agentic features one by one against true positive matches to see whether the system correctly predicted that a business succeeded or didn't succeed. Add agentic features into your system one feature at a time so you can control the experiment, and tune non-deterministic systems carefully because the way you tune them is really important — searching one way may succeed more often, another way less often.

Method, not a verified protocol.

AI Engineer · Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa

Five tests for whether your product has a defensible learning loop

Five tests: (1) Does the product take the action itself — it can't just be recommending? (2) Can the product see what happened afterwards? (3) Does the outcome change what the product does next? (4) Can the lesson improve results for other customers — the ideal case, though not everything is like that? (5) Does the accumulated learning stay with you — the crucial part?

Method, not a verified protocol.

MicroConf · SaaS Moats in the AI Era: Sell the Outcome, Not the Tool

CTX verification flow: agents report back through MCP and a centralized verifier gates completion

In CTX, agents are assigned work via prompts, go do the work, then come back through the MCP connection to CTX and ask 'this is what I've completed, am I done? Please verify it.' A workflow that needs to verify a PR was created correctly runs an evaluator on a known verifier, which can start as a simple CLI and becomes a hardened piece you can reuse again and again. A PR verifier goes to the GitHub API and verifies the PR truly exists, is healthy, and all merge checks pass. Only after verification passes will the workflow mark that node satisfied; otherwise it redirects the agent with the list of things still failing. The verification gate requires that all pull requests are merged, and that work is done by the centralized authority, which checks the API or whatever the artifact is and del

Method, not a verified protocol.

AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS

The eval workflow loop: trace, read, write evals, verify judges, change the agent, prove it, lock in with CI

Voss describes the overall workflow the talk builds: trace the agent, read what it did, write evals for what matters, verify the evals can be trusted, change the agent, prove the change worked, then lock it in with CI. You can run your evals, test things, make sure they match your human judgment, put them into CI, and make sure your builds are gated on whether the LLM's judgment of your encoded definition of good really works.

Method, not a verified protocol.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Start with two memory structures: short-term and long-term memory

Nacho Martinez recommends starting by experimenting with only short-term memory and long-term memory — just two structures. When the agent considers something in short-term memory or the context window worth keeping, promote it to long-term memory and store it as raw text; then vectorize that text and let the agent loop explore it.

Method, not a verified protocol.

AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle

Developer security job: declare, contain, detect

A developer's job for agent security is threefold: declare, contain, detect. Declare what your agent is — its permissions, what it can do, its persona, skills, and tools. Contain the agent physically so that even in the worst case, the harm it could do on the open internet is minimal: use explicit environment policy configurations to restrict agent access to specific domains, and configure environment policies to allow web search but not web fetch, or vice versa, depending on needs. Detect: listen to the durable event stream in real time and immediately interrupt any sessions going in an undesired direction — Anthropic provides a durable event stream capturing the full trajectory of everything an agent does across its lifecycle. Supporting practices: establish permission policies so only

endorsed

Method, not a verified protocol.

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Agent-level tracking and eval: per-agent projects, six tables, quality checks, and executive dashboards

Harish Gaggar describes implementing agent-level tracking and eval, distinct from MCP gateway monitoring, to track what queries are submitted to the agent and what responses come back — including which LLM model was used, token cost, tokens burned, evaluation criteria, and response quality. He recommends storing this data from day one and giving every agent its own tracking and eval project and tables (a thousand-plus agents means a thousand-plus separate projects; with 100 agents you have 100 different projects). The setup includes six tables: a job tracking table (request ID, session ID, agent name, question, output, completion status, time taken, model used, metadata), quality-related tables (eight quality checks per run — success, latency, cost, tool accuracy, safety — stored as rows;

Method, not a verified protocol.

AI Engineer · Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

Agent query flow: interpret, classify, plan via MCP gateway, then policy check with HITL for write/sensitive operations

Harish Gaggar describes the agent's handling of a query: on first query submission the agent interprets the question (determining the question type and desired response) and creates a plan; each query is classified as a read operation, modification-level change, deletion, or sensitive operation, adjusting the overall structure and plan accordingly. In the plan step (a subgraph), evidence is fetched by making the call through the MCP gateway — ensuring authentication — then calling the MCP server with the right tool to get data and generate evidence. After validating findings, policies are checked: if the operation is read-only, it executes and all record points are saved into audit logs; for write and sensitive operations, the request routes through a human-in-the-loop (HITL) process and

Method, not a verified protocol.

AI Engineer · Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

Governed agent lifecycle: registration, dev portal, repo, local testing, PR, CI/CD, and Kubernetes deployment

Harish Gaggar describes an end-to-end governed workflow: register the agent with its use case and generate an ID; ingest details (framework, language, skills, prompts) into an internal dev portal; the portal creates a repository containing the agent framework plus memory, tracking/eval, context optimizer, and PII blocking frameworks, and also provisions workload identity, memory DB, tracking/eval tables, and secret management; developers clone the repo, build a Docker image, test locally (local memory, Cloud Spanner evaluator, tracing); after two PR approvals the PR registers the agent into a centralized agent registry and the CI/CD pipeline scans the image for vulnerabilities and, with security approval, moves it into the registry; finally the agent is deployed into a Kubernetes cluster,

Method, not a verified protocol.

AI Engineer · Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

Lever 3: downsize the model via component splitting or small-model search loops

The third lever is to downsize the model — with a tight prompt and a measurable search loop, you don't always need a massive frontier model. Strategy one: break a massive prompt into smaller components — use a cheaper model for initial routing or data extraction, and only wake up expensive frontier models for heavy reasoning. Strategy two: plug a smaller model into the same DSPy search loop and let it search its own prompt space to match the benchmark answers generated by your large production model — essentially using JEPA to teach a tiny model to match the task performance of a massive one for your specific dataset.

Method, not a verified protocol.

AI Engineer · How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival

Ship on confidence intervals, not point estimates

Do not ship on a point estimate — you have to account for sampling noise. Calculate a confidence interval around the net gain, keeping the comparisons strictly paired (both versions answering the exact same requests) and keeping repeated runs of the same conversations grouped together. Ship when the lower confidence bound is above zero; if the interval crosses zero, you need more evidence before making the claim. Run the confidence-interval math on your critical categories separately, because a great global average can silently hide that one vital customer intent is tanking. If your critical categories hold up and your lower confidence bound is above zero, you have your answer: ship it and keep measuring. This process replaces the guesswork of prompt engineering with an explicit release.

Method, not a verified protocol.

AI Engineer · How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival

Turn prompting into a search problem with DSPy and an optimizer like JEPA

Instead of manually writing instructions, turn prompting into a search problem. With a framework like DSPy you can stop manually writing instructions: define the exact interface contract — the inputs, the required output schema, and the scoring function — then hand it off to an optimizer like JEPA. The optimizer treats your prompt as a search space, runs against your benchmark to match historical labels, and uses feedback on failures to revise the instructions. Since nobody wants to be the one who deletes a line and breaks prod, you automate the cleanup instead of doing it manually.

Method, not a verified protocol.

AI Engineer · How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival

Instrument the shared model client and build a production benchmark

Alvin's first step: you can't fix what you can't measure. Instrument your shared model client so each attempt becomes part of a trace connecting model calls, tools, and retries for a single task. Record the prompt version, model settings, token usage, latency, and cost in telemetry (OpenTelemetry is a great framework for this). With that telemetry, build a benchmark of how production is behaving.

Method, not a verified protocol.

AI Engineer · How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival