Explore

Follow the same eval pattern: get runs, evaluate, log results back to Phoenix

Every eval follows the same pattern: get the runs, evaluate them, then log the results back to Phoenix via log span annotations, which attaches each result to its span. The correctness evaluator is one import and one line of setup; call evaluate to get back a label, score, and explanation, and log results the same way as the ticker check. Phoenix evals in TypeScript use the Vercel AI SDK to talk to models, so any model the AI SDK supports can be a judge.

Method, not a verified protocol.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Deliberately include accountability gates in harness design

When designing a harness, include on purpose those gates so you can still be accountable for what is being produced. Make sure any important judgment call or disambiguation — not the trivial ones — you can still get hold of. Without accountability gates, the only answer to 'what happens?' is 'the stochastic machine said it was fine.'

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Lift on game day during the 162-game MLB season

Because professional baseball players compete nearly every day across a 162-game season, lifting must happen on game days; if you skip game-day lifts, you won't lift at all during the season.

Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs

Measure token spend per department and per team

Back to the first step: measure. Know the token spend per department and per team so you can see which team is performing better, how much tokens they use, and what works.

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Build fine-tuning datasets with an expensive model at small scale, then distill to a smaller model

A common approach: deploy an expensive model at small scale, generate your dataset from it, then reduce dramatically to a smaller model (e.g., 7B scale).

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Split API keys by department and track spend at the use-case level

Split API keys by department so you can see where token spend is going, and track it down to the team and use-case level rather than just organization level. A single shared API key across the whole organization makes token spend unmeasurable. Featherless AI is open-sourcing a tokenomics tracking tool that splits API keys and tracks per-key spend.

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Set a budget on agent flows to prevent runaway sessions

Set a budget on agent flows based on whatever your budget might be to prevent runaway sessions; relying on Claude alone to detect misbehaving sub-agents without a budget risks runaway sessions. When enabling multi-agent sessions via the API, set up a budget, and know you can extend it if you end early without spending it.

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Copy a specific goal into every sub-agent's context to prevent drift

Set a very specific goal at the beginning of a flow and copy it consistently into each sub-agent's context across all phases to prevent goal drift.

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Monitor runs via reports, outputs directory, durable event stream, and thread messages

Prompt Claude to give you reports or update a specific report as the run progresses (typically markdown files); have Claude write to its outputs directory so you get durable downloadable files via the v1 files API; stream the durable event stream in real time or list events via a static API endpoint to monitor what agents are doing and decide whether to take action; use thread message sends and receives to inspect inter-agent communication and verify middle-management agents aren't doing something unwanted with their underlings.

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Kick off long-running multi-flow projects and walk away; chain flows for large projects

Kick off long-running multi-flow projects and walk away, checking back later rather than waiting synchronously; when a lead agent's flow finishes, start another flow to chain large projects across multiple flows, and a lead agent with full state of where the flow is in its process can inspect where it left off and kick off a new flow incorporating both new information and past history.

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Use abstractions so credentials never touch the sandbox

Use the suite of abstractions that ensure credentials never have to touch the sandbox — an incredibly important decoupling. Claude takes on the persona of whatever credentials you give it, so credential handling determines your agent's effective identity and permissions.

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Interrupt a running flow to correct direction or inject new information

You can interrupt a running flow session to correct its direction mid-run with a direct stop command, and when new external information changes the trajectory of a run, interrupt the session and provide that information as a direct instruction referencing the new material, rather than burning two days of tokens and starting from scratch. Note: you can't inject a single step halfway through and rerun later steps.

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Fan out one agent per service for health checks

Fan out across all services and dedicate an individual agent context to each service so each agent is only responsible for its small scope, yielding a much better answer and guaranteeing every service was checked.

AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

Combining layers: adopting a teammate's exploration

When a teammate's exploration contains something you like, select both their layer and yours, then ask the agent to combine them into a new code layer, naming the exact feature to take from each (e.g., 'create a new code layer, combining the grid clustering from the grid layer and the delivery feature from the delivery code layer'). The result is a single code layer with the best of both explorations.

Figma · Explore ideas in code | Figma

Editing extracted layers and pushing changes back to code

Extracted layers are fully editable — move components, adjust typography, or make other design adjustments — and changes can be pushed back to the code layer so they make their way into the code.

Figma · Explore ideas in code | Figma

Requesting multiple design explorations at once

When exploring different design directions, ask the agent for multiple explorations in a single prompt; specify one variation explicitly and let the agent surprise you with the others. Example prompt: 'Make three variations of this and add a new delivery button to the map page. When clicked, the user can choose their address. The first one should be an Omni search address field, but surprise me with the other two.'

Also discussed by Kova on AI Engineer, Magic Patterns (guest) on Y Combinator (unclassified source — not counted until classified). Not counted as independent sources.

Figma · Explore ideas in code | Figma

Pushing a code layer to a branch or pull request

Push a new code layer back to a branch or create a pull request so others can decide to merge it into the repository, contributing the exploration back to the codebase.

Figma · Explore ideas in code | Figma

Model, reward, and punish to create culture — and cascade it through leadership

The speaker advises business owners to create culture by modeling the behavior they want, rewarding the behavior they want, and punishing the behavior they don't want. Because reinforcement happens 24-7, 365 — including in teams the owner isn't involved in — the same behaviors must be cascaded down through leadership so leaders reinforce them further down. He frames owners as the chief reinforcement officers of their organizations.

Alex Hormozi · My honest advice on how to raise your team’s bar

Validating pocket binding via competitor displacement and pocket-breaking mutations

Leash Bio routinely validates that hits bind a specific pocket with a two-step workflow. First, run a competitive assay: buy a known pocket-binding chemical, add it at high concentration, and re-screen — chemicals that normally bind the pocket will be displaced. Second, mutate the pocket residue: change the DNA construct to physically substitute one amino acid in the pocket (e.g., insert a big bulky side chain to fill it), then re-screen. Chemicals displaced by the competitor should also disappear when the pocket is broken, giving strong evidence they all bind that pocket. To confirm a mutation didn't globally break the protein, check that only the chemical series kicked out by the competitor disappeared while all other chemicals binding elsewhere remain intact. Quigley notes pocket-breaks

described as routine Leash practice ('we do this all the time')

Method, not a verified protocol.

Latent.Space · Lab-in-the-Loop at Compute Speed — Ian Quigley, Leash Bio

Context optimizer pipeline: shape, prune, re-rank, dedupe, compress

Harish Gaggar describes a context optimizer framework that shapes and prunes queries before they reach the LLM: select the payload based on the token limit, re-rank for the most relevant information, dedupe to a unique curated set, and finally compress the large payload before sending it to the LLM.

Method, not a verified protocol.

AI Engineer · Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma