Explore

Disaster recovery is mandatory for banks, and ClickHouse lacks managed failover

Al Chen explains that disaster recovery is not optional infra hygiene for a bank. The main database platform most AI observability tools use has no managed service for failover across regions and data replication: Redis and Postgres have managed failover across regions, but ClickHouse does not — even its own data resiliency docs call this out. Because of that, banks are left to figure out the disaster recovery situation themselves.

described

AI Engineer · Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco

Backup design is implicitly a data residency decision

Al Chen calls out that architects creating backup and restore within their cluster are also making a decision around data residency, whether they like it or not. When replicating to a bucket, the bucket doesn't care if it's in Oregon or Virginia, but data residency cares about where individual records are stored. One solution: replicate just the IDs and metadata around traces and data into different regions, while the actual core trace data stays within one region because of jurisdiction laws.

endorsed

AI Engineer · Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco

Manual trace review remains important but doesn't scale to production volume

Doneyli argues that manually looking at traces is still a very important process despite its cost — Annabell spent days doing it — but that once a system is in production it is not feasible for a human to look at thousands of traces. The resolution is to curate a representative sample of production traces for a human expert to review.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Banks face financial and legal liability from non-deterministic LLM agents

Banks face financial liability, legal liability, and a bunch of risk when putting out LLM-powered customer-facing solutions because LLMs are inherently non-deterministic. When evaluating a customer-facing agentic solution, leadership or teams will ask how to evaluate it after the first prototype or demo is built. Reduce the unpredictability of LLM-powered agents so you can reproduce outcomes as much as possible and figure out failure modes.

described

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Isolate variables when experimenting: change one thing at a time

When testing whether the same model performs better with a different prompt, change only the prompt and not five other things. Changing multiple variables at once makes results hard to interpret; isolating concerns lets you see whether a change better handles your existing data.

AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Keep the process deterministic on the outside and adaptive on the inside

Evan Kormos describes the recipe for bringing agents into existing workflows as keeping the process deterministic on the outside and making it adaptive on the inside.

AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase

The loop is only as trustworthy as the check — define what good looks like

Define what good looks like at the check; that is the hard, reliable way to evaluate whether work is good enough to move forward. Without defining what good looks like, more passes don't necessarily mean better results. The loop is really only as trustworthy as the check.

AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase

The recipe: keep it deterministic on the outside and adaptive on the inside

The recipe for bringing agents into existing processes is to keep it deterministic on the outside and make it adaptive on the inside.

AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase

Trust is not a feeling about a model — evaluation provides research and statistics about agent quality

Quality through evaluation establishes a complementary confidence mechanism, so trust is not a feeling about a model but research and statistics about the quality of agents. Providing research and statistics on agent quality builds confidence and inherently trusts output. Relying on a feeling about a model instead of evaluation data undermines confidence in agent quality.

AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase

Agents are probabilistic: exact output comparisons aren't good enough for evaluation

In traditional software, confidence that software is high quality is built primarily and systematically with test cases and unit tests checked as part of the delivery flow, so code changes don't break other things. Agents are probabilistic, so the same requests can produce different answers, and more than one answer may be acceptable. Exact output comparisons alone aren't good enough for evaluating agents; you still need evidence.

AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase

Harness choice significantly affects cost per task

Both harness studies (UC Berkeley and Apple/EPFL) found a significant difference in cost per task even though eval scores did not differ by harness.

AI Engineer · Smarter Models Killed the Harness. Here's What's Left — Jediah Katz, SpaceXAI

Multiple-choice discovery question steers toward solvable problems

In discovery, offer options A, B, and C that must all be problems you can solve — that's how you steer the conversation toward where you can help.

30 Minutes to President's Club | No-Nonsense Sales · #614: How to Use LinkedIn Sales Navigator for Lead Generation

Reassess your career when your children are born and pivot gradually

When his twin sons were born, Diamond realized their future did not depend on gallbladders, reassessed whether his career served their future, and gradually shifted his time toward history and geography rather than making an abrupt switch; he also reduced his New Guinea trips from three months to one month at a time.

anecdotal

Andrew Huberman · What Makes Great Leaders & How to Live an Extraordinary Life | Dr. Jared Diamond

Devoting an entire career to one narrow specialization is an anti-pattern

Diamond describes realizing with horror after his PhD that he was expected to devote himself to the gallbladder for the rest of his life, which did not fit him, framing lifelong narrow specialization as contrary to broad intellectual development.

anecdotal

Andrew Huberman · What Makes Great Leaders & How to Live an Extraordinary Life | Dr. Jared Diamond

Scale and time window determine whether individuals matter in history

When evaluating a leader's impact, consider the scale and time window of analysis: individuals matter on local scales but not on continental scales over 10,000 years. When analyzing historical success, distinguish individual agency from structural advantages like geography, timing, and platform power.

Commercial context: The creator describes this as their own product.

Andrew Huberman · What Makes Great Leaders & How to Live an Extraordinary Life | Dr. Jared Diamond

Plan your remaining finite time around a deliberate synthesis project

Diamond reasons that his remaining time is finite, asks what he will do with it, and has committed to a specific synthesis project for the next ten years of his life; he also urges starting long-desired learning projects now rather than deferring them indefinitely.

anecdotal

Andrew Huberman · What Makes Great Leaders & How to Live an Extraordinary Life | Dr. Jared Diamond

When a career conflicts with your interests, shop around and evaluate alternatives before committing

Diamond shopped around for another career when his path conflicted with his interests, seriously considering multiple fields including pre-Columbian ceramics, orchestra conducting, and being a simultaneous translator in Switzerland, and tested his aptitude against specific skill requirements (e.g., conductors must ignore time lags).

anecdotal

Andrew Huberman · What Makes Great Leaders & How to Live an Extraordinary Life | Dr. Jared Diamond

Maintain multiple careers simultaneously rather than abandoning one

Diamond has kept up two careers in history and geography simultaneously rather than abandoning one when starting another, and persisted through administrative difficulty to get UCLA to transfer him between departments.

anecdotal

Commercial context: The creator presents this as their own offering and directs viewers to get it.

Andrew Huberman · What Makes Great Leaders & How to Live an Extraordinary Life | Dr. Jared Diamond

Anthropic uses a different API and has built-in citations

Pamela Fox reports that with Anthropic you have to use a different API (the Messages API) so you have to switch your code around, but Anthropic models have citations built in — you provide sources in a structured format and enable citations, and the models are trained to return back blocks of text with citations that you can extract programmatically, which is really nice when you're building frontends because you can just link your citations.

AI Engineer · The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

Agents loop on tools; efficiency is the differentiator

Pamela Fox explains that an agent calls tools in a loop until it reaches a goal. A word problem like a store sale with shirts and tax can be solved either with multiple tool calls to a calculator (one small expression at a time) or, if the LLM is clever, by batching everything into one long expression in a single tool call. Once you have tool calling in a loop, the differences between models is often their efficiency: are they able to get to the goal faster and accomplish the same goal with fewer tool calls? She builds up to agents by starting with the most low-level tool calling possible before abstracting on top of it.

AI Engineer · The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft