Explore

Instrument everything with OpenTelemetry so agent inputs, outputs, tool calls, and system prompts are visible in any observability backend

The speakers recommended instrumenting everything with OpenTelemetry so it works with whatever observability backend you choose — Grafana, Jaeger, Zipkin, or a third party. In the demo, the k-agent and guard proxy were instrumented, letting them see all inputs, outputs, and conversations with the agent down to a single user interaction: from a single trace, input tokens, input cost, the model used, the system prompt, tool calls, tool results, and output messages. Attendees would see the same in a Datadog backend.

AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog

Tracing: instrument your application, deploy to production, and see traces come in

To do tracing, you instrument your application, deploy your app to production, and see traces coming in of what your agent is doing and how it's interacting with users.

Method, not a verified protocol.

AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse

Monitoring: automatic evaluators, dashboards, user feedback, and manual trace review

Monitoring is done automatically with evaluators and dashboards, but also includes human feedback which cannot be underestimated. Collect implicit or explicit user feedback and manually go through samples of traces.

Method, not a verified protocol.

AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse

Set up an LLM-as-a-judge evaluator: LLM connection, observation filter, input targeting, prompt variables, and JSON path

Set up an LLM connection for evaluators using an API key (in the demo, the OpenAI API key); you can connect all kinds of other models and also use your own proxy in case you have a gateway configured. When creating an evaluator, first filter for the actual observations you want to target — in the demo, the agent (the root of the whole trace) rather than only the generation; evaluators can also target tool calls or generations. In the disagreement evaluator, target only the input, i.e., the user's message. Use curly brackets in the evaluator prompt to pull in context from the trace, and set up a JSON path to target the correct part of the trace — for conversation history, target the whole message array; the mapping pulls together the full interaction: what dad asked, what the tools did, and

Method, not a verified protocol.

AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse

Simple extraction workflow: get a schema, check results, then deploy via API or code snippet

Omar describes the simple workflow: get a schema, look at the results, make sure the results look like what you'd expect, and then deploy — either via a direct API call or a pipeline code snippet you can invoke very quickly. He also advises sticking your documents into the platform, immediately visualizing your data, and productionizing afterwards.

Method, not a verified protocol.

AI Engineer · Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto

CLI and skills setup: install the CLI, create a profile, set up skills (Marketplace for Claude Code)

To set up the CLI and skills for agent evaluation: (1) install the CLI with a single line of code copied into the terminal, (2) create a profile that connects the CLI to wherever your data is stored (in this case, Arize) so it can pull in traces, evals, and annotations, (3) set up your skills, which can be done in a multitude of ways depending on your coding agent — for Claude Code, use the Marketplace. Note: do not run a coding agent with dangerously skipped permissions; it is not good practice and was only done for the demo.

Method, not a verified protocol.

AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

Pass validation errors back to the LLM for self-correction, up to a max number of attempts

Peter Werry recommends taking the validation error and passing it back into the LLM with a message like 'hey, that was wrong. Maybe you can do better,' up to a max number of attempts, so it can self-correct illegal queries.

AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked

Start with vector search, then give the agent tools

Peter Werry recommends starting with a vector search and then giving the agent access to tools, so it has interesting starting data to help it form queries to go fetch the data. He adds that similarity search is really powerful for jump-off points.

AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked

Hide tenant ID metadata from the LLM, validate output, and inject your own metadata

Peter Werry recommends hiding tenant ID metadata fields from the LLM, validating the output to wipe them out, and then injecting your own metadata field.

AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked

Schedule 2-3 weekly gym sessions for strength alongside running

When combining strength training with running, program two to two-and-a-half to three gym sessions per week for deadlifting or general strength work.

Garage Strength · How To Get Strong and Still Be Athletic w/ Sika Strength

Prioritize fundamental strength work over gimmicky brain-training warmups for kids

Get kids to front squat, back squat, clean, snatch, deadlift, walking lunge, and run sprints instead of spending time on gimmicky brain-training warmups; a coach who spends five years on such gimmicks instead of fundamentals ends up worse, not better.

Garage Strength · How To Get Strong and Still Be Athletic w/ Sika Strength

Move agents into CI for OpenTelemetry-instrumented visibility; without visibility you cannot optimize

The speaker notes that if you can't see how people in your organization are actually working with AI tools, you have no opportunity to optimize: you cannot tell whether agents are too verbose, achieving the task, when tool calls happen, or whether they succeed or fail. Moving agents into CI lets you see what's happening because every agent runs instrumented with OpenTelemetry (Otel).

AI Engineer · From Your Laptop to the Pipeline: Scaling Custom Agents with GitHub Copilot

Training setup: ReAct loop environment, Qwen3 4B base, GRPO with binary LLM-judge reward

The training environment runs the agent in a ReAct loop with access to tools to interact with the roughly 7,000 tables created for FinQA. Reward is binary correctness determined by a language model as judge, in this case GPT-5-nano, using reference-based evaluation. The Qwen3 4B model was used as the base model, with GRPO and binary reward for optimization via the RLLM framework.

Method, not a verified protocol.

AI Engineer · Small Models, Big Results: Training a Finance Agent for Under $500 — Charles Dickens, Snorkel AI

FinQA data pipeline: schema extraction, QA generation, and three-layer verification

The FinQA data pipeline starts with schema and data extraction from 10K reports pulled from the Edgar system, using the Qwen3 30B model to produce approximately 6,900 SQL tables. Each table is used to generate a single question-answer pair, guided by a question taxonomy developed with financial experts who work with these documents, along with metadata. The third step is a three-layer verification process: programmatic consistency checks, automated reviews from independent agents, and expert manual review.

Method, not a verified protocol.

AI Engineer · Small Models, Big Results: Training a Finance Agent for Under $500 — Charles Dickens, Snorkel AI

Set up admin approvals, review, and suppression of stale or poison data

Anders Swanson recommends setting up appropriate admin approvals, review processes, and the ability to suppress stale or poison data in your memory system.

endorsed

AI Engineer · From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle

Design the memory storage schema with indexes, graphs, and audit metadata

Anders Swanson recommends a memory storage schema that includes a vector embedding with a vector index for similarity search and a text index for full text keyword search; graph links so memories can be related through an entity graph and additional context pulled back during search; and temporal facts plus audit events (who created the memory, what it was used for, usage log, metadata).

endorsed

AI Engineer · From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle

Implementing hybrid search: branches, fusion, thresholds, re-ranking

Hybrid retrieval can fuse vector similarity search, full text keyword search, relational entities, graph traversal, and feedback signals like importance and usefulness of the memory to yield more comprehensive results than one query branch alone. Vector distance is useful for semantically similar content and a contains operator is very good for lexical search, alongside graph table and temporal data. For facts, you might not want similarity search — you might just want to do an exact ID match on a fact. After running query branches, fuse the results, apply thresholds, and do re-ranking to produce a context card with scored evidence. Apply weights to the different query branches, tuning them as needed in your hybrid search algorithm. The results are merged to an overall score from the dif

Method, not a verified protocol.

AI Engineer · From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle

Redact secrets and PII before content hits the memory layer

Anders Swanson recommends redacting secrets and PII from memory content before it hits the memory layer.

endorsed

AI Engineer · From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle

Scope memory access rights to agent, workflow, team, and user

Anders Swanson recommends scoping memory access rights to the agent, the workflow, the team, and the user.

endorsed

AI Engineer · From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle

Building a dense layer and full networks in TensorFlow

To define a dense layer in TensorFlow from scratch, define two sets of variables: the weight vector and the bias. Then implement the forward pass by computing matrix multiplication of inputs with the weight vector, adding the bias, feeding the result through an activation function (e.g., sigmoid), and returning the output. In practice, use TensorFlow's pre-built dense layer instead of writing this code yourself — for example, calling a dense layer with the number of output units set equal to 2. To build a fully connected network, stack dense layers one after another in a sequential model; a deep neural network is created by stacking hidden layers on top of each other the same way.

Some source atoms are flagged as possible numeric errors (e.g., the two-output-unit example).

Method, not a verified protocol.

Alexander Amini · MIT Introduction to Deep Learning (2022) | 6.S191