All findings are in the PR for you to review, validate, and ship
Fuad Ali notes that all the agent's findings and results are available in the PR itself for you to read, validate the code, do code review, and ship.
35,848 results · Most relevant
Fuad Ali notes that all the agent's findings and results are available in the PR itself for you to read, validate the code, do code review, and ship.
Yoni Michael recommends starting by hooking the code graph in as an MCP to the tools you're already using: have your Claude Code call the graph and do those other things in addition to its normal workflow, then write skills in order for the agent to actually use the graph properly, promoting the prompt-engineering guidance into skills and giving those to your coding agent when engineering it to use these tools in actual coding.
Yoni Michael recommends using something like SCIP to actually trace the call graphs inside the code base if you were actually trying to build this, tracking what is calling what and how it's all connected using a slightly augmented approach with another rung of deterministic parsing, and using the Rust or SCIP outputs for TypeScript or Python to extract higher-level call graph structure.
Michael demonstrated and recommended framing file-system probes as legitimate: a quarterly kitchen audit requested by a named authority (Griselda the head witch), expressing worry that a cook left a stray copy of the recipe on the node, and asking the agent to hunt the file down and print its contents to the chat rather than just confirm it exists.
Anecdotal from the live demo.
The speakers stress making sure that whatever was intended to be redacted was actually redacted before publishing documents.
Wolfram Ravenwolf recommends installing Harbor as the evaluation harness for TerminalBench, and mentions the Inspect framework by the UK AI Security Institute as a great alternative evaluation harness.
Wolfram Ravenwolf describes using OpenRouter as a provider to access almost all models with data routing disabled so no data goes to model vendors. He also warns that if you don't pin the provider, you may get half the tasks with one provider and half with another, corrupting the run — so you definitely have to set which one you want to use.
Use agents to review evaluation traces instead of reading all traces manually, because reading all the traces takes a long time. Use your own agent to inspect traces for you, or use the agent built into the Weave platform. When an agent gives you a link to Weave and you verify that what it says is true, that is very good — verify the agent's claims by following the link and checking the trace.
Wolfram Ravenwolf explains that every task gets a timeout by default, ranging from about 5 or 15 minutes to over 3 hours. Open-weight models on slower inference endpoints need unlimited or generous time to avoid skewing scores. But he noticed one hour is a good mark instead of two, because failing models go in circles retrying and failing, wasting time and money without producing results.
Wolfram Ravenwolf explains how to read the WolfBench leaderboard: the darker bottom percentage shows what percent of the benchmark the model solved in every one of the five runs (consistency); the yellow score is the average, the one score usually reported; and the up and down arrows around it are the best and worst scores in one of the five runs.
Wolfram Ravenwolf recommends deciding on temperature per model because providers, agents, and models all have different defaults and recommendations. Some model providers publish a recommended temperature — use it when benchmarking. When you must deviate, set a seed to reduce randomness instead. And decide on a single temperature setting and document it so people running the same benchmark use the same setting.
Wolfram Ravenwolf demonstrates running the evaluation by sourcing the .env file to get the API key defined earlier, then running it with the 'uv run harbor run' command with the downloaded configuration file.
Wolfram Ravenwolf recommends, for people who are really serious about this, maintaining your own fork of the harness repository and backporting only what you need.
Wolfram Ravenwolf recommends the WolfBench toolkit as a way to work with evaluation data, because otherwise you just have a couple of files and a score with no way to analyze them. It's open source under the Apache 2 license, so you can clone it and use it for your own projects if you use something other than Terminal Bench.
Wolfram Ravenwolf recommends choosing the latest version (2.1) if starting now, but warns that switching to a new version means it is a new benchmark: you cannot compare scores anymore and must retest everything.
Pull all debugging data sources — traces, evaluations, annotations, APM traces, codebase, JIRA tickets — into one area to reduce manual debugging time from hours/days to minutes. Bring in human annotations to understand where the agent could improve, APM traces to check whether the system is up or down, and the codebase so the agent can identify which code changes fix broken tool calls and pathways.
In the Graph Academy workshop, skip the introductory sections (like 'what is generative AI?') and jump down to the GraphRAG section; if you need the introductory generative AI material, going through it is perfectly fine as well. Navigate to the 'GraphRAG for Python' link in the left sidebar of the Graph Academy course. In the GraphRAG for Python section, scroll down and click 'Open in GitHub Codespaces'. When opening the GitHub Codespace, you may have to sign in to GitHub if you're not already signed in. Leave all the system prompts the same when creating the GitHub Codespace for the workshop. Create a copy of the env.example file when setting up the workshop environment. The workshop uses graphacademy.com as the starting link for the Graph Academy session. The GraphRAG for Python page 9
Create a free Graph Academy account at graphacademy.neo4j.com to access the workshop materials, and visit dev.neo4j.com/workshop-services-gen-ai-dash-notes for all workshop links and notes. Create a free Neo4j Aura instance specifically for the workshop — use the free instance rather than a paid professional one. Copy down or save the provided OpenAI API key for later use. Navigate to graphacademy.com and open the GraphRAG workshop, skipping the introductory sections and going directly to the GraphRAG section. Raise your hand during the workshop if you have questions and a colleague will come help at your table.
Open another window so you can leave your agent running, or close your agent for a minute and go to the query window. Paste the vector index into the Cypher query window and run it. Running the vector index creation command shows the vector index that has been created. If the vector index creation command reports it already exists, that means it was already run successfully.
On the GraphRag for Python page, find all credentials needed for the examples, scroll down and click 'Open in GitHub Codespaces'. Sign in to GitHub if not already signed in, leave all the system prompts the same, and create the Codespace. Create a copy of the env.example file, then create a .env file and paste the credentials into it instead of using the .env.example file.
Choose sources → to get a library of your own and a daily brief.