35,848 results · Most relevant
The speakers recommend preserving what excellent human performance looks like (e.g., how a human would answer a help ticket well) as 'receipts' and rolling this concept into more applications.
Martone recommends building a full spectrum of AI tools — frontier LLMs, custom deep neural networks, and simple ML models — and making deliberate per-problem decisions about which to apply, rather than having one solution for everything.
endorsed advice
In Phoenix you can see the results of the various evaluations you've run (correctness, faithfulness, human actionable, mentions ticker) in the annotations column; filter traces by any eval, sort by score to find your worst runs, and click straight from a failing score into the step that caused it — from score to cause in one click.
Use Allowed Tools to give the agent web search so it can find current financial data. Set permission mode to Don't Ask so the agent runs non-interactively. Add a couple lines of code to ensure the Claude SDK runs in its own isolated version and doesn't steal your keys and data from your Claude Code copy.
Include edge-case queries (two-company comparison, non-public company like Rivian, different industry like Coca-Cola, fake ticker ZZZQ) and slang-style queries like 'yo, is Tesla a buy right now?' because real people send queries in slang without specifying exactly what they want; vary the same query in multiple ways.
You don't always have to write your own evals right off the bat: Phoenix ships correctness, faithfulness, completeness, hallucination, refusal, PII, and toxicity checks, plus tool-use evals for agents (did it pick the right tool, pass the right arguments, do something sensible with the result). You can write code evals to get a bunch of stuff out of the way and use built-ins for the obvious stuff without inventing your own tool-calling correctness evaluator.
There is no built-in eval that checks for a buy, sell, or hold recommendation because that would be a very specific thing, so you write your own judge for application-specific evals.
Next to each faithfulness result you get an explanation — read it to understand why a report was judged unfaithful, e.g., the Microsoft explanation flagged claims the context does not support and fabricated 2027 targets not in any search results; the Azure claim and backlog-visibility claim were unsupported by context. A faithfulness judge can also accurately detect reasoning errors like the NVIDIA case where the response conflated two different data points.
If you don't have users, get an LLM to generate lots of test queries and vary them on purpose, then get a human to review the list.
Run the agent with npm run agent -- Tesla 'growth outlook' to test it. Run npm run batch to execute 15 queries in a row — one test is not enough data to draw conclusions. If your batch run fails, use the seed project Phoenix 101 Golden which has 16 pre-run results.
The simplest useful eval is checking whether the report actually mentions the stock asked about — if the agent writes about the wrong company, the report is completely useless. It costs nothing to run and catches real errors, like the agent that got focused on AWS and wrote a report only about AWS instead of Amazon.
Use code evals for deterministic checks like: did the output parse as JSON, is it under the length limit, does it avoid phrases you never want to see like 'as an AI language model'.
Laurie Voss sets a standard of actionability for financial reports: they must tell you whether to buy, sell, or hold the stock, without leaving it ambiguous. He illustrates the failure with the Coca-Cola report, which called the stock a good investment without saying buy, sell, or hold.
Use Phoenix, Arize's open source AI observability platform, to collect traces from your AI agent. Phoenix is open source, so users can mess with the code and report mistakes back to the team.
Use a tapering lift duration schedule: 25-30 minutes furthest from game day, then 20-25 minutes, 15 minutes, and 10 minutes closest to game day.
Test athletes on force plates once per week during a long season to track fatigue, monitor force plate jump metrics across consecutive weeks to detect travel-related fatigue drops, and cross-reference force plate data with a wellness questionnaire, Hawkeye or Catapult distance-covered data, and subjective hamstring soreness when assessing athlete readiness.
Assess site-specific/tissue-specific metrics based on athlete injury history, especially for older or transferred athletes: test hamstring/posterior chain strength with Nordics (Nordbord), groin strength with Force Frame for soft tissue information, and pitchers' internal and external shoulder rotation strength with Force Frame.
Use the Nordbord, Force Frame, and 1080 as training modalities, not just testing tools. The 1080 is primarily a training modality and in-house equipment, not a league-wide testing standard. Use the Force Frame for arm care isometrics at 30% and 50% of max as a safe training method, and use the 1080 for plane-specific missions like frontal plane work or linear sprint training.
Grubbs tells people reaching out to her that you can get into the field knowing no one, but you have to be strategic and tactful about where you pick to do your internships. She advises looking at the 'Clemson tree' and the projectability of a program — where you'll go next — noting projectability is super important.
Give infielders more change-of-direction (COD) type work, and give outfielders more high-speed running based on distance covered.
Choose sources → to get a library of your own and a daily brief.