Add LLM-as-a-judge to score generated question quality
You can add LLM-as-a-judge to score how your model is doing — e.g., another model judges the quality of the question the previous model created.
40,602 results · Most relevant
You can add LLM-as-a-judge to score how your model is doing — e.g., another model judges the quality of the question the previous model created.
Create questions that are difficult so they invoke reasoning and tool calling from the model.
Annabell Schäfer advises leveraging coding agents to 10x yourself, but staying in the loop and not letting the coding agent do everything on its own.
Lotte Verheyden advises treating your data sets as structured test cases, heavily inspired by what you see during monitoring.
Annabell Schäfer advises deciding what is interesting to monitor for your specific application rather than monitoring generally.
Register for a free Weights & Biases account to follow along with the workshop; the account comes with free inference credits you can use to run evaluations. If you prefer, use OpenRouter or a different provider instead of the default, and tell your AI agent to change something and make it work with your setup.
Wolfram Ravenwolf recommends being very clear what you are using when publishing your results, so others know what you actually measured.
Do not use LLMs for the validation/sanitization part of the pipeline; write procedural code instead. Technically, the sanitization and validation implementation is not that difficult — you just have to write the code, and it's all procedural.
You can express a structured query interface through MCP, even dynamically updating the tool every time the schema changes. You could also provide an MCP interface that takes natural language queries and translates them on the back end, or a direct interface to the database, depending on what the customer interface looks like.
partial
Jesse Hall recommends testing the conversation rather than individual components, and continuing to test in production.
Patterson advises that before adopting AI agents at a company, you need security approval from CISOs and buy-in from DevOps, since they're the ones who have to deploy it to teams. You should talk to security people about what the CISO is concerned about and how to put boundaries around agents.
Patterson urges teams bringing AI into an organization to rethink their architecture to protect against agents dropping databases, deleting personal information, uploading customer information, or extracting company IP to the public internet. He also recommends setting up AI agent rollout in a repeatable, consistent, reliable, and secure way for everyone at the company, and going slow enough to think about architecture and know what infrastructure actually needs.
Pedro Lopez's guidance on choosing between MCP and CLI: MCPs are great for non-technical users to quickly prototype things, build one-off reports, and leverage the MCP ecosystem; the CLI is better for more technical users and longer-running tasks. Both have their place.
Derive eval scenarios from production rather than imagination. Creating synthetic data might be very important at the beginning of the process, but as soon as you start getting customer signals, make sure they feed your eval.
Matt Gibiec quotes his CTO Bernd: 'The agents operate at speed but without sight' — agents are fast but are only as good as the data available to them for reasoning decisions.
When the prototype preview misbehaves: hit refresh (stale context is a common cause), switch to present mode, check whether images are too large, check naming or text differences, verify the relevant frame setting is turned off, and test whether renaming affects the prototype.
partial
When passing a prototype to Figma Make, build section by section in small chunks rather than giving it a massive prototype and saying 'just build this'.
The host recommends valuing full self-expression, claiming that when you do, you make bolder decisions than other people, and suggests putting your art out there and risking ridicule instead of calculating social costs.
The host recommends preferring offense over defense as one of the three core charisma traits, framing it as a stance to adopt rather than a defensive posture.
Lore advises founders to know who they are, what they stand for, their values, and what type of people will work well in that culture and system — then hire those people and be really consistent.
Choose sources → to get a library of your own and a daily brief.