Report · MoneyIn the demo, the audit log couldn't tell who was using the API keys — credentials weren't attributed to a user or an agent's identity — and the agent had unrestricted access to any and all permissions granted by those keys.
AI Engineer · Your Agent Is a Lethal Trifecta (And What to Do About It) — Kim Maida, Keycard
Report · MoneyKevin describes pointing a coding agent at his team's cloud sandbox hosting environment with the instruction 'Try to break it' and letting it run overnight to discover vulnerabilities.
AI Engineer · https://www.youtube.com/watch?v=k72eLcYpVys
Report · MoneyThe speaker reports that most code generated with AI coding tools has security issues that you can find by scanning it afterwards, and that unless you tell the AI in the prompt to generate functional tests, integration tests, and documentation, the tool may not do it by default. Vibe coding gives you fast, functional software but does not give you secure software by default. AI coding tools are evolving quickly and are much better now than they were six months ago.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Creator perspective · MoneyThe speaker argues that today it may be more important to build for agents than for people, framing agents as a new consumer class, and that going forward builders need to think about making something agents want in addition to something people want.
hedged (speaker frames this as a possibility, not a certainty)
AI Engineer · https://www.youtube.com/watch?v=nSkmXakJ6eo
Creator perspective · MoneyThe conference app is built by an external vendor they purchase, and swyx says next year he wants to rip it out and build their own. He also notes he has never liked a conference app anyone else made — he only likes the one he makes — and assumes it's the same for every attendee.
swyx's dislike of other apps is his own stated preference and an assumption about attendees
AI Engineer · Building AIE: Open Q&A and Feedback Session with swyx
Principle · MoneyYou can install browse CLI and use all of these skills completely for free; just give the browse skills CLI to your agent (e.g., open claw) and it knows when to pull the right website skill (e.g., alltrails.com).
AI Engineer · Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase
Principle · MoneyThe fast path for browser agents is to encode the whole query in the URL — strongly preferred. Build a search URL with date math and known metro pins (e.g., San Francisco) encoded in it; if a site blocks a path, use query parameters (latitude, longitude, metro ID) to reach the working UI directly. If you avoid blocked paths and follow this learned technique, your agent will work.
AI Engineer · Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase
Report · MoneyAmazon Nova Act is a first-class AWS service that lets developers build and run reliable browser-based workflows at scale. Using natural language, developers can drive a browser workflow to populate a browser, navigate, search, and fill out forms, and Nova Act can call a human in the loop when needed, such as for expense report approval or when the agent gets stuck.
AI Engineer · Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab
Principle · MoneyRisk must be defined in synergy with all other departments so the agent runs under the organization's risk blueprint. In the example, the agents measured score 97% on risk because risk compliance is defined for the agent, and compliance is evaluated by checking whether the agent is compliant with SOC 2 and bank policies. Observability scores well when the agent logs everything: input, reasoning, decisions, and then call integrations. Accuracy in the example is measured with 3% hallucination, part determined automatically and part via manual annotation, and is tracked over time. For conversational agents, tone and sentiment should be observed and extracted from the conversations. Drift scores well when the agent is still within the boundaries established with the first go-live date, tracked
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Principle · MoneyVLM agents are probabilistic models that produce a probability each time, so they won't give the same performance running 24/7 every day. Running the same agent three times degrades performance by almost 10% success rate even for the best-performing agent, and perturbing the environment (slightly different queries and environments each run) makes reliability even worse. None of the current agents (Codex, Claude) produce reliable repeatable processes.
publisher-reported OS World numbers; benchmark context
AI Engineer · The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, Simular
Creator perspective · MoneyFrame the agent prompt as 'you are an expert search agent that uses web search to answer questions accurately, provide valid JSON with the final answer, supporting URLs, and a short justification.' Include usage guidelines so the agent doesn't call the search tool excessively — calling the tool 100 times and spending $1,000 on an easy question is an anti-pattern. As a heuristic, keep search agent hops between 4 and 8 to prevent going off topic.
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Creator perspective · MoneyUse LLM-as-a-judge to score each trajectory and track keep percentage alongside word count metrics. Wikidata seeds can become stale, so verify answers against current web sources before trusting them — trusting a verifier's correct answer without cross-checking can produce stale training data. Also, implementing a simple matching function for evaluation will yield low percentages (e.g., ~24%) and make you think your data is terrible when the evaluation is the problem.
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Principle · MoneyYoni Michael says you can build your graph context in an afternoon, but the hard part is actually making the agent use it, which requires prompt engineering, evaluations, and hard work.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Report · MoneyIn the live demo, Michael reported that the Burrito Bot with no guardrails could be talked into deploying a public image to the Kubernetes cluster, and that an attempt to get it to exfiltrate data outside the cluster to a public endpoint actually worked at least once (though the model later stopped allowing it regardless of iteration).
The exfiltration attempt worked at least once, possibly a few times; the model later stopped allowing it regardless of iteration.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Report · MoneyThe Langfuse workshop website offers a step-by-step guided walkthrough of all eight modules, including prompt management and dataset setup. If starting tomorrow, use the Langfuse skill with your coding agent to set up tracing first: take your repository and use the skill command to get set up with tracing so you know what's happening in your application. Go to the monitoring module on langfuse.com/workshop to work with traces during the workshop.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Report · MoneyDo a small test run of the evaluator on the right side before executing it; once happy with the test run results, click execute to get a live running evaluator that processes incoming traces. The test run of the all-caps evaluator returned false when only asking the Bluetooth on/off question, because no frustration was present.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Report · MoneyThe presenters describe installing the Langfuse skill via your coding agent (the speaker usually just tells their coding agent to install it, though it can also be installed separately) and note the Langfuse CLI exposes all APIs — traces, scores, datasets, experiment runs — so you can access and start them from the command line or with a coding agent. The skill can also automatically add tracing to your application when instrumenting it for the first time.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Report · MoneyThe first step when wiring up evaluators is configuring the Langfuse evaluator model. For LLM-as-a-judge evaluators, an LLM connection with an API key (e.g., OpenAI) must be set up first; you can also connect other models or use your own proxy if you have a gateway configured yourself.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Creator perspective · MoneyOverthinking causes models to fail time-boxed agent tasks: the model thinks for minutes before each command and runs out of its one-hour budget, while a non-thinking agent just tries commands, gets errors, and gets more turns. A model being too smart for its own good also causes failures — Fable decided to do tasks its own way even when the benchmark specified otherwise, and the verifier then failed it. A smaller, faster agent reaction could work better than maximum thinking when the model thinks too long.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Principle · MoneyWolfram Ravenwolf suggests that if you want to follow along and use OpenRouter because you didn't get your own API key, you could use OpenRouter as an inference provider. He recommends putting the API key variable in an .env file and using that with the agent — source the .env file to get the API key before running. He stresses that if you use OpenRouter, set special settings to use the W&B provider — this is very important when benchmarking a model via OpenRouter. OpenRouter by default routes to one of many providers they decide is best at the moment, so a single benchmark run with 89 tasks may get half the tasks on one provider and half on another. He advises definitely setting which provider you want to use when benchmarking through OpenRouter.
endorsed
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases