RecommendationFor LLMs we control the training data, pre-train, RLHF, RSI, and all sampled data, so we can take out the bad data — e.g., run a small model over pre-train data to remove bad stuff (as Percy is doing). If bad stuff is removed, it's never in the pre-train, so the model can't possibly learn those bad things. After cleaning, do a failure analysis on the trained data: estimate failures, sample some, estimate failure modes, and get to a p fail (or p success).
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
RecommendationDeploy an always-on AI system that monitors traffic, data, and system activity to detect anomalous behavior such as unusual wire transfers or keys being shipped to unexpected domains. When it detects anomalous behavior that has no hard-and-fast rule but looks wrong given enough context, flag it to humans for review. AI watching the system can do a much better job than traditional alerting because it can be highly targeted and reduce alert fatigue — traditional systems produce a thousand garbage alerts nobody looks at, causing 'death by a thousand alerts' where real issues get missed. AI can also serve as a watchdog to catch social engineering attacks.
Also discussed by Jiachen Jiang on AI Engineer (unclassified source — not counted until classified). Not counted as independent sources.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
RecommendationApply version control, pull requests, and issue filing to benchmark development, and curate benchmarks through a vetting process rather than letting anyone publish one — like Apple's App Store review. A curated 'Apple App Store of benchmarks' is floated as a great startup idea.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
Recommendation · SocialCoach Kyle recommends recording your conversations and breaking down the recordings so you can hear what you sound like and identify where you're going wrong.
Coach Kyle · Don't know what to talk about with women? Watch This
Recommendation · SocialCoach Kyle emphasizes that adding the tiniest bit of laughter after a fun, playful tease provokes a much bigger emotional response; without it, she'll just flatly answer literally (e.g., 'No, I was just at work').
Coach Kyle · Don't know what to talk about with women? Watch This
Recommendation · MoneyMake the cheaper model also the faster model so users naturally switch to it for speed, not cost. When teams the speaker worked with kept both models at the same tokens-per-second speed with budget in excess, no one switched to the cheaper model; conversely, when teams set an extremely low budget so the smarter model burned through it in one day, everyone switched to the faster model.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneyThe speaker generally tries to split the data set because people are lazy filter, so he tries to at least get a golden trace data set, and the golden trace data set maybe is much smaller respectively, and then they fine tune that with a much heavier weightage as well. There have been multiple cases, especially with thinking tokens, that even the failed task, a much smaller model when you fine-tune tends to benefit from the training because it actually improves the thinking process, even if it got the wrong answer. This is really a good case of try and then measure, and it works 70% of the time. Sometimes you need to increase the golden trace ratio, but it's a general rule of thumb. Golden samples are critical.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneyRoute incoming calls to a much smaller model and escalate to larger models via classifier triggers. Even after fully splitting to a smaller model, cut a portion of traffic as shadow traffic to the best model — this lets you ride the wave of progress of the best open models.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneySet up faster tokens-per-second for human users but much slower and much cheaper speed for the CI/CD pipeline.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneyA common way to incentivize teams to change agent harnesses is to introduce a good enough model that is really fast, like 100 tokens per second. In a common transition pattern, teams keep Claude Code as their planner and instruct Claude Code to use the Pi agent with a specific model for implementation — the planner-implementation split.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneyA simpler offloading approach is having Pi installed and running the bash command 'pi this'. MCP agents are tricky for offloading because, unless done right, they don't have access to your file system — per an audience member's suggestion to expose a smaller agent as an MCP inside Claude.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneyIt's not that hard to just download 10 models from Hugging Face and just try them to find the cheapest model that passes most of your evals. Fine-tune a domain model for your use case if the model is much smaller, especially for your domain.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneyUsing department model labels gives you more control when swapping models out. Add a dash-preview variant of a labeled model to get control without taxing the communication burden across the organization. Set up labeled model options in place because without them no one switches models. The downside is users may complain when you swap models under labels and get it wrong.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneyNacho Martinez recommends that when working in a specific domain like frontend, you create a skill capturing what worked well including your preferences (e.g., coloring style), so the agent can reuse it — and because skills are vectorized, mentioning a phrase like 'coloring style' makes the agent automatically retrieve the relevant skill.
Also discussed by The Official SaaStr Podcast: SaaS | Founders | Investors (unclassified source — not counted until classified). Not counted as independent sources.
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyIf the agent considers something in short-term memory or the context window worth keeping, promote it to long-term memory and put it somewhere as raw text — that's the easiest thing you can do. Once the long-term memory is in text somewhere, you can take it, vectorize it, and let it be explored by the agent loop in some way.
creator's recommendation
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyNacho Martinez recommends adding an extra step between observing and reasoning that lets the agent recall memories, making the agent introspective before solving the problem.
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyNacho Martinez recommends setting a maximum iteration cap (8 in the workshop harness) and a maximum of 60 seconds of reasoning per iteration so the agent doesn't run out of OpenAI credits.
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyYou are not using a re-ranker in the workshop, but you could put an embedding model plus a re-ranker at the end of the similarity search that takes the three retrieved skills and tells you which is the most important one. Adding a re-ranker gives better refinement because a task might be solvable by only one skill rather than a total of three loaded for every task.
creator's recommendation, not used in the workshop
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyNacho Martinez recommends using a gateway and MCP inside the data layer to connect agents to third-party external data sources.
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyNacho Martinez recommends adding an embedding model plus a re-ranker after similarity search: the re-ranker takes the retrieved skills (e.g., three) and identifies the most important one for better refinement.
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle