Treat eval prompts like code: version and test them
Treating eval prompts like code (not versioning or testing them) turns evals from a useful tool into noise because small wording shifts cause dramatic result changes.
40,602 results · Most relevant
Treating eval prompts like code (not versioning or testing them) turns evals from a useful tool into noise because small wording shifts cause dramatic result changes.
Do things by hand first so you have a fundamental understanding of what it is that you're doing and you're not just vibing your way to success. Then start making your own changes to the prompts in your own notebook to improve on the basic ones provided.
Leaders, especially the CEO, must personally and constantly operate as AI-native and demonstrate it to the team through their own actions. When a customer describes a frustrating problem, the CEO should personally build a solution over a weekend. This is more effective than issuing ultimatums like '10x or you're out'.
Instead of asking 'how many fewer people do we need now that AI makes us more efficient?', ask 'how much more could we build now and how many more customer needs can we meet?' The right answer is more high-agency people and more builders, not fewer.
Derrick Reimer and Rob Walling advise against hiring a developer who has only worked at large companies (e.g., Fortune 500) because the change of mindset and pace is dramatic, and you may not have the luxury to reorient them to a small-company culture.
confident
Derrick Reimer suggests evaluating developer candidates on three dimensions: technical acumen, interpersonal dynamics (flexibility, team player), and reliability (shipping on time, doing what they say). Reliability is the hardest to test and requires working with the person.
confident
Erik Allebest advises founders to pick a tiny market, ignore conventional founder advice, listen to their heart, define the minimum resources needed to prove their vision, find great people to work with, and stop listening to advice and just execute. He notes that one curse of being a founder today is the abundance of founder advice.
Erik Allebest recommends using AI tools for research and deep-diving into subjects for faster learning, but warns against relying on AI to do all the thinking, as it can stop you from engaging your brain. He notes that there are no shortcuts to practice and repetition, and that acquiring expertise requires putting in the repetitions to store knowledge in your brain.
Ofir Ehrlich and Gonen Stein advise organizations to assume a breach is inevitable, whether malicious or accidental. They must prepare for both external attackers (empowered by AI) and internally approved AI agents that can act maliciously or accidentally at extreme velocity, such as dropping database tables.
In a collaboration-based industry, the most valuable thing is mutual trust between client and designer; without it the product won't come out in a way both celebrate and find pride in.
To remain competitive and differentiated against frontier models, a vertical AI company would have to bring in new types of data.
Ara Khan states that many of the agent projects being pursued are just one model re-solve from being completely wiped out, framing this as a durable risk heuristic for project selection.
Law firms should build durable intangible capital by codifying their way of working into a system, such as a self-improving knowledge graph or precedent files, as their IP; codifying human expertise and judgment into a system is a way for law firms to create IP; this requires an incentive structure where partners are rewarded for codifying knowledge, since partners currently get no dividends from assets they built after leaving the partnership, which undermines codification incentives.
Legal tech historically failed because lawyers had no incentive to be more efficient since efficiency reduces billable hours; now law firms want to adopt legal AI because they understand the gravity of the situation and face high competitive pressure on their business model.
The legal billing model is changing: in the US, more work is priced on project fees or flat rates instead of hourly fees; some lower-end legal work, like small reviews, is now done for free or priced at a flat rate; the money is increasingly made in strategic work, big litigation, and client relationships; law firms have historically done work for free just to maintain client relationships, and AI lets them keep doing that without losing money.
Dharmesh Shah describes the journey from product one to product two to expanding into a strategic adjacent thing: being in marketing software was very close to sales and CRM, making CRM the logical adjacent category. He notes the pattern carries over to entrepreneurs beyond the CRM case, and that existing CRM players were likely to enter marketing over time, so staying only in marketing software was not sustainable forever.
hedged
Gully Labs was founded on a contradiction: India hosts many international brands with design prowess in premium malls, yet few Indian brands. Walking into a premium mall in a big Indian metro, you see many international brands but few Indian brands, despite many global products being made in India. The diagnostic question: how many products for global brands are made in India versus how many Indian brands exist — India is the world's factory but not the world's brands.
Anti-pattern: using workers' stories as a decorative element plastered onto existing products — people will see through it, and have been seeing through it for years. Also an anti-pattern: Indians view their own local stuff as cheap while anything from outside India is seen as premium and luxury.
If you're not the one creating the content and the context about your brand, someone else will.
Sophia Bush offers the lesson that if you are sure something is wrong and everyone is telling you that you're crazy, you are probably not crazy.
hedged
Choose sources → to get a library of your own and a daily brief.