35,595 results · Most relevant
If your judge agrees with you 100% of the time, it is probably sycophantic, has bias poisoning, or you have overfit your data. Overfitting means your judge is trained exactly on your specific examples and is not ready for the real world. 100% is kind of a red flag. Don't fail the eval at 100% agreement; instead, have both a lower bound and an upper bound, because 100% means you've overfit, and we don't want that.
Your judge needs to be a different model family from the rest of your corpus if you're generating text — this addresses self-preference bias, where a judge will always choose text generated by its own model family.
When a judge eval fails, you have a choice: introduce policy (RAG) or use a smarter model — a real-world tradeoff decision. Prefer refining the policy and using RAG over upgrading the model, because of cost. Check for case-sensitivity issues (e.g., lowercase mismatched verdicts) before concluding a judge eval has failed. Getting under five failures out of 30 is achieved by an iterative process: find where the knowledge gap is, then fill it with policy. For a 30-case judge eval to be acceptable, no more than five cases can fail — that's the 80% threshold. Run the judge over all 30 dataset examples and check the agreement rate between AI verdicts and human verdicts. After adding the change-of-mind policy, failures dropped to three, which is way better than before. When the eval
Evals can lie sometimes, and it's not good when they do. Position bias: given a pairwise comparison, a poor judge will almost always pick the first option — this happens pretty often, especially with cheaper models and fine-tunes; to check for it, swap the order of options. Sycophancy: an LLM judge will choose 'yes, you can return it' even when it's not within policy, because LLMs love making you happy; system prompts like 'you are a personal assistant, you are a helpful assistant' push judges toward the nicer response. Self-preference bias: an LLM judge will always choose text generated by the same model family over human input — they love their own generated text; this is really well proven, with arXiv papers on it. LLM judges also love verbosity: if you give them the wrong answer but it
The production LLM agent and the judge should share the same policy — the policy must be one-to-one. Don't copy the policy over; instead, have it live in a single source that both the judge and the production agent query with RAG. The policy should be a living thing retrieved by both the judge and the in-production agent. Copying the policy separately into the judge and the production agent creates drift — store it in one queried source instead. When you want to enforce policy everywhere, make sure everybody knows the policy.
Job one: create, maintain, and deepen the executive relationship — everything in account management happens from it. Job two: leverage that relationship to understand the outcomes the customer needs (the target). Job three: ensure the CSM is driving the success plan to hit the target on target and on time. The three form a sequence: right person → right target → right delivery.
Three things literally increase contract value: (1) attach a new product, (2) expand the scope (new personas or departments), and (3) change commercial mechanics (new packages, prepay, restructure, early renewal). Four triggers activate them: a new use case, a pricing/packaging change, consolidation of competing or tangential products, and M&A or org change.
Piggybacking on calendar-based meetings like QBRs is an anti-pattern: there's no real reason to be there, it gets awkward, and it steals time from the CSM. Random 30/90-day recurring meetings produce too much, too little, or nothing new to share. Instead, align everyone (CSM included) to a project-based rhythm: kickoff, stage-progress checks, mid-project meetings, and milestone updates.
The AM's job: act as the 'grocery shopper' — identify the executive that matters, the outcome they need, the metric proving achievement, and the timeline — then hand those ingredients to the CSM chef. Verify the CSM's deliverable matches what the executive actually wants (right dish, no unwanted ingredients) before it's presented. Then act as the 'waiter': present and sell the delivered value proof back to the buyer — don't assume the product speaks for itself, and don't walk in saying 'our product does what it does.' A successful delivery leads to the customer 'ordering dessert' (expansion).
Gemini models use thought signatures: when the model calls a function or gives a response, it passes back an opaque series of numbers that must be preserved. If you don't give back the thought signatures with the new Gemini series of models, you will see a decrease in performance. Manually managing them was very difficult for many people — startups lost their cache with a single mistakenly added white space.
Companies often ask how to secure their agent so it doesn't leak a network credential. To secure agents, Google implemented a man-in-the-middle proxy. The proxy stands in the middle of the model's API calls, looks at the header, and can replace items at will. In the demo, every single outbound call to the GitHub API is transformed and the token is injected dynamically. Even if the model is prompt injected and leaks your GitHub API token, it never actually sees the token; it only executes code that runs against the GitHub API, so your token is never exposed.
Harald Kirschner reports that previously VS Code did YOLO releases: on release day, after testing, they shipped the release to 100% of users at once. Because of agents, de-risk the release process with staged rollouts instead of shipping to 100% at once. During staged rollouts, monitor error logs and issues and everything else. Apply the same staged rollout practices as good web application citizens to installed apps because rollbacks on an installed app are so expensive.
Harald Kirschner observes that when agents work on UI, they give you UI and say it looks perfect, and you open it and it's all misaligned — even with SVG and the latest models. If your agent cannot do it, use your application, your product directly to get the feedback loop of is it all working. Using your application directly for UI feedback is a really big investment that pays off every time you work on UI.
Adding a search tool to an agent is not enough; a two-step process is required: instructing your agent when to use web search (often not built in) and then actually executing the search. You have to provide the model with a set of rules so it knows when to use web search. If you just bolt a web search onto your agent, the model may say a requested model version doesn't exist even though it has access to a web search tool, and it won't actually do anything. Natively, Claude Code does have a web search tool, but the model simply doesn't know that a requested version exists. Exa's coding agent customers often add the search tool to their agent, but adding it to the harness is simply not enough.
None stated.
The human role in code review is evolving from direct inspection to designing and tuning the review systems, defining 'good' code, and managing the review harness.
high
Hyperscaler CapEx should be viewed as 'someone else's order books' for the broader technology supply chain (chips, power, construction). Investors should monitor businesses serving this industrial boom rather than just traditional software/IP monetization models.
endorsed
Leaders should share open memos, post board decks, and admit when the company got lucky or made wrong calls. This transparency removes mystery and keeps employees connected to leadership thinking.
high
Zach Lieberman describes his own practice: he writes code and makes sketches that are often moving, animated, and interactive. He also loves printing and studied printmaking. He describes the process of printmaking (wiping a plate, putting paper in water, pulling it, holding an image) and notes that holding an image in your hand allows you to look at it in a different way than on a screen. His prints are a way of exploring code and seeing what's possible, making images he can live with and see in a new way.
Growth is the dominant factor when it comes to valuation. If you can only fix one thing, fix growth. More growth is always better for valuation.
Jeff Cavaliere explains that poor chest growth is often due to poor activation, with the delts dominating the movement. He recommends lightening the weight, slowing the pace, and focusing on letting the chest do the work.
Choose sources → to get a library of your own and a daily brief.