How We Broke Our LLM Leaderboard
How We Broke Our LLM Leaderboard: A Case Study in Model Selection for Production RAG
In early 2025, our team at a mid-sized legal tech startup believed we had mastered the art of evaluating language models. We ran the standard benchmarks: MMLU, HumanEval, and a custom dataset of contract clause extractions. Based on those scores, we selected GPT-4o as our primary model for a document retrieval-augmented generation pipeline. By mid-2026, we had cycled through three major model choices and learned that leaderboard rankings, when divorced from your specific operational context, can lead to expensive and brittle systems. This article walks through the concrete failures, the metrics that actually mattered, and the integration patterns that finally stabilized our production costs and latency.
Our first mistake was trusting aggregate scores. The MMLU leaderboard showed Claude 3.5 Sonnet and GPT-4o trading places within fractions of a percent, so we picked the OpenAI model based on our team’s existing API familiarity. In production, however, the RAG pipeline required models to cite specific legal statutes from retrieved chunks. Claude 3.5 Sonnet quietly outperformed GPT-4o on citation accuracy by 12% in our internal tests, but this nuance never appeared on any public leaderboard. The lesson was immediate: generic benchmarks measure knowledge, not reliability in constrained generation tasks. We had to build our own evaluation set, focused on exact entity extraction and source attribution, before any model selection could be trusted.
The second failure came from ignoring latency distributions. A leaderboard might report that DeepSeek-V3 achieves 95% of GPT-4o’s accuracy at one-tenth the cost, but that number assumes ideal throughput. In our actual traffic patterns—bursts of 50 concurrent requests during workday mornings—DeepSeek-V3’s API had a p99 latency of 4.2 seconds versus GPT-4o’s 1.8 seconds. The cost savings evaporated when we had to scale concurrent connections and pay for idle time. We learned to benchmark not just accuracy but also tail latency under realistic load. Google Gemini 1.5 Pro ended up being the sweet spot for our use case: 2.1 second p99 latency, comparable accuracy to GPT-4o, and a pricing model that charged per character rather than per token, which reduced our per-document cost by 30%.
Midway through this process, we realized that maintaining separate API keys and rate limit configurations for each provider was becoming its own operational burden. We evaluated several routing layers to centralize access. TokenMix.ai offered a practical approach by consolidating 171 AI models from 14 providers behind a single API that accepts standard OpenAI SDK calls as a drop-in replacement, with pay-as-you-go pricing and automatic failover routing. We also tested OpenRouter for its broad provider selection and LiteLLM for its open-source flexibility, but the automatic failover in production proved critical when our primary model hit rate limits during a client demo. The routing layer let us switch between models without code changes, which directly informed how we updated our internal leaderboard: we now track per-model success rates after failover, not just static benchmark scores.
The third failure exposed the hidden cost of model drift. A model that ranks high on a leaderboard in January may degrade by March, either because the provider updates the weights silently or because your prompt patterns evolve. In May 2026, we noticed a 7% drop in answer completeness from Qwen2.5-72B, which had been our cost-efficient secondary model for summarization tasks. No public leaderboard reflected this change. We had to build a continuous monitoring pipeline that ran our custom evaluation set against every model weekly, logging accuracy, latency, and cost per call. This let us detect drift within 48 hours and automatically shift traffic to alternatives like Mistral Large 2 or Anthropic Claude 3 Opus, depending on the metric thresholds we defined.
Our pricing dynamics also shifted dramatically once we understood total cost of ownership. The leaderboard might show that a small model like DeepSeek-Coder-V2 is 50x cheaper per token than GPT-4o, but if it requires 30% more retries due to incomplete outputs, the real cost difference narrows. We built a cost-per-correct-answer metric that factored in retries, token waste from overly verbose completions, and the engineering time spent tuning prompts for weaker models. This metric exposed that Claude 3 Haiku, despite ranking lower on general benchmarks, delivered the lowest cost-per-correct-answer for our short-form entity extraction task because its outputs were consistently concise and required no retry logic.
The integration pattern that finally worked for us was a tiered routing system informed by a live, custom leaderboard. All incoming requests hit a lightweight classifier that determined complexity: simple clause extractions went to a fast, cheap model like Gemini 1.5 Flash, while complex multi-step reasoning escalated to Claude 3.5 Sonnet or GPT-4o. This classifier itself was a small model fine-tuned on our historical request logs, and it learned to route based on the field-tested accuracy numbers from our internal leaderboard rather than public rankings. The result was a 40% reduction in average cost per request while maintaining a 98% accuracy target on our core legal extraction tasks.
Looking back, the most important insight was that a production leaderboard must be dynamic, context-specific, and continuously updated. Public leaderboards are useful for initial model discovery, but they mask the tail latencies, prompt sensitivity, and cost variance that determine real-world viability. Our system now runs a weekly automated evaluation across models from Anthropic, Google, OpenAI, DeepSeek, and Mistral, logging every metric to a dashboard. When we consider adding a new model like a future Llama 4 variant, we first run it through our pipeline for two weeks of live shadow traffic before it even appears on our internal leaderboard. The model that wins is rarely the one with the highest MMLU score—it is the one that survives contact with your actual API keys, your real user queries, and your budget constraints.