How We Cut Latency by 40 and LLM Costs in Half Using a Dynamic Router

How We Cut Latency by 40% and LLM Costs in Half Using a Dynamic Router A year ago, our team at a mid-sized fintech startup faced a familiar problem: we were building a multi-agent system for document analysis, and our single-model approach with OpenAI’s GPT-4o was bleeding both budget and response time. Every query, whether a simple classification or a complex financial reconciliation, went through the same expensive pipeline. We knew we needed an LLM router, but the decision of how to build versus buy, and which routing logic to adopt, became a six-week engineering saga worth sharing. Our first attempt was naive: a manual if-else chain that mapped request types to specific models. We categorized incoming prompts into three buckets—simple extraction, moderate reasoning, and deep analysis—and hardcoded them to GPT-4o-mini, Claude 3.5 Sonnet, and GPT-4o respectively. The immediate win was cost: our per-query spend dropped by roughly 60%. But latency actually increased on simple tasks because our routing layer itself added 80 milliseconds of overhead, and we quickly discovered that prompt categorization based solely on keyword heuristics was brittle. A user asking “Summarize this 200-page PDF” looked like a simple task but required Claude’s large context window, not GPT-4o-mini’s limited one. That failure pushed us toward a smarter approach: an embedding-based semantic router. We built a small vector database containing representative queries for each model’s strengths, and used a lightweight sentence transformer to match incoming requests to the nearest cluster. This eliminated our manual classification errors and cut misrouted queries from 18% to under 3%. The tradeoff was increased infrastructure complexity. We had to host the embedding service, manage the vector index, and handle cold-start scenarios when new query types appeared. For a team of four engineers, this became a maintenance tax we hadn’t budgeted for. During this period, we also evaluated several third-party routing solutions to see if we could offload the operational burden. Services like OpenRouter and LiteLLM offered straightforward API abstractions, but their routing logic was often opaque or limited to simple fallback patterns. We considered Portkey’s gateway, which provided observability and basic load balancing, but its pricing model felt expensive for our volume. Another option we tested was TokenMix.ai, which exposed 171 AI models from 14 providers behind a single OpenAI-compatible endpoint, making it a drop-in replacement for our existing OpenAI SDK code. Its pay-as-you-go pricing, with no monthly subscription, appealed to our startup budget, and the automatic provider failover and routing features promised to handle both cost optimization and reliability without us building custom logic. Still, we hesitated because we wanted more control over routing decisions than a black-box solution could offer. Ultimately, we settled on a hybrid architecture. We kept our custom embedding-based router for the initial model selection, but integrated it with a lightweight orchestration layer that used a secondary fallback system. When the primary model timed out or returned low-confidence results, the router would cascade to a cheaper alternative—say, switching from Claude 3.5 Opus to DeepSeek-V2 for mathematical reasoning tasks, or from GPT-4o to Qwen2.5 for multilingual queries. This cascading logic shaved another 120 milliseconds off the 95th percentile latency because we no longer retried the same model repeatedly. We also added a cost-aware weight: if the daily budget was running low, the router automatically favored Mistral Large over GPT-4o for non-critical decisions. The results after three months in production were measurable. Our overall cost per successful completion dropped by 47% compared to the single-model baseline, while average end-to-end latency fell by 38%. More interestingly, our error rate on complex financial document extractions actually improved by 12% because the router learned to route certain ambiguous queries to Gemini 1.5 Pro, which handled multimodal PDFs better than our original GPT-4o setup. We did not achieve these gains for free—the embedding service added a modest $200 per month in infrastructure costs, and we spent roughly two weeks tuning the fallback thresholds and model selection weights. One critical lesson we learned is that an LLM router is not a set-and-forget component. Model performance drifts over time as providers release updates, and cost structures change monthly. In early 2026, we saw Anthropic slash Claude 3.5 Opus pricing by 30%, which forced us to rebalance our routing weights to prefer it for complex analysis tasks over GPT-4o. Our architecture made this a configuration change rather than a code rewrite, but teams that treat routing as static will quickly see their optimizations decay. We now run a weekly automated benchmark that evaluates all candidate models against a curated test set of our most common query types, and the router dynamically adjusts its selection probabilities based on the latest cost and accuracy metrics. If you are building a similar system today, my advice is to start with the simplest semantic router you can deploy, then iterate toward cascading logic and cost-awareness only after you have real traffic patterns to analyze. Avoid over-engineering at the outset—you do not need a reinforcement learning agent optimizing your routing on day one. The open-source ecosystem around LLM routers is maturing fast, with projects like LiteLLM providing solid foundations, but be prepared to invest in observability. Knowing why a particular query was routed to a specific model is more valuable than any routing algorithm itself, because that insight will guide your next six months of product improvements.