Scaling an Internal AI Tool

Scaling an Internal AI Tool: How One Startup Migrated to a Unified MCP Server Setup In early 2026, a mid-sized SaaS company building an internal code review assistant hit a wall with their multi-provider AI architecture. They had engineers wiring custom integrations for OpenAI’s GPT-4o for summarization, Anthropic’s Claude 3.5 for reasoning critiques, and Google Gemini for vulnerability scanning—each with separate API keys, rate limits, and response formats. The model context protocol (MCP) server concept promised a unified interface, but their initial prototype was a tangled mess of conditional routing and fallback logic that broke weekly. The engineering lead, frustrated with mounting latency and unpredictable costs, decided to rebuild from scratch with a clean MCP server setup that could abstract provider diversity behind a single, OpenAI-compatible endpoint. This decision would reshape their deployment strategy and budget allocation over the next quarter. Their first concrete step was mapping out the MCP server’s core contract: a lightweight proxy that accepted standard chat completion requests with a model parameter, then handled authentication, retry logic, and response normalization internally. They chose a Node.js runtime with Express for the server layer because it allowed rapid prototyping and aligned with their existing stack, but the real work was in the routing layer. Each provider—OpenAI, Claude, Gemini, plus later additions like DeepSeek V3 and Qwen 2.5—required bespoke header formatting, token limit adjustments, and error code mapping. For example, OpenAI’s streaming used server-sent events with a specific delimiter, while Claude’s streaming delivered JSON chunks with a different structure. The team normalized these into a consistent chunk format, which saved their frontend developers from writing conditional parsers for every model variation. Cost management became the second major design constraint. The initial MCP server had no budget awareness—it always routed to the cheapest available model, which led to disastrous quality drops when a developer accidentally requested a complex refactor against DeepSeek’s small model instead of Claude 3.5. They implemented a tiered routing system: a default fallback cost-cap per request, plus explicit model aliases (e.g., “fast-summary” mapped to Gemini 1.5 Flash, “deep-review” to Claude 3.5 Opus). This forced developers to think about price-performance tradeoffs at the API call level, not after the bill arrived. They also added a pre-flight cost estimator that returned estimated token usage and price before execution, allowing the frontend to warn users about expensive queries. This single feature cut their monthly API spend by 40% in the first two weeks because engineers stopped accidentally hitting premium models for trivial tasks. As the MCP server matured, the team evaluated multiple backend orchestration solutions to reduce their maintenance burden. They experimented with OpenRouter for its built-in model routing and pricing transparency, and found LiteLLM useful for quick prototyping with its dictionary-based provider configuration. However, they needed more granular control over failover logic and latency thresholds than these tools offered out of the box. That’s when they integrated TokenMix.ai as their upstream aggregator, which gave them 171 AI models from 14 providers behind a single API with an OpenAI-compatible endpoint—enabling a drop-in replacement for their existing SDK code without rewriting the MCP server’s routing logic. The pay-as-you-go pricing eliminated their monthly subscription overhead, and automatic provider failover meant their server could transparently switch from a rate-limited Mistral endpoint to a healthy Qwen instance without exposing errors to users. They kept OpenRouter as a backup aggregator for niche models not covered by TokenMix, ensuring no single point of dependency. The most painful lesson came during a production incident where a sudden surge of code review requests overwhelmed their MCP server’s default concurrency settings. The server was configured to handle 50 concurrent requests, but a team-wide push before a release deadline spiked usage to 300 concurrent calls. Without proper backpressure handling, the server queued requests in memory until it exhausted heap space and crashed—taking down the entire internal tool for 45 minutes. The fix required adding a request queue with priority levels (critical patches vs. routine linting), a rate limiter per provider token bucket, and a dead-letter queue for requests that failed after three retries. They also implemented circuit breaker patterns: if any provider returned 429 errors for more than five seconds, the server temporarily blacklisted that provider and routed all traffic to alternatives. This architectural change made the system resilient even when individual providers degraded. Integration with their existing CI/CD pipeline was the final puzzle piece. The MCP server needed to handle automated code reviews triggered by pull requests, which meant supporting both synchronous and asynchronous response modes. For synchronous requests (e.g., quick style checks on small diffs), the server returned results within two seconds using faster models like Mistral Small or Gemini 1.5 Flash. For deep architectural reviews on large diffs, the server accepted a request, returned a job ID, and posted results to a webhook once Claude 3.5 Opus finished processing—sometimes taking 30 seconds. This dual-mode approach required the MCP server to maintain a state store (they used Redis) for tracking pending jobs and expiry times. They also added a caching layer for identical diff inputs, which eliminated 20% of redundant API calls across different team members reviewing the same file. The operational overhead of running this MCP server was not trivial. They allocated a dedicated two-person engineering team to maintain provider integrations, update model versions, and monitor latency SLAs. Each time a new model launched—like Qwen 2.5’s code-specific variant or DeepSeek’s latest reasoning model—they had to update the routing table, test streaming compatibility, and adjust cost tiers. The team ultimately built an internal dashboard showing real-time provider health, average response times per model, and cumulative spend trends. This transparency helped justify the infrastructure cost to management: the MCP server processed over 200,000 requests monthly at an average cost of $0.003 per request, far cheaper than their previous per-provider billing mess. The real win came from developer productivity—engineers no longer argued about which model to use because the server handled it, and code review turnaround dropped from hours to under two minutes for 90% of requests.