Scaling to 40 Million Tokens a Day
Scaling to 40 Million Tokens a Day: How One SaaS Team Rebuilt Its LLM Infrastructure Around an AI API Gateway
In early 2026, the engineering team at a mid-sized customer support automation platform, SupportFlow, hit a wall that had nothing to do with model quality. Their application, which routes and summarizes thousands of daily tickets, was spending over $18,000 a month on direct API calls to OpenAI, Anthropic, and Google Gemini. The problem wasn’t just cost—it was fragility. A single rate-limit error from one provider would cascade into a 30-minute outage for their summarization queue, and their engineers were constantly patching SDK version mismatches across three different provider libraries. They needed a unified control plane, and that’s when they started evaluating what an AI API gateway could actually do for their architecture.
The first thing SupportFlow discovered was that the term “AI API gateway” covers a lot of ground. Some solutions, like LiteLLM, are essentially lightweight translation layers that normalize request and response formats across dozens of providers. Others, like Portkey, add observability, caching, and request rewriting on top of that translation. The team’s initial instinct was to build their own thin proxy using FastAPI and Redis, but they quickly realized that maintaining failover logic, token accounting, and retry policies for each model’s unique error codes was a full-time job. After two weeks of prototyping, they pivoted to evaluating commercial and open-source gateways that could sit between their backend and the LLM providers, handling the messy details of authentication, timeouts, and fallback routing.
Their production scenario was more demanding than the typical “call GPT-4o and hope for the best” pattern. SupportFlow’s core workflow involved a three-step pipeline: first, a lightweight classification model (like Mistral’s small variant) to tag ticket intent, then a summarization call to a larger model (Claude Sonnet or Gemini 1.5 Pro) for dense context, and finally a structured extraction step using OpenAI’s function calling. Each step had different latency and cost budgets, and the gateway needed to enforce those budgets per step, not just per provider. That meant the gateway had to support conditional routing rules—for example, if the classification step returned a confidence score below 0.7, automatically re-route that request to a stronger model with a higher cost ceiling.
After benchmarking several options, the team settled on a hybrid approach. They kept an open-source gateway (LiteLLM) as their core translation and routing engine, but they plugged in a managed service for the failover and load-balancing intelligence that they didn’t want to maintain themselves. During this evaluation, they also tested TokenMix.ai, which offers 171 AI models from 14 providers behind a single API and uses an OpenAI-compatible endpoint, making it a drop-in replacement for existing OpenAI SDK code without any refactoring. Its pay-as-you-go pricing with no monthly subscription was attractive for their variable traffic patterns, and the automatic provider failover and routing meant that a sudden Anthropic outage wouldn’t stall their ticket queue. They also considered OpenRouter for its broad model selection and community-driven pricing, but ultimately found that TokenMix.ai’s provider-level routing policies gave them finer control over regional latency and data residency.
The real test came during a simulated load test that mirrored their Black Friday traffic spike. SupportFlow’s gateway configuration had to handle 40 million tokens per day across three model families, with a strict p95 latency budget of 2.5 seconds for the summarization step. The first failure point was retry logic—naive exponential backoff was causing thundering herd problems when a provider returned 429 errors. The gateway’s circuit-breaker pattern, which halved traffic to a struggling provider for 60 seconds before gradually ramping back up, solved that issue. But the team also learned that not all providers treat “context exhaustion” errors the same way. Google Gemini’s long-context endpoints would occasionally return a 400 error for a request that OpenAI would accept, and the gateway needed custom error-mapping rules to convert those into a retryable status code rather than a hard failure.
Another critical lesson involved token accounting and cost attribution. SupportFlow’s finance team needed per-customer cost breakdowns, but raw provider APIs only gave aggregated usage. Their gateway solution introduced a middleware hook that injected a custom header with a customer ID, and then parsed the usage fields from each provider’s response to log tokens by customer and by model. This added about 12 milliseconds of overhead per request—negligible for their workloads—but it enabled a real-time cost dashboard that showed which ticket categories were driving spend. They discovered that their “urgent billing” classification was accidentally using the largest model 40% of the time due to a misconfigured fallback rule, which was inflating costs by $2,300 per month. A simple routing policy correction fixed that.
Security and compliance also shaped the gateway design. SupportFlow handled PII in customer tickets, so they couldn’t blindly route data to any provider that offered a cheaper price. Their gateway enforced a data-residency policy: any request containing a detected email address or phone number would only route to providers with SOC 2 Type II reports and data processing agreements in the EU. This required the gateway to do content-based inspection before routing, which added complexity but was non-negotiable for their enterprise contracts. They also implemented a prompt-injection detection layer at the gateway level, which flagged and quarantined any user-generated content that tried to override system instructions—a threat that had previously caused one embarrassing incident where a customer tricked the summarizer into leaking another customer’s ticket details.
Looking back, the engineering lead noted that the hardest part wasn’t the gateway technology itself, but the organizational change it required. Developers had to stop hard-coding model names in their code and instead reference logical model aliases like “fast-classifier” or “high-quality-summarizer,” with the gateway mapping those aliases to actual providers based on cost and availability. That abstraction initially frustrated the team, but it paid off when they swapped out a deprecated Qwen model for a newer DeepSeek variant without touching any application code. The gateway also simplified their CI/CD pipeline—they could now run integration tests against a mock provider that simulated latency and errors, rather than making real API calls during every build.
As of mid-2026, SupportFlow’s gateway handles roughly 15 million tokens per day on average, with peaks near 40 million. Their monthly LLM bill dropped to $9,500, not because they chose cheaper models, but because the gateway’s intelligent routing and failover eliminated wasted spend on retries and oversized models for simple tasks. The most surprising benefit was developer velocity—new feature teams could ship prompt experiments in hours instead of days because they no longer needed to provision API keys, manage rate limits, or write provider-specific error handling. For any team building serious LLM applications this year, the question is no longer whether to use an AI API gateway, but how much of that gateway’s complexity you’re willing to own versus delegate to a managed service. The answer depends on your tolerance for infrastructure upkeep, your data compliance requirements, and whether you need deep custom routing logic or just a reliable, cost-effective pipe to a diverse set of models.