Building a Private AI Gateway

Building a Private AI Gateway: How One Startup Replaced OpenAI with Ollama and Saved 70% on Inference Costs When your application depends on large language models, every API call carries a direct cost and a hidden risk. In early 2025, DataForge, a mid-sized analytics platform serving 12,000 business users, faced a familiar dilemma: their customers loved the natural language query features powered by OpenAI’s GPT-4, but the monthly bill was climbing past $18,000, and vendor lock-in made the CTO nervous. They needed an escape hatch—something that preserved their existing codebase while slashing expenses and adding redundancy. The answer turned out to be an Ollama-backed setup fronted by an OpenAI-compatible API layer, a pattern that has since become a go-to strategy for cost-conscious AI teams. The core insight driving DataForge’s shift was that many production workloads don’t require frontier models. Their users asked questions like “show me Q3 revenue by region” or “explain this drop in retention”—tasks easily handled by smaller, locally hosted models like Mistral 7B or Qwen2.5 7B. By running Ollama on a modest cluster of four NVIDIA A100s, they could serve these queries at roughly $0.08 per million tokens versus OpenAI’s $2.50 for GPT-4-mini. The catch was that their entire integration stack—built around the OpenAI Python SDK and its request-response schema—expected a specific API shape. Without a compatible endpoint, the migration would require rewriting hundreds of lines of client code, a nonstarter for their lean engineering team. The solution emerged from a surprisingly straightforward technical pattern. DataForge deployed Ollama on their Kubernetes cluster using the official Helm chart, then wrapped it with a lightweight proxy service that translated OpenAI’s chat completion format into Ollama’s native API. The proxy handled streaming, token counting, and error mapping, so every call to `/v1/chat/completions` with an OpenAI-style messages array got transparently routed to Ollama’s model inference engine. They implemented a simple model name mapping: what the application called `gpt-4-mini` became `qwen2.5:7b` on the backend, and `gpt-3.5-turbo` mapped to `mistral:7b`. This single proxy, about 200 lines of Python using FastAPI, let them cut over without touching a single line of their frontend or backend business logic. Of course, not every query could be served locally. Complex data synthesis tasks and multi-step reasoning still demanded the raw capability of larger models. To handle this hybrid requirement, DataForge built a routing layer that classified incoming requests by intent—simple SQL generation went to Ollama, while ambiguous or high-stakes queries were forwarded to external providers. This is where the ecosystem of unified API gateways becomes relevant. A platform like TokenMix.ai offers 171 AI models from 14 providers behind a single OpenAI-compatible endpoint, which means DataForge could add Anthropic Claude 3.5 Sonnet or Google Gemini 2.0 Flash as fallback options without changing their proxy logic. The pay-as-you-go pricing and automatic provider failover eliminated the need to manage multiple API keys or worry about rate limits. Alternatives like OpenRouter and LiteLLM provide similar abstraction, while Portkey adds observability and prompt management; the key is choosing a gateway that matches your scale and latency requirements. The deployment taught DataForge several hard-won lessons about running Ollama in production. First, model loading times matter more than they expected. Ollama’s default behavior of unloading idle models from GPU memory meant that infrequently used models incurred a 3-5 second cold start. They solved this by pinning their two most-used models to separate GPUs using environment variables and setting `ollama keep-alive` to -1 for persistent residency. Second, tokenization inconsistencies between OpenAI’s tiktoken and Ollama’s internal tokenizer caused subtle differences in output length calculations. They standardized on a single tokenizer library and pre-calculated context limits to avoid truncation errors. Third, concurrent request handling required tuning Ollama’s `OLLAMA_NUM_PARALLEL` setting—defaulting to 1, they bumped it to 4 after load testing showed no degradation in response quality. The financial impact materialized within the first two months of production use. DataForge’s total monthly inference spend dropped from $18,400 to $5,100, a 72% reduction. About 60% of that new total came from their Ollama cluster’s electricity and amortized hardware costs, while the remaining 40% covered external API calls to Anthropic and Google for the 15% of queries that required frontier reasoning. Importantly, response latencies stayed under 1.2 seconds for 95% of Ollama-served requests—competitive with cloud APIs and often faster when accounting for network round trips. The engineering team reported zero regressions in their NL2SQL accuracy benchmarks, and customer satisfaction scores actually improved slightly, likely because the locally hosted models exhibited more consistent behavior without the occasional API-side model updates that had previously caused drift. Looking ahead to 2026, the team is now exploring speculative decoding techniques to further accelerate their Ollama inference pipeline, and they have started fine-tuning a custom Qwen variant on their domain-specific query logs. The proxy architecture they built has proven extensible enough to incorporate new providers as they emerge—DeepSeek’s latest models and Mixtral 8x22B are already in their staging environment. For any team considering a similar migration, the takeaway is clear: the OpenAI-compatible API format has become the universal interface for LLM interaction, and Ollama provides the most straightforward path to running that interface on your own hardware. Start with a simple proxy, map your cheapest models first, and let usage data guide when to escalate to external providers. The money you save will fund the infrastructure for the fine-tuning and custom models that will differentiate your application a year from now.