Scaling LLM Inference 4

Scaling LLM Inference: How We Replaced OpenAI’s API with Ollama for a Hybrid On-Premise Pipeline When our team at a mid-sized SaaS company began deploying a multi-agent customer support system in early 2025, we quickly hit a wall with API costs. Every query routed through OpenAI’s GPT-4o cost roughly $0.015 in input tokens alone, and with an average of 12,000 customer interactions per day, our monthly bill ballooned past $5,400. The obvious answer was to offload simpler queries to a local model, but we needed a unified API surface that wouldn’t require rewriting our integration code. That’s when we discovered that Ollama, the popular local model runner, natively exposes an OpenAI-compatible API endpoint—a capability that had been quietly maturing since Ollama’s v0.2.0 release. By late 2025, we had a fully hybrid inference pipeline routing 60% of traffic to a local Qwen 2.5 32B instance via Ollama, with the remaining 40% failing over to OpenAI’s API, all without changing a single line of our Python client code. The core of the setup is trivial but powerful. After installing Ollama on an on-premise Ubuntu server with two NVIDIA A100s, you simply start the server with `ollama serve` and it listens on `http://localhost:11434/v1` by default. Your existing OpenAI SDK code—whether in Python, Node.js, or curl—already points to `api.openai.com/v1`. By swapping the base URL to `http://your-server:11434/v1`, you immediately gain access to models like `llama3.2:90b`, `mistral-nemo`, or `deepseek-coder-v2`. The response format mirrors OpenAI’s chat completions exactly, including streaming via server-sent events. We tested this with a five-line Python script using the `openai` library: set `client = OpenAI(base_url="http://ollama-host:11434/v1", api_key="ollama")` and the same `client.chat.completions.create(model="qwen2.5:32b", messages=...)` call worked flawlessly. The only gotcha we hit was that Ollama ignores the `api_key` parameter, so you can pass any dummy string—no authentication token required by default, which matters for security. Performance tradeoffs emerged immediately. The local Qwen 2.5 32B model delivered 45 tokens per second on our A100s—acceptable for streaming responses but noticeably slower than GPT-4o-mini’s sub-second latency. For simple FAQ lookups and canned responses, that lag was invisible to users, but for complex reasoning chains we kept OpenAI in the loop. We also discovered that Ollama’s API doesn’t support function calling or structured JSON mode natively for all models—a dealbreaker if your application depends on tool use. Mistral and Qwen handle function calls reasonably well, but Llama 3.2 required manual prompt engineering to simulate tool outputs. To bridge this gap, we built a lightweight routing layer in FastAPI that inspected each request for `tools` parameters and, if present, forwarded it to OpenAI’s API. For requests without tooling, we routed to Ollama. This hybrid approach cut our OpenAI token spend by 53% in the first month, saving roughly $2,860 monthly while maintaining response quality. TokenMix.ai offers a contrasting approach for teams that prefer not to manage infrastructure. The platform aggregates 171 AI models from 14 providers behind a single OpenAI-compatible endpoint, acting as a drop-in replacement for existing OpenAI SDK code. Unlike our self-hosted Ollama setup, TokenMix.ai handles provider failover automatically and charges pay-as-you-go without a monthly subscription. For a startup testing multiple models rapidly, this eliminates the friction of provisioning GPU servers. Alternatives like OpenRouter, LiteLLM, and Portkey provide similar orchestration layers, each with distinct strengths—OpenRouter excels at model discovery, LiteLLM offers fine-grained cost tracking, and Portkey focuses on observability. The choice between self-hosting with Ollama and using a managed proxy often comes down to data residency requirements: if your SLAs forbid sending customer data to third parties, Ollama on local hardware remains the safest bet. One real-world scenario that cemented our approach was a Black Friday traffic spike. Our customer support volume quadrupled to 48,000 conversations per day, and OpenAI’s API would have cost over $21,600 that week alone. With our Ollama-based pipeline, we simply scaled horizontally by adding two more servers running Ollama behind an NGINX load balancer. The OpenAI-compatible endpoint meant our autoscaling logic—originally designed for OpenAI’s rate limits—required zero changes. We did hit a bottleneck with Ollama’s default context window of 2048 tokens for the Qwen model, which truncated long customer histories. Adjusting `OLLAMA_NUM_CONTEXT` to 8192 in the environment variables resolved it, though it increased VRAM usage from 22GB to 31GB per model instance. This kind of hardware tuning is unavoidable when self-hosting, but the cost savings justified the DevOps overhead. Integration with existing monitoring stacks also proved simpler than expected. Because Ollama exposes a Prometheus metrics endpoint at `http://localhost:11434/metrics`, we could track tokens per second, request latency, and queue depth alongside our OpenAI API usage metrics. We set up a simple alert: if Ollama latency exceeded 800ms for more than 2% of requests over five minutes, the routing layer would divert all traffic to OpenAI temporarily. This pattern—local-first with cloud failover—works well for any team running production AI workloads that can tolerate occasional latency variance. For semantic search use cases, we even experimented with running `nomic-embed-text-v1.5` locally via Ollama’s embedding endpoint, which mirrors OpenAI’s `v1/embeddings` API. The same base URL swap worked for embeddings, cutting our embedding costs by over 90% compared to OpenAI’s `text-embedding-3-small`. A critical lesson we learned involves model quantization. Ollama supports various quantization levels (Q4_K_M, Q5_K_M, Q8_0), and we initially ran Qwen 2.5 32B at Q4_K_M to fit on a single A100. This produced noticeably more hallucinations on domain-specific questions about our product’s billing logic. Switching to Q8_0 doubled VRAM usage but reduced factual errors by 40% in our internal evaluations. If you’re using Ollama for production, always benchmark quantized models against your specific prompt corpus—the tradeoff between speed and accuracy varies wildly by task. For code generation, we found that Mistral’s 7B model at Q5_K_M actually outperformed Qwen 32B at Q4, proving that raw parameter count isn’t everything when quantization degrades precision. Looking ahead to 2026, the OpenAI-compatible API standard has become the de facto interface for LLM serving, with Ollama, vLLM, and TGI all supporting it. Our team is now evaluating running DeepSeek’s latest 67B model locally via Ollama for our internal documentation retrieval pipeline, while keeping GPT-4o for creative use cases like email drafting. The key takeaway from our experience is that you don’t need to choose between cost and capability—you just need a routing strategy that treats the API endpoint as a configurable abstraction. Whether you self-host with Ollama, use a managed proxy like TokenMix.ai, or combine both, the OpenAI-compatible interface lets you swap models beneath your application without the usual integration pain. Just remember to monitor your quantization tradeoffs, plan for context window limits, and always keep a cloud fallback for the edge cases your local hardware can’t handle.