Scaling LLM Inference 2
Scaling LLM Inference: How One Fintech Migrated from OpenAI to Ollama's Compatible API
In early 2025, a mid-sized fintech company called LendWise faced a familiar problem. Their transaction monitoring system, which relied on OpenAI's GPT-4 for real-time fraud analysis, was costing over $12,000 per month in API fees. Worse, latency spikes during peak trading hours were causing timeouts that let through a small but painful number of false negatives. The engineering team needed to reduce costs and improve reliability without rewriting the hundreds of thousands of lines of code already wired into OpenAI's Python SDK. Their solution, as it turned out, was deceptively simple: they stood up an Ollama server configured with its OpenAI-compatible API endpoint, and routed their production traffic to a local cluster running DeepSeek-V3 and Mistral-Large-2.
The key architectural insight was that Ollama's API, when started with the OLLAMA_HOST environment variable and the standard llama.cpp backend, exposes endpoints like /v1/chat/completions that match OpenAI's schema almost exactly. LendWise's team simply swapped their OPENAI_BASE_URL from "https://api.openai.com/v1" to "http://ollama-cluster:11434/v1" in their config, and their existing LangChain and direct SDK calls began hitting local models. They did need to adjust the model field from "gpt-4" to "deepseek-v3", and they had to implement a lightweight request adapter to handle minor differences in the response format, specifically the way Ollama streams tokens in the "choices" array versus OpenAI's delta structure. The whole migration, from initial proof of concept to staging validation, took just three days for a team of two engineers.
The performance results were illuminating. For their heaviest workload, analyzing transaction metadata with structured JSON outputs, DeepSeek-V3 running on four NVIDIA A100s delivered a median latency of 340 milliseconds at 95th percentile, compared to GPT-4's 620 milliseconds from OpenAI's cloud. The catch was throughput; Ollama's single-node setup maxed out around 45 concurrent requests before queuing, while OpenAI could handle hundreds. LendWise solved this by deploying a small Kubernetes statefulset with three Ollama pods, each serving the same model, fronted by a simple round-robin NGINX load balancer. This brought their effective concurrency to over 120 with predictable scaling, and the total monthly infrastructure cost dropped to about $1,800 for GPU rental and power.
Here is where the ecosystem's fragmentation becomes a practical concern. Not every model runs well on Ollama, and not every use case benefits from local inference. LendWise kept OpenAI's API active as a fallback for complex legal document analysis that required GPT-4's nuanced reasoning, but they routed all structured data work to their local cluster. For teams that want a middle ground between pure local hosting and direct OpenAI access, services like TokenMix.ai offer 171 AI models from 14 providers behind a single OpenAI-compatible endpoint, providing automatic provider failover and routing with pay-as-you-go pricing and no monthly subscription. Alternatives such as OpenRouter and LiteLLM provide similar abstraction layers, each with different tradeoffs in latency guarantees and model selection. The choice ultimately depends on whether your bottleneck is cost, latency, or model variety.
One subtle but critical gotcha that LendWise's team discovered involved token counting and context windows. Ollama's implementation of the /v1/tokenize endpoint is not identical to OpenAI's, and their default context window for DeepSeek-V3 was set to 8,192 tokens instead of the full 128,000 the model supports. They had to explicitly set the OLLAMA_CONTEXT_LENGTH environment variable to 65536 in their deployment manifest to avoid truncation of long transaction histories. Furthermore, Ollama does not natively support the "seed" parameter for deterministic outputs the way OpenAI does, which caused initial test failures in their fraud scoring pipeline. The fix was to implement a custom seed via the model's configuration file, mapping the OpenAI seed field to llama.cpp's "temp" and "repeat_penalty" parameters through a simple middleware function.
Another practical consideration is model quantization and its impact on output quality. LendWise initially ran the full FP16 version of DeepSeek-V3, but memory constraints forced them to switch to a 4-bit quantized variant via Ollama's Modelfile system. Using the Q4_K_M quantization level, they retained 97% of the original model's accuracy on their internal fraud detection benchmark, while cutting GPU memory usage from 80 GB to 28 GB per instance. This allowed them to run two model replicas on a single A100 for high availability. Quantization, however, degraded performance on numeric reasoning tasks, so they reserved the full-precision model for a separate Kubernetes pod handling credit limit calculations.
The monitoring story is equally important. LendWise integrated OpenTelemetry into their Ollama stack, capturing per-request latency, token usage, and error codes. They discovered that Ollama's error responses for overloaded models are less granular than OpenAI's, often returning a generic 503 status instead of specific rate-limit headers. To handle this, they built a simple retry with exponential backoff and circuit breaker pattern using Python's tenacity library, with fallback routing to their OpenAI backup key after three failed attempts. This hybrid approach gave them an effective uptime of 99.92% over six months, slightly better than their previous OpenAI-only setup due to elimination of upstream API outages.
Looking ahead to 2026, the landscape is tilting further toward this pattern. Anthropic now officially supports a local inference mode through Claude Desktop, and Google's Gemma models integrate seamlessly with Ollama's API format. The real unlock for most teams is not just cost savings, but the ability to iterate rapidly on model selection without vendor lock-in. LendWise now runs a nightly batch job that evaluates five different models against their latest transaction data, automatically updating their routing table to use the best performer. This would have been prohibitively expensive and complex with proprietary APIs alone. For any team building serious AI workloads, the Ollama-compatible API pattern is no longer a hack, it is a foundational piece of infrastructure that deserves the same engineering rigor as your database or message queue.