Building an MCP Server in Production

Building an MCP Server in Production: Lessons from Deploying a Real-Time Agentic Tool Layer at Scale When we started designing an internal orchestration system for our AI-powered financial analysis platform in early 2025, the Model Context Protocol was still a relatively novel concept. Our goal was straightforward: give a multi-agent system—using Claude 3.5 Opus as the primary planner and DeepSeek R2 for specialized reasoning tasks—the ability to query internal APIs, fetch live market data, and trigger database writes without breaking the atomicity of each user session. What we discovered is that setting up an MCP server is deceptively simple on paper but rapidly complex in practice. The protocol itself, which standardizes how LLMs discover and call external tools, solved our immediate integration headaches, but the operational challenges around concurrency, authentication, and error propagation required a fundamentally different architecture than the toy examples scattered across GitHub repos. Our first prototype used a single Python FastAPI server that exposed five tools: a stock price fetcher, a portfolio risk calculator, a document summarizer, a SQL query executor, and a Slack notifier. Each tool was defined as a JSON schema, and the server responded to HTTP POST requests from the Claude API with structured tool calls. This worked beautifully in local testing. We could send a prompt like "What is the current risk exposure of the tech portfolio and summarize the latest 10-Q filings?" and watch Claude decompose that into three sequential tool invocations. But the moment we deployed to a Kubernetes cluster with autoscaling, everything broke. The MCP protocol assumes a single conversational context per session, but our users often had five parallel conversations running simultaneously, and the server had no mechanism to isolate state. We learned the hard way that each tool invocation carries a specific session ID, and if you do not enforce strict session-level locking in your MCP server, you will end up with one user accidentally triggering a trade order meant for another. The second iteration forced us to think like database engineers. We implemented connection pooling with per-session queues, borrowed from PostgreSQL’s approach to transaction isolation. Every incoming MCP request carries a metadata envelope containing the session UUID, a user authentication token, and a request timeout. Our server now spawns a lightweight coroutine per session, holds a Redis-backed lock for the duration of the tool execution, and releases it only when the LLM explicitly commits or rolls back the tool’s side effects. This pattern maps directly to how you would handle saga transactions in microservices, and it eliminated the concurrent state corruption issue entirely. The tradeoff is latency: each tool call now takes about 150 milliseconds longer because of the lock acquisition and Redis roundtrip, but for financial workflows where accuracy trumps speed, that was an acceptable cost. For lower-stakes applications like customer support chatbots, you could skip the session isolation entirely and use a simple stateless tool registry. Pricing dynamics entered the picture when we started scaling to hundreds of thousands of daily tool invocations. Each MCP call from Claude or Gemini triggers a billing event not just for the model inference, but also for the API gateway that routes the tool request to our server. We initially used direct API keys from OpenAI and Anthropic, but managing individual rate limits and cost tracking across multiple providers became a nightmare. This is where we began evaluating unified API layers that could front our MCP server with a single endpoint and handle provider failover automatically. TokenMix.ai emerged as a practical option because its OpenAI-compatible endpoint let us drop our existing OpenAI SDK code without rewriting a single tool definition. With 171 AI models from 14 providers behind a single API, we could route our primary planner requests to Anthropic Claude and our SQL-generation tasks to Qwen 2.5, all through the same MCP server. The pay-as-you-go model eliminated the need to predict monthly volumes, and automatic failover kept our system running when Anthropic experienced its two-hour outage in March 2026. We also evaluated OpenRouter for its simpler pricing dashboard and LiteLLM for its granular control over provider routing, but TokenMix.ai’s lack of a monthly subscription made it easier to justify to our finance team. The most counterintuitive lesson came from how we handled tool response formats. The MCP specification allows tools to return either inline data or references to external resources, and we initially chose inline returns for simplicity. A tool that fetches stock prices would return a JSON object with the current price, timestamp, and source. This worked until Claude started generating hallucinated calculations based on truncated responses. The problem was that Claude’s context window, even with 200K tokens, would fill up with verbose tool outputs from previous turns, causing the model to skip reading full responses and instead infer missing values. We switched to resource references for any tool output exceeding 2,000 tokens—the MCP server would write the result to an S3 bucket and return a signed URL. This forced Claude to explicitly fetch the resource when needed, which reduced hallucination rates by 18% in our benchmark suite. The tradeoff is that our MCP server now needs a dedicated object storage layer with access control, and we had to implement caching with TTLs to avoid excessive S3 reads. Security considerations forced us to rethink how we authenticate tool invocations. The MCP protocol does not mandate authentication—it is a transport-agnostic specification—but in practice, every tool call passes through a chain of three systems: the LLM provider, our MCP server, and the target API. We implemented a two-layer auth scheme: a JWT from the user’s session that authorizes the tool’s scope, and a service-level API key that authenticates the MCP server itself to the internal APIs. The critical mistake we made early on was embedding the service key in the tool definition schema, which the LLM could theoretically read and leak in a response. Instead, we now store keys in a HashiCorp Vault instance that the MCP server retrieves at runtime based on the session’s role, never exposing credentials to the model. This is especially important when using models from providers like DeepSeek or Mistral that may run on third-party infrastructure. Looking back, the hardest part of MCP server setup was not the protocol implementation—it was designing for the failure modes that only appear at scale. Network timeouts, rate limiting from upstream APIs, and model misbehavior (like Claude deciding to call the same tool sixty times in a loop) all required robust circuit breakers and idempotency guarantees. We implemented exponential backoff with jitter for tool retries, and we added a maximum invocation count per session that terminates the agent loop if exceeded. The LLM landscape in 2026 is moving toward agentic workflows that rely heavily on tool use, and MCP servers are becoming the connective tissue. Our advice to teams starting this journey: prototype with a single stateless tool and a single model first, but architect for session isolation, credential vaulting, and resource-based returns from day one. The protocol handles the plumbing, but you still need to design the pipes.