Most teams experimenting with the Model Context Protocol are still in the demo phase: a local MCP server, a handful of tools, an AI agent that calls them in a sandbox nobody depends on. Moving that same setup into production, where the agent has real credentials and touches real systems, is a different problem entirely. We have spent the last several months building and hardening MCP servers for clients who wanted agentic AI to do actual work, not just answer questions, and the gap between a working prototype and something you can trust in a regulated environment is wider than most teams expect.
Why MCP Changes the Integration Model
Before MCP, connecting an AI agent to internal systems usually meant custom function calling code wired directly into a single application, brittle, hard to reuse, and impossible to audit consistently across teams. MCP standardizes that boundary. A server exposes a defined set of tools and resources over a consistent protocol, and any compliant client, whether that is Claude, another model runtime, or an internal orchestration layer, can discover and call them the same way. That consistency is what makes MCP worth adopting, but it also means the server itself becomes a real piece of infrastructure with its own uptime, security, and versioning requirements. Treat it like a demo script and it will behave like one.
Design the Tool Surface Around Least Privilege, Not Convenience
The most common mistake we see is exposing an entire internal API as MCP tools because it is easy. A tool that lets an agent “run any SQL query” or “call any endpoint” is not a tool, it is an open door. We design the tool surface the same way we design IAM policies: enumerate the specific actions the agent actually needs, scope each tool to a narrow, well-defined operation, and reject the temptation to add a generic escape hatch “just in case.” On engagements involving Azure Entra ID and SAML federation, we map MCP tool access to the same least-privilege identity model used everywhere else in the environment, so the agent’s permissions are provisioned and revoked through the same governance process as a human user’s, not a separate shadow system nobody reviews.
Authentication and Authorization at the MCP Layer
Running MCP servers on GKE, we put OAuth 2.0 and an API gateway (APIM) in front of every server rather than relying on the protocol’s transport alone. Every tool call carries a verifiable identity, every session is scoped to a token with an expiry, and every call is logged with enough context to answer “which agent, acting on whose behalf, did what” after the fact. This matters more than it sounds like it should, because agentic workflows tend to chain calls together autonomously, and if step three of a five-step chain fails an authorization check, you need to know that immediately rather than discovering it in a downstream data anomaly three days later.
Observability and Failure Modes Are Different for Agents
Traditional application monitoring assumes a human or a fixed batch job triggered the request. Agentic systems introduce variability: the same user intent can produce a different sequence of tool calls depending on what the model decides, which makes traditional request tracing insufficient on its own. We instrument MCP servers to log tool inputs, outputs, and timing independently of whatever the agent’s internal reasoning does, so when something goes wrong, you can reconstruct exactly which tool returned what, rather than trying to reverse engineer it from a model’s summary. Pairing this with existing observability stacks, rather than standing up a parallel system just for AI traffic, keeps the operational burden manageable.
Versioning and Change Management for Tool Definitions
An MCP server’s tool schema is a contract. Change a parameter name or the shape of a response and every agent relying on that tool can silently start behaving differently, not fail loudly the way a broken API call normally would, because the model will often just adapt its usage and produce a plausible but wrong result. We version tool definitions deliberately, run schema changes through the same CI/CD pipelines (GitHub Actions, Azure DevOps) we use for the rest of the platform, and test agent behavior against the new schema before promoting it, the same discipline you would apply to any breaking API change, just with a less forgiving consumer on the other end.
None of this is exotic engineering. It is the same platform discipline, least-privilege access, CI/CD, observability, identity federation, that we apply to every cloud environment we manage, pointed at a newer protocol. The teams that get burned by agentic AI in production are usually the ones who treated MCP as a shortcut around that discipline instead of another surface it needs to cover.
If you are moving MCP servers or AI agent integrations from prototype to production and want a second set of eyes on the architecture, book a consultation at https://cloudevtech.net/contact/.
#MCP #AgenticAI #CloudSecurity #AzureOpenAI #ZeroTrust