TL;DR: An AI gateway is AI infrastructure that manages traffic to models and, increasingly, to tools. An AI control plane decides what agents may do and keeps the evidence. The public documentation of six real products shows the two categories now overlap, so compare them on five rows: agent identity, per-tool authorization, controls on tool results, how an agent is stopped, and where it runs. Prediction Guard, a sovereign AI control plane, is one of the six, and its limits are stated.
On 24 March 2026, credential-stealing versions of LiteLLM, an open-source AI gateway, were live on PyPI for about 40 minutes, according to LiteLLM, which tells anyone who installed it that day up to 16:00 UTC to treat the system as suspect. An AI gateway holds the keys to every model provider it connects to, which makes it a high-value target and a poor place to stop thinking about agent security. That is the practical reason the AI gateway vs AI control plane question matters for regulated teams. Prediction Guard, a sovereign AI control plane that runs inside your boundary, is one of the six products compared here, and it is held to the same questions as the rest.
This guide compares four AI gateways, LiteLLM, Kong AI Gateway, Portkey, and Cloudflare AI Gateway, with two AI control planes, Microsoft Agent 365 and Prediction Guard, in one table. The aim underneath is operational control of agents: knowing what each one may do, being able to stop it, and being able to prove both. A disclosure: Prediction Guard publishes this guide and builds one of the six, so it is held to the same questions and its limits are stated. Every statement about another product comes from that vendor's own documentation as read in November 2026. A cell that says "not described in the pages reviewed" means exactly that, not that the capability does not exist. Nothing here is legal advice.
On 24 March 2026, two malicious versions of LiteLLM, the widely used open-source AI gateway and Python library, were published to PyPI. LiteLLM's own account gives this sequence:
LiteLLM attributed the incident to a compromised Trivy security scanner in its CI/CD (build and release) pipeline, which was its "understanding" at the time. It named three contributing factors: a shared build environment, long-lived credentials stored in environment variables, and a scanner dependency that was not pinned to a fixed version. Published accounts differ on the exact mechanics of the upload.
The payload was a credential stealer. Cycode's analysis describes it harvesting environment variables, including OpenAI, Anthropic, and Azure keys, along with cloud credentials and Kubernetes configuration from the host. LiteLLM's GitHub advisory told anyone who installed version 1.82.7 or later to rotate every credential on that system, including those in environment variables and config files. LiteLLM's advisory says its official Docker image and installs pinned to version 1.82.6 or earlier were not affected. CloudSEK, a threat intelligence firm, says files captured from the attackers map potential exposure to more than 2,500 organizations. The Hacker News stresses that this is not a victim count and that the scale figures are not independently confirmed, though the wider campaign's downstream impact is.
This belongs in a gateway comparison because of what a gateway holds. The pattern, illustrated:
applications and agents
(hold virtual keys only)
|
v
+-----------------------------+
| AI GATEWAY |
| holds the real keys for: |
| model provider A |
| model provider B |
| model provider C |
| tool servers (MCP) |
+-----------------------------+
| | | |
v v v v
provider provider provider tools
compromise the gateway host or its software supply chain, and everything below the box
is within reach
A gateway often holds credentials for multiple model providers and tools. LiteLLM's virtual keys documentation and configuration examples show the pattern: applications get virtual keys while the gateway keeps the real provider credentials, and its MCP docs show static headers and server variables for tool credentials. This makes a gateway an attractive target, because compromising it or its software supply chain could expose access to several connected systems at once. That is a property of the pattern and not a verdict on one product, since every gateway is software with a supply chain. LiteLLM's published response is the kind of fix to ask any vendor about: it rotated the affected credentials, paused releases, brought in Google's Mandiant for forensics, pinned its CI dependencies, moved to a new release pipeline, and signs container images from v1.83.0-nightly onward.
"Documented" means the vendor's own public documentation describes it. Absence of a description is not evidence of absence, so confirm any cell that matters with the vendor.
Vendor feature names, such as "guardrails," Access policies, and conditional access, appear as each vendor's documentation writes them, so every cell can be checked against its source.
A note on Portkey. Palo Alto Networks completed its acquisition of Portkey on 29 May 2026, according to its Form 10-K, and on 16 July 2026 announced general availability of Prisma AIRS AI Gateway, which it describes as bringing innovations from the Portkey acquisition and which adds capabilities such as ephemeral agent identity. The Portkey row reflects Portkey's own documentation, which still carries the Portkey name and a banner reading "Portkey is now PRISMA AIRS AI Gateway." Confirm the current feature set and packaging of the commercial product with Palo Alto Networks.
LiteLLM: MCP gateway docs, MCP guardrails. Kong: AI MCP Proxy docs, AI Gateway overview. Portkey: MCP guardrails, tool provisioning, deployment options, and Palo Alto Networks' Form 10-K and general availability announcement. Cloudflare: MCP server portals, AI Gateway changelog, AI Gateway pricing page. Microsoft: Agent 365 overview, block action, service description. Prediction Guard: product overview, supply chain page, Manage console.
The headline finding: a product's category label no longer predicts which rows it covers.
Gateways have moved up the stack. All four now document tool-level or tool-adjacent authorization. LiteLLM scopes MCP tools by key, team, and organization, Kong applies per-tool ACLs, Portkey provisions tools at three levels, and Cloudflare curates tools per portal. Earlier descriptions of gateways as traffic managers, including Prediction Guard's own definitional guide, no longer fit the documentation as it stands.
Tool-result inspection is the least uniform row. LiteLLM and Portkey document it. Kong's docs list input and output checks on MCP traffic as not supported, Cloudflare's pages describe prompt and response checks and DLP but not MCP tool results, Agent 365's runtime protection is in preview, and Prediction Guard's pages do not break it out. It matters because an agent reads tool results and acts on them, and an attacker can hide instructions there. That is how goal hijack, where an attacker redirects what the agent is trying to do, arrives.
"Kill switch" means three different things. For the gateways it is revoking a key or a user's authorization, for Agent 365 it is blocking an agent for users across the organization, with scope differences by agent type, and for Prediction Guard it is model kill switches in the Admin Console. Microsoft's page is the only one that addresses credentials, and it says blocking an agent "doesn't necessarily grant or revoke connected-service permissions or credentials." None of the others says what happens to credentials an agent already holds at downstream tool servers, so ask.
Where it runs is the deciding column for regulated teams. Cloudflare AI Gateway and Agent 365 are managed services, LiteLLM and Kong can be self-managed, Portkey offers SaaS, hybrid, and air-gapped options, and Prediction Guard runs inside your boundary. Where the keys and the evidence live decides what an examiner can ask you to produce.
The labels have blurred. Palo Alto Networks calls Prisma AIRS AI Gateway, built on its Portkey acquisition, "the AI control plane for the enterprise," Microsoft calls Agent 365 a control plane, and LiteLLM ships an Agent 365 integration that evaluates MCP tool calls before they run. Compare capabilities, not names.
The table shows why the label is not the useful question: each product covers some rows, and the gaps differ. A better test is to follow one agent action and count how many questions get answered in one place you control. Take an illustrative invoice agent that asks to refund an order:
A sovereign AI control plane is the architecture that identifies, authorizes, enforces, meters, records, and contains agent actions, running inside your boundary. Prediction Guard's own pages describe it as a governed model gateway, MCP proxy, and agentic API layer, so these answers come from one layer. When they come from different products, the gaps between them are where an incident hides, and the evidence has to be stitched together afterward. Several gateways in the table deliver some of these through plugins, security integrations, or separate products. Shipping them as one layer is a design choice with trade-offs.
Its published limits apply. It enforces controls on traffic routed through it. Per-invocation authorization at the tool server is not yet generally available. Token-based limits currently apply per API key. On human approval, its blast radius guide says human-in-the-loop approval gates with workflow automation are not yet generally available, while its zero trust post describes a per-function "ask" mode in Agent Forge, so confirm what applies to your agents. Agent registration with per-agent enforcement is described as in development. AI systems pull their configuration from the Admin Console, so measure how long a stop takes. It does not replace your AI governance process, your identity provider, or the sandbox and network controls around an agent. Run the same demonstration on Prediction Guard that you run on everyone else.
In an AI management architecture, a gateway is AI infrastructure that handles routing, keys, quotas, and traffic. A control plane decides what an agent may do and records what it did. The two can work together: LiteLLM's Agent 365 integration shows a gateway in the request path asking a control plane for a verdict. AI governance, meaning ownership, risk tiers, and approvals, is a process that neither AI platform component replaces. The useful question is which component is the system of record for each decision, and where that record lives.
What secrets does it hold, and how are its releases verified?
This is the LiteLLM lesson. Look for virtual keys for clients, short-lived or per-user credentials for tools, secrets outside environment variables, pinned dependencies, and signed releases.
Does every agent have its own identity, and can one be revoked without touching the others?
A shared key means revoking one agent revokes everything that uses it.
Can authorization reach the tool, and who can change it?
Ask whether a tool-level rule is enforced before the call runs and recorded when it is applied.
What inspects tool results before the agent reads them?
If the answer is only prompt and response checks, tool results are a gap to close elsewhere.
Where does it run, and where do the evidence and keys live?
For a regulated team, a record that sits only in the vendor's service is a record you may not be able to produce.
Ask each vendor to show a blocked call, a stopped spend, and a revoked identity in the same session. A table cell is a claim, and a demonstration is evidence.
Not by label. Both terms are used for products that overlap, so compare what each does on the five rows above.
Sometimes one layer is enough. If a gateway's documented controls cover the rows you need and it runs where your rules require, it may suffice. Where it does not, add a layer that does.
No. It shows that a gateway often holds many credentials, so its software supply chain needs the same scrutiny as any critical system. LiteLLM's published fixes are the kind to ask every vendor about.
As little as possible: virtual keys for clients, short-lived or per-user credentials for tools, secrets managed outside environment variables, and pinned, verified releases.
Where it runs, because it decides where keys and evidence live, and then controls on tool results, because they are the least uniform.
Choosing between an AI gateway and an AI control plane comes down to how much authority your agents have, how their actions are governed, and whether your organization can demonstrate control when something goes wrong. For regulated teams, the right architecture is one that makes those decisions enforceable, observable, and auditable within the boundaries they already trust. A practical next step is to list what each agent may do today, then test every candidate against that list. That is operational control of agents, and it is what the choice should be judged on.
Prediction Guard gives regulated enterprises operational control of their AI agents: one sovereign AI control plane that identifies, authorizes, enforces, meters, records, and contains every action an agent takes, running inside the customer's own boundary. It answers the five rows in this guide from one layer you control: per-application, per-user, and per-integration keys that are independently revocable, an MCP proxy with per-user OAuth and per-function allow, ask, or deny, Component Input/Output Controls and Agent Behavior Controls, model kill switches in the Admin Console, and Immutable Audit Logs generated on Kubernetes inside your infrastructure, on-premises, hybrid, cloud, or air-gapped.
It enforces on traffic routed through it. Tool-result checks are not broken out in its public pages, agent registration with per-agent enforcement is in development, and per-invocation authorization at the tool server is not yet generally available, so test each one. Ask us to show a blocked call, a stopped spend, and a revoked agent identity in the same session.