TL;DR: Self-hosted AI agents on open-weight models give regulated teams control over where data, model versions, and evidence live, but the operating burden rises with isolation. Choose a rung on a five-step control ladder, from a public vendor API to an air-gapped enclave, and for most regulated workloads that must keep data and evidence inside without owning a data center, run inference in your own cloud account. The security, governance, and model-supply-chain work then becomes yours, and hybrid arrangements that route by data sensitivity are normal.
For regulated enterprises, deploying AI agents on infrastructure they control is not simply a question of where the model runs. It is a decision about data boundaries, model control, operational responsibility, and the evidence required to demonstrate control.
Between a public vendor API and a fully air-gapped environment, several deployment options exist. Each shifts a different combination of infrastructure, security responsibilities, and operational burden from the vendor to your team.
This guide is for regulated enterprise AI teams weighing self-hosted AI agents on open-weight models in their own cloud. It sets out the options as a five-rung control ladder, draws a cloud deployment reference architecture for the rung that keeps data and evidence inside without owning a data center, and then covers what you are choosing with open-weight models, the AI agent security work that moves to you on private AI infrastructure, how governance and evidence work inside your boundary, and the tradeoffs against vendor API alternatives for enterprise AI agents. One disclosure: Prediction Guard publishes this guide and builds a sovereign AI control plane, which appears in the architecture. Nothing here assumes you need it at every rung, and the vendor API rungs are described from the vendors' own documentation.
Control and operating burden rise together. Each rung gives you more of one and costs you more of the other:
| Rung | Where Inference Runs | What Stays Inside Your Boundary | What You Take On | Typical Fit |
|---|---|---|---|---|
| 1. Public vendor API | The vendor's infrastructure | Your application and its credentials | Reading the vendor's retention, training, and model-lifecycle terms | Low-sensitivity workloads and fast starts |
| 2. Vendor API with contractual controls | The vendor's infrastructure, under negotiated terms | Your application, your own logs, and any data controls the vendor approves | Approvals, endpoint eligibility limits, and a model still run by someone else | Moderate-sensitivity workloads with strong contractual needs |
| 3. Managed open-weight endpoints in your cloud account | Your cloud provider's managed hosting, in your tenancy | The network path, identity, and logs you configure in your account, plus whatever the provider's terms commit to for the managed service | Model choice, endpoint configuration, and the provider's service terms | Teams that want open-weight choice without running GPUs |
| 4. Self-hosted inference in your VPC | GPU nodes you run inside your cloud account | Prompts, outputs, weights, logs, and enforcement | The serving stack, patching, capacity, evaluation, and the model supply chain | Regulated workloads where data and evidence must stay inside |
| 5. Air-gapped or on-premises | Hardware you own, with no outbound path | Everything, including updates | Physical-media updates, hardware lifecycle, and the slowest change cycle | Defense-adjacent or contractually isolated workloads |
Two caveats apply to the middle rungs. Rung 2 terms are vendor-specific: the example in this guide is OpenAI's, whose data controls require approval and vary by endpoint, and other providers' controls and eligibility differ. Rung 3 varies by provider, so confirm what your provider's managed hosting actually keeps inside your tenancy. Three points help in reading the ladder:
Rung 4, a self-hosted LLM deployment inside your own cloud account, suits teams that want the data and the evidence to stay inside without owning a data center. The components below sit in one cloud account and one network boundary:
+----------------- your cloud account / VPC -----------------+
users and apps --> | agent runtime (sandboxed) |
| | |
| v |
| control plane: identity, policy, tool proxy, audit |
| | | | |
| v v v |
| inference servers MCP tool servers data stores |
| (open-weight) (scoped per user) (entitlements) |
| |
| default-deny egress | private registry mirror | secrets |
+----------------------------|--------------------------------+
v
SIEM and evidence store (inside your boundary)| Component | Its Job | Design Note |
|---|---|---|
| Agent runtime and sandbox | Runs the agent's loop and any code it executes | Isolate it, and let it reach only the control plane |
| Control plane | Identity, policy enforcement, tool access, and audit records | The one place model and tool calls are decided and recorded |
| Inference servers | Serve the open-weight models on GPU nodes | Authenticated, patched, and reachable only from the control plane |
| MCP tool servers | Give agents access to internal systems | Scoped per user, and registered before they can be called |
| Data stores | Hold the data agents read | Enforce entitlements at retrieval, not only in the model |
| Network egress | Controls what can leave | Default-deny, so an agent cannot call public APIs or install packages |
| Model and image registry | Holds weights and container images | A private mirror with pinned hashes, not live downloads |
| Observability and evidence | Collects logs and decision records | Stored inside your boundary and streamed to your SIEM |
Pointing an agent at this setup is mostly configuration. Frameworks that speak the OpenAI API can target a self-hosted endpoint by changing the base URL, and a serving stack such as vLLM exposes an OpenAI-compatible server. This is an illustrative pattern, not a tested deployment. On the server, automatic tool calling has to be turned on explicitly, and the model family decides the parser and template:
vllm serve <model> --enable-auto-tool-choice --tool-call-parser <parser>import os
from openai import OpenAI
client = OpenAI(
base_url="https://llm.internal.example/v1", # an endpoint inside your VPC
api_key=os.environ["INTERNAL_LLM_KEY"],
)
response = client.chat.completions.create(
model="<model>", messages=messages, tools=tools,
)The tool calling documentation for vLLM is specific about the catch: automatic tool choice requires the flag and a model-specific parser, and parallel tool calls are model dependent, so an agent that worked against one model can behave differently on another. Treat tool-calling reliability as something to test, not assume.
An open-weight model is one whose trained parameters are published for download and local use. That is a narrower promise than open source. The Open Source Initiative's Open Source AI Definition 1.0 asks for training code and data information as well as weights, which is why, as IEEE Spectrum's coverage of OpenAI's gpt-oss release noted, a model can ship under a permissive license and still not meet that bar. Four practical consequences follow:
When you run the model, the vendor's security work becomes yours. The main items:
Running the model yourself does not give you governance by itself. You still need policy applied to each call, an identity behind each action, and records that someone else can follow. Four requirements matter at this rung, and Prediction Guard's published description addresses each:
The same published guidance states the limits: it enforces on traffic routed through it, per-invocation authorization at the tool server is not yet generally available, token limits apply per API key, and human approval gates are on the roadmap, as its blast radius guide says.
Vendor APIs are a legitimate choice, and their terms are often strong. The comparison is not safe against unsafe. It is which risks you want to own:
| Dimension | Vendor API | Self-Hosted Open-Weight in Your Cloud |
|---|---|---|
| Where prompts and outputs go | The vendor's systems, under its terms. OpenAI's data controls page says API data is not used for training by default, abuse-monitoring logs are retained for up to 30 days, and zero-retention controls need approval and do not cover every endpoint | Your VPC, with retention set by your own policy |
| Evidence of what happened | Your own application logs, plus whatever the vendor exposes about its side of each call | Records generated inside your boundary for every call, including the serving side, kept under your retention policy |
| Model versions | Set by the vendor's schedule. OpenAI's deprecations page commits to at least six months' notice for generally available models, with exceptions for safety or compliance | You pin the weights and decide when to change them |
| Capability | Frontier models may lead on the hardest tasks | Varies by model, so evaluate on your tasks |
| Cost shape | Per-token, and it scales to zero | Capacity-based, so GPU utilization decides the economics |
| Time to a first agent | Fast | Slower |
On cost alone, self-hosting tends to pay when utilization is high and steady. At low or spiky volume, per-token pricing often wins.
Skip rung 4 when volume is low, when your hardest tasks need frontier capability you have not matched with an open-weight model, or when the team cannot yet operate a serving stack. A hybrid arrangement is the usual answer in those cases: sensitive workloads go to models inside your boundary and the rest to vendor APIs. A control plane can apply one policy across both, but a request sent to a vendor still leaves your boundary, so route by data sensitivity and not only by convenience. Prediction Guard's technical evaluation guide works through that decision for regulated workloads.
Few regulated teams end up at one rung, and self-hosted AI agents on open-weight models are one option on that spectrum, strongest where data, versions, and evidence must stay inside. The deciding factor is not simply which model you run or where it runs. It is whether you can demonstrate what each agent was authorized to do, enforce those boundaries at runtime, and produce the evidence to prove it. That is operational control of agents, at any rung.
Self-hosted AI agents are agents whose model inference, tool access, policy enforcement, and evidence run on infrastructure you control, such as GPU nodes inside your own cloud account. The agent code can still be built with familiar frameworks, because many of them can target a self-hosted endpoint by changing the base URL.
An open-weight model has its trained parameters published for download and local use. That is a narrower promise than open source: the Open Source Initiative's Open Source AI Definition 1.0 also asks for training code and data information, so a model can ship under a permissive license and still not meet that bar.
There are five rungs, from a public vendor API to an air-gapped environment. For regulated workloads where data and evidence must stay inside without owning a data center, rung 4, self-hosted inference in your own cloud account, is usually the fit. Few teams end up at one rung, and a hybrid that routes by data sensitivity is common.
The model supply chain, safer model handling such as preferring Safetensors and pinning hashes, the security and patching of the serving layer, default-deny egress, scoped tool access, and defenses against prompt injection, which self-hosting does not remove.
Skip self-hosting when volume is low, when your hardest tasks need frontier capability you have not matched with an open-weight model, or when your team cannot yet operate a serving stack. Hybrid arrangements are the usual answer: sensitive workloads go to models inside your boundary and the rest to vendor APIs.