Inference providers#

Agents need a model. Lens Agents can proxy every LLM call an agent makes, which is what makes spending limits, PII masking, and per-call audit possible. That is managed inference: the provider credential is held by the install or by the organization, it is never exposed to the sandbox, and each call passes through the platform's LLM proxy.

Managed inference is opt-in per policy. Configuring a provider here makes it available; a policy has to select it before any sandbox can use it.

Supported providers#

Provider key	Backend	Model families
`bedrock`	Amazon Bedrock	Claude
`azure`	Microsoft Foundry	Claude (Anthropic Messages API) and GPT (OpenAI Chat Completions and Responses APIs)
`bedrock-mantle`	Amazon Bedrock Mantle	Claude and GPT off the same Bedrock key
`openai`	OpenAI	GPT
`openrouter`	OpenRouter	Roughly 400 models from around 60 vendors, on either wire format
`litellm`	Your own LiteLLM proxy	The model names your proxy defines, on either wire format

Ask which providers an organization without its own keys can use on a running install:

curl -fsS -H "Authorization: Bearer $LENS_AGENTS_TOKEN" \
  https://agents.example.com/v1/inference/providers

{ "providers": ["bedrock", "bedrock-mantle"], "defaultProvider": "bedrock" }

The list is empty when inference.deploymentKeyFallback is false. An organization with its own keys can use more; GET /v1/orgs/<org-id>/inference/providers names each backend the install serves and whose key it uses for that organization. A policy selecting a backend that no key applies to gets 404 from the proxy. See Per-organization keys.

defaultProvider is the backend a policy gets when it enables managed inference and names no provider: bedrock when it is in the list, otherwise the first backend listed. It is bedrock when inference.deploymentKeyFallback is false. An organization that can use a Bedrock key, its own or the install's, gets Bedrock. With no default, such a policy reaches no backend, and inference is refused with 403.

Endpoints in Helm, keys in Helm or the admin UI#

Helm owns each backend's endpoint. It does not have to own the key.

Provider	Needed in Helm
`bedrock`, `bedrock-mantle`, `openrouter`	Nothing. Always served.
`openai`	Nothing. `inference.openai.baseUrl` is optional and defaults to `https://api.openai.com`.
`azure`	`inference.azure.baseUrl`.
`litellm`	`inference.litellm.baseUrl`.

A key reaches a backend in one of two ways:

Install-level, from inference.<provider>.token or a Kubernetes Secret named in inference.<provider>.existingSecret, or for native Bedrock an IAM role. Every organization can use it, unless inference.deploymentKeyFallback is false.
Per organization, added by an org admin on the Inference Keys page of the admin UI. No kubectl, no helm upgrade, no restart. The organization's key wins over the install-level one.

Endpoints are not configurable from the UI, on purpose: an organization can bring a key, never a destination. See Per-organization keys.

AWS Bedrock#

Bedrock is the default provider wherever it is available. At the install level it needs credentials, resolved one of two ways; an organization can also bring its own key.

On a cluster with an IAM role#

On EKS, or any cluster where the platform's service account can assume a role, attach the role and leave inference.bedrock empty. The AWS SDK default provider chain resolves it.

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=\
"arn:aws:iam::123456789012:role/lens-agents-bedrock" \
  ...

This is the preferred shape: no long-lived key is stored in the cluster at all.

Without a token, the platform offers Bedrock only when the AWS SDK default chain finds credentials at boot, such as IRSA, EKS Pod Identity, or an instance role. Override that check with inference.bedrock.ambientCredentials:

Value	Default	Purpose
`inference.bedrock.ambientCredentials`	`""`	Empty: serve Bedrock on the pod's AWS credentials when the check finds them. `"true"`: serve it where the check cannot see a credential source the SDK will find. `"false"`: do not serve it, for a role with no Bedrock access.

With a Bedrock API key#

Where there is no IRSA or instance role — local clusters, on-prem clusters outside AWS — supply an Amazon Bedrock API key. The platform forwards it upstream as a bearer token.

kubectl create secret generic lens-agents-bedrock \
  --from-literal=NEXUS_BEDROCK_TOKEN="<bedrock-api-key>"

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.bedrock.existingSecret=lens-agents-bedrock \
  ...

The Secret is optional: skip it and an org admin adds the key on Inference Keys instead.

A Bedrock API key — the install's or an organization's — also serves Bedrock Mantle. Mantle is AWS's Anthropic- and OpenAI-compatible endpoint, serving both Claude and GPT off the same key. It needs no extra configuration — the host is derived from the request region. Mantle needs the API key itself: an IAM role covers native Bedrock only.

Microsoft Foundry#

One Foundry resource and key serve both model families. A policy selecting azure is authorized for both the Claude and GPT surfaces.

kubectl create secret generic lens-agents-azure \
  --from-literal=NEXUS_AZURE_TOKEN="<azure-api-key>"

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.azure.baseUrl="https://<resource>.services.ai.azure.com" \
  --set inference.azure.existingSecret=lens-agents-azure \
  ...

The Secret is optional: leave out existingSecret and an org admin adds the key on Inference Keys. The baseUrl is still required.

Use the Foundry resource root, not the Azure OpenAI host

inference.azure.baseUrl is the resource root on the *.services.ai.azure.com host — not the Azure OpenAI *.openai.azure.com host. Leave the surface suffix off: the proxy appends /anthropic for Claude and /openai for GPT itself. Copy the host from a deployment's Target URI in the Foundry portal; the key is that deployment's Key.

The platform sets no model on either surface: the agent names a Foundry deployment itself. See No model is set.

OpenAI#

For an install holding its own OpenAI platform account. An API key is all it needs. Without an install-level key, only an organization with its own key can use the provider.

kubectl create secret generic lens-agents-openai \
  --from-literal=NEXUS_OPENAI_TOKEN="<openai-api-key>"

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.openai.existingSecret=lens-agents-openai \
  ...

The Secret is optional: skip it and an org admin adds the key on Inference Keys.

This provider serves GPT only. OpenAI hosts no Claude, so a sandbox under a policy selecting openai gets the OpenAI-compatible endpoint and no managed Anthropic one. Installs that need both model families choose Foundry or Bedrock Mantle instead.

Only the paths whose spend can be metered are proxied: /v1/chat/completions, /v1/responses, and /v1/embeddings. Any other path returns 404 naming those three, because batches, fine-tuning, and uploads bill the key without reporting token usage, which would put spend outside the reach of spending limits. Realtime voice runs over a separate WebSocket route; see Realtime voice sessions.

Pointing at a compatible endpoint#

inference.openai.baseUrl points the backend at any OpenAI-compatible endpoint — a corporate gateway, a self-hosted server — instead of the vendor's platform. For a LiteLLM proxy, use the LiteLLM backend instead: it serves the Anthropic wire format too, and meters the cost LiteLLM reports rather than a rate-card price.

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.openai.baseUrl="https://gateway.example.com/openai" \
  --set inference.openai.existingSecret=lens-agents-openai \
  ...

The agent's request path is appended to the root unchanged, so a root carrying its own prefix keeps it. The root must be https, because the API key travels upstream as a bearer token. A baseUrl with no install-level key is valid: organizations then bring their own keys.

What the sandbox receives#

A sandbox under a policy selecting openai starts with the variables the OpenAI SDK reads, so an unmodified agent needs no configuration of its own.

Variable	Value
`OPENAI_BASE_URL`	`<platform-url>/v1/projects/<project-id>/llm/openai/v1`
`OPENAI_API_KEY`	An inert placeholder. The sandbox proxy replaces the `Authorization` header with the sandbox's own platform token on the proxy path, and the platform forwards the real OpenAI key upstream.

The trailing /v1 is load-bearing. The OpenAI SDK treats OPENAI_BASE_URL as already version-scoped and appends only /chat/completions, so a base URL that stops short of the version segment makes every managed call return 404.

No OPENAI_MODEL is set, because the SDK takes its model per request. No ANTHROPIC_BASE_URL is set either, because this backend hosts no Claude. The platform sets each variable only where the policy's own env does not already carry one, so a value set in the policy wins. A value baked into the agent image does not — the platform layers its env over the image's.

OpenRouter#

OpenRouter is a reseller in front of roughly 60 vendors. One API key reaches roughly 400 models — Anthropic, OpenAI, Google, Meta, DeepSeek, Mistral, xAI — with no cloud account, IAM role, or region to decide first. It is the shortest path from an empty cluster to an agent that answers.

kubectl create secret generic lens-agents-openrouter \
  --from-literal=NEXUS_OPENROUTER_TOKEN="<openrouter-api-key>"

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.openrouter.existingSecret=lens-agents-openrouter \
  ...

There is nothing else to configure, and the Secret is optional: skip it and an org admin adds the key on Inference Keys. Without an install-level key, only an organization with its own key can use the provider.

Both wire formats, one key#

Foundry and Bedrock Mantle also serve two surfaces, but there each surface is a model family: the Anthropic surface serves Claude and the OpenAI surface serves GPT. OpenRouter is different. It normalizes whichever model it routes to into the format that asked for it, so the surface an agent speaks does not constrain the model it can drive.

An agent built on the Anthropic SDK — Claude Code included — can therefore drive a Gemini or Llama model, and an agent built on the OpenAI SDK can drive Claude. A sandbox on this backend is seeded with both endpoints and the agent's own SDK picks.

Choosing a model is the agent's job#

A policy grants access to the backend. It does not name a model, and the platform sets none. The agent names one per request as vendor/model — google/gemini-3-pro, deepseek/deepseek-v3.2.

To pin a model for a whole project, set ANTHROPIC_MODEL in the policy's env, where every other agent-facing variable already lives. See No model is set.

What the sandbox receives#

Variable	Value
`ANTHROPIC_BASE_URL`	`<platform-url>/v1/projects/<project-id>/llm/openrouter/anthropic`
`OPENAI_BASE_URL`	`<platform-url>/v1/projects/<project-id>/llm/openrouter/openai/v1`
`OPENROUTER_BASE_URL`	The same OpenAI-compatible root, under the name a client that supports OpenRouter natively looks for.
`OPENROUTER_API_KEY`	A credential placeholder the sandbox arms on the proxy path, replacing the `Authorization` header with the sandbox's own platform token.
`ANTHROPIC_API_KEY`	The literal `nexus-managed`. An ordinary string rather than a placeholder, satisfying a client that insists on the variable being set.

The trailing /v1 on the OpenAI-shaped roots is load-bearing, and its absence on ANTHROPIC_BASE_URL equally so: the Anthropic SDK appends /v1/messages to its base, while the OpenAI SDK treats its base as already version-scoped and appends only /chat/completions.

ANTHROPIC_API_KEY is inert by construction: the proxy overwrites Authorization with the real credential, and these surfaces drop a client-supplied x-api-key outright, so the value authenticates to nothing.

Nothing here speaks AWS, so no CLAUDE_CODE_USE_BEDROCK and no SigV4 placeholder keys are set. No model variable is set on either surface.

A client that supports OpenRouter natively is best configured as OpenRouter, using OPENROUTER_BASE_URL and OPENROUTER_API_KEY. Configured that way it lists the catalogue under OpenRouter and leaves its Anthropic slot for real Anthropic. It also sends the key as Authorization: Bearer, which is the header this credential's injection arms — a key sent as x-api-key stays an unarmed placeholder the sandbox refuses to forward.

Billed on what OpenRouter charged#

OpenRouter states its cost on every response, on both surfaces, streamed or not. That reported figure is what cost aggregation and spending limits use for this backend, rather than a token count priced against a rate card.

This matters because the model most likely to be missing from a rate card is the newest one, which is exactly what someone evaluating the platform reaches for. A model the table cannot price meters as zero and counts toward no cap; a reported cost has no such gap. openrouter/auto and openrouter/fusion — where a request naming no model routes — are among the ids this covers.

Where a request reports no cost, its tokens are still priced from the table, so the other backends behave exactly as they did. Costing is per request rather than per reporting period, so a period spanning the upgrade mixes reported and priced requests without billing either twice. The per-dimension cost breakdown describes only the priced remainder, which is why a fully reported period shows its total with zeros across the breakdown rather than a split the provider never gave.

Deny openrouter.ai in policy

Metering happens because a request travels through the managed endpoint, not because the platform intercepts outbound TLS. One key reaching 60 vendors makes a direct dial to openrouter.ai the cheapest way for an agent to spend money nothing meters. Keep the host denied — which is the default — so the agent can reach it only through the managed endpoint. See Metering follows routing.

LiteLLM#

For an install that already runs its own LiteLLM proxy. The platform authenticates to it with one LiteLLM virtual key or master key, and the proxy's own routing, deployments, and pricing stay yours. There is no default endpoint, so baseUrl is required:

kubectl create secret generic lens-agents-litellm \
  --from-literal=NEXUS_LITELLM_TOKEN="<litellm-key>"

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.litellm.baseUrl="https://litellm.example.com" \
  --set inference.litellm.existingSecret=lens-agents-litellm \
  ...

The Secret is optional: leave out existingSecret and an org admin adds the key on Inference Keys. The baseUrl is still required.

The root must be https, because the key travels upstream as a bearer token. A trailing /v1 is accepted. The install fails when a key or forwardTags is set without baseUrl. A baseUrl with no install-level key serves only organizations with their own key.

Both wire formats, your model names#

Like OpenRouter, LiteLLM translates between wire formats, so each surface is the caller's SDK rather than a model family. A sandbox on this backend is seeded with both endpoints and the agent's own SDK picks.

Model names are the ones your proxy defines, and the platform sets none. Name one of them in the agent's own configuration, or pin it for a project with ANTHROPIC_MODEL in the policy's env. See No model is set.

Only the metered paths are proxied: /v1/messages and /v1/messages/count_tokens on the Anthropic surface, /v1/chat/completions, /v1/responses, and /v1/embeddings on the OpenAI surface, and GET /v1/models on both. Any other path returns 404.

LiteLLM's x-litellm-* headers are its control channel, and the sandbox gets neither side of it. Request headers of that shape, and tags in the request body, are removed before the call goes upstream, so an agent cannot pick a key, a tag, or a deployment. Response headers of that shape, which carry the key's spend and the upstream deployment URL, do not reach the sandbox.

What the sandbox receives#

Variable	Value
`ANTHROPIC_BASE_URL`	`<platform-url>/v1/projects/<project-id>/llm/litellm/anthropic`
`OPENAI_BASE_URL`	`<platform-url>/v1/projects/<project-id>/llm/litellm/openai/v1`
`ANTHROPIC_AUTH_TOKEN`, `OPENAI_API_KEY`	Credential placeholders the sandbox arms on the proxy path, replacing the `Authorization` header with the sandbox's own platform token.
`ANTHROPIC_API_KEY`	The literal `nexus-managed`, inert for the same reason as on OpenRouter.

The trailing /v1 rule is the one OpenRouter follows: present on the OpenAI root, absent on the Anthropic one.

Billed on what LiteLLM reports#

LiteLLM spend is metered only by the cost LiteLLM reports, never by the bundled rate card, because LiteLLM model names are your own aliases and the prices are yours too. A complete response carries that cost in its x-litellm-response-cost header. A stream carries it only when the LiteLLM config asks for it:

litellm_settings:
  include_cost_in_streaming_usage: true

A call whose cost is missing, or 0 for a model LiteLLM has no price for, is recorded at unknown cost: its tokens reach the audit trail, it counts against no spending limit, and the platform logs a warning. A streamed /v1/messages cost is used only when the same usage carries the input tokens, because without them LiteLLM has priced the output alone.

Set that option, and give every model a price in its model_info, before you put a capped sandbox on this backend.

Spend by project#

Set inference.litellm.forwardTags=true and each call is tagged project:<org-slug>/<project-slug>, so LiteLLM's own spend reports break down by project. It defaults to off.

Leave it off where your proxy sets router_settings.enable_tag_filtering. LiteLLM then routes by tag, and refuses a call whose tag no deployment has.

Realtime voice sessions#

Two backends relay a realtime speech API over a WebSocket, so an agent can hold a low-latency speech session. The provider selection is the only switch — there is no separate voice setting.

Backend	WebSocket route
OpenAI	`wss://agents.example.com/v1/projects/<project-id>/llm/openai/v1/realtime`
Microsoft Foundry	`wss://agents.example.com/v1/projects/<project-id>/llm/azure/openai/v1/realtime?model=<deployment-name>`
OpenAI — GPT-Live	`wss://agents.example.com/v1/projects/<project-id>/llm/openai/v1/live/sessions`
Microsoft Foundry — GPT-Live	`wss://agents.example.com/v1/projects/<project-id>/llm/azure/openai/v1/live/sessions`

A session needs a policy that grants that backend, as primary or in providers, and there is no fallback between the two. Each route is the backend's OpenAI-compatible base URL with /realtime or /live/sessions appended and the scheme switched to wss. Where the backend is primary, that base URL is OPENAI_BASE_URL; where it is a secondary backend, it is LENS_INFERENCE_OPENAI_BASE_URL or LENS_INFERENCE_AZURE_OPENAI_BASE_URL. Granting a backend does not configure an agent's voice client: the agent dials the route and, on Foundry Realtime, names the deployment.

Foundry runs over the existing inference.azure resource and key, with no separate endpoint or key. Its model query parameter names an Azure Realtime deployment and is required: a request with no model, more than one, or any other query parameter is refused with 400.

GPT-Live sessions#

OpenAI's GPT-Live sessions run over the same backends, gates, and relay as Realtime, at /live/sessions in place of /realtime. A session names its model when it starts, so the route takes no query: on Foundry, a query parameter is refused with 400.

GPT-Live bills voice by the second, not by the token. The platform records voice time at gpt-live-1's $0.05 per minute for every model on this route, Foundry deployment names included, and logs a warning when a session names another model. The tokens of the model a session delegates its responses to are priced from the rate card, like any Responses call.

The path, bearer token, project access, policy, and concurrency checks all run before the upstream connection opens, so a refused session never opens a billing connection. Each response writes an audit trail record in the shape the HTTP path writes, attributed per response rather than to the session as a whole. GPT-Live voice time is recorded as each usage report arrives: about once a minute, and once more when the session closes.

Four behaviours differ from the HTTP path:

PII masking does not run. Where a policy asks for masking, the session is served unmasked and every audit record carries piiMaskingSkipped. The HTTP path fails closed instead. See Privacy and PII controls.
The spending limit gates the open, then acts as a backstop. A caller already over its limit is refused with 429 before the upstream connection opens. Once the session runs it reports usage only as it goes, so the limit is re-checked after each usage record and closes the session once passed, trailing actual spend by roughly 90 seconds.
Concurrency is capped at eight sessions per project, shared across both backends and both routes, and counted per replica. The cap is not a Helm value.
Foundry costs are estimates, not Azure invoice amounts. Both backends are priced from the OpenAI rate card. A model id the upstream reports wins over the requested deployment name; where only a custom deployment name is known and no rate matches it, the usage is recorded at zero cost and counts toward no cap.

More than one backend at a time#

A policy can enable several backends for the same sandbox, as shown in Turning it on for agents. providers must be non-empty, name each supported provider once, and include the provider named beside it. A list with no explicit primary is refused rather than defaulted, because the primary is what decides where the standard variables point. Omit providers and the grant is the single backend provider names.

What the standard variables point at#

A process has one ANTHROPIC_BASE_URL and one OPENAI_BASE_URL, so the platform has to choose a backend for each. It prefers the primary where the primary serves that wire format, and otherwise the first backend in the list that does. Bedrock with OpenAI beside it therefore sets AWS_ENDPOINT_URL_BEDROCK_RUNTIME for the primary and OPENAI_BASE_URL for OpenAI; Foundry primary with OpenAI beside it points OPENAI_BASE_URL at Foundry, and the direct OpenAI backend stays reachable under its own names below. A value the policy's env already carries wins over all of it, as everywhere else.

Native Bedrock is not an Anthropic-compatible surface, so where another enabled backend serves the Anthropic format, that backend supplies ANTHROPIC_BASE_URL and its auth token. With Bedrock primary, Claude Code keeps its Bedrock startup flag. Model ids are backend-specific, so a client reaching a secondary Anthropic surface names a model that backend recognizes.

A secondary backend never sets the process-wide AWS_REGION. Bedrock as a secondary backend still supplies the Bedrock runtime endpoint, which only a Bedrock client reads, but withholds its startup shims: no AWS signing placeholders, no metadata-lookup switch, no Claude Code Bedrock flag. Those belong to a backend that starts the agent, and applying them would reconfigure every unrelated AWS client in the sandbox.

Per-backend variables#

Every enabled backend also arrives under a name of its own, so an agent can address a backend that does not own the standard variables. A single-backend grant carries these too, which is what lets a template address a backend by name before anyone adds a second one. Each credential is a placeholder the sandbox arms on the proxy path, exactly as the standard ones are.

Backend	Endpoint	Credential
Bedrock	`LENS_INFERENCE_BEDROCK_BASE_URL`	`LENS_INFERENCE_BEDROCK_BEARER_TOKEN`
Foundry — Claude	`LENS_INFERENCE_AZURE_ANTHROPIC_BASE_URL`	`LENS_INFERENCE_AZURE_ANTHROPIC_AUTH_TOKEN`
Foundry — GPT	`LENS_INFERENCE_AZURE_OPENAI_BASE_URL`	`LENS_INFERENCE_AZURE_OPENAI_API_KEY`
Bedrock Mantle — Claude	`LENS_INFERENCE_BEDROCK_MANTLE_ANTHROPIC_BASE_URL`	`LENS_INFERENCE_BEDROCK_MANTLE_ANTHROPIC_AUTH_TOKEN`
Bedrock Mantle — GPT	`LENS_INFERENCE_BEDROCK_MANTLE_OPENAI_BASE_URL`	`LENS_INFERENCE_BEDROCK_MANTLE_OPENAI_API_KEY`
OpenAI	`LENS_INFERENCE_OPENAI_BASE_URL`	`LENS_INFERENCE_OPENAI_API_KEY`
OpenRouter — Anthropic format	`LENS_INFERENCE_OPENROUTER_ANTHROPIC_BASE_URL`	`LENS_INFERENCE_OPENROUTER_ANTHROPIC_AUTH_TOKEN`
OpenRouter — OpenAI format	`LENS_INFERENCE_OPENROUTER_BASE_URL`	`LENS_INFERENCE_OPENROUTER_API_KEY`
LiteLLM — Anthropic format	`LENS_INFERENCE_LITELLM_ANTHROPIC_BASE_URL`	`LENS_INFERENCE_LITELLM_ANTHROPIC_AUTH_TOKEN`
LiteLLM — OpenAI format	`LENS_INFERENCE_LITELLM_OPENAI_BASE_URL`	`LENS_INFERENCE_LITELLM_OPENAI_API_KEY`

An Anthropic surface adds one more: ..._ANTHROPIC_API_KEY, the inert string a client that insists on the variable needs. Authenticate with the auth token — the API-key entry is not the placeholder the sandbox arms. Bedrock and Mantle add ..._REGION. The OpenAI-shaped roots already carry their /v1, as the standard ones do.

No routing, no failover#

A request reaching a backend the policy did not enable is refused with 403 naming the backends it did, on the HTTP and realtime routes alike, and a backend that fails stays a failed request rather than becoming a retry somewhere else. Nothing selects a backend from the model an agent asked for. Making the second backend reachable and naming it in the environment is the whole of what this adds; using it is the agent's own configuration.

Metering is unchanged by the count. Every backend is priced and attributed on its own, and all of them count against the same applicable spending limits — enabling a second one widens what an agent can reach without widening what it may spend.

An agent reads these variables when its process starts. Change a policy's providers and restart the agent, or it keeps the environment it was given.

What every backend tells the agent#

Beyond the provider-shaped variables above, a sandbox with managed inference enabled receives the selected backend by name.

Variable	Value
`LENS_MANAGED_INFERENCE_PROVIDER`	The primary provider key the policy selected: `bedrock`, `azure`, `bedrock-mantle`, `openai`, `openrouter`, or `litellm`.
`LENS_MANAGED_INFERENCE_PROVIDERS`	Every backend the policy enables, as a JSON array: the policy's `providers` list as written, or the primary alone.

The endpoints alone are ambiguous — azure, bedrock-mantle, openrouter, and litellm all seed ANTHROPIC_BASE_URL, yet their model-id conventions differ (claude-opus-4-8 on a Foundry deployment, anthropic.claude-sonnet-4-6-v1 on Mantle, vendor/model on OpenRouter, your own names on LiteLLM). An agent that guessed the backend from the URL shape would send an id the backend does not recognize.

Naming it explicitly also survives a policy edit. A sandbox template is authored before anyone selects a provider and its env is static, so these are resolved per sandbox rather than baked in: change managedInference.provider and the sandbox follows, instead of pinning whatever the template's author assumed.

No model is set#

The platform sets no model on any backend: no ANTHROPIC_MODEL, and no model variable under a per-backend name. A policy grants a backend, not a model, and the agent chooses one per request.

To pin a model for a project, set ANTHROPIC_MODEL — or whichever variable the agent's harness reads for its model — in the policy's env. The platform passes it to the sandbox unchanged. Use an id the backend recognizes, in the conventions above.

Metering follows routing#

Managed inference is metered, budgeted, and PII-masked because the request travels through the platform's inference endpoint — not because the platform transparently intercepts a sandbox's outbound TLS. The forward proxy authenticates the connection and pipes it; it does not read the body.

The consequence is worth stating plainly. If a policy grants a sandbox direct network access to a provider's runtime — an allow entry for bedrock-runtime.*.amazonaws.com, bedrock-mantle.*.api.aws, an Azure Foundry host, openrouter.ai, or your LiteLLM proxy's host — that traffic is neither metered nor gated by a spending limit. It is governed only by the network policy that let it out, and it writes no usage record, so a cap set on that sandbox reads as a cap on its spend and is not.

This is deliberate: what a sandbox may reach directly is a policy decision, and metering is a consequence of choosing the managed endpoint. To guarantee coverage, leave the provider host denied, which is the default posture. The guardrail is a deny rule, not silent interception.

Per-organization keys#

The install owns each backend's endpoint. An organization can own the key: an org admin sets the organization's own API key for a key source, and calls from that organization's projects go upstream with it.

Key source	Backends it serves
`bedrock`	Bedrock and Bedrock Mantle
`azure`	Both Foundry surfaces and Foundry realtime voice
`openai`	OpenAI and OpenAI realtime voice
`openrouter`	OpenRouter
`litellm`	LiteLLM

An organization key replaces only the credential. The endpoint stays the install's, and a key names no model. A key is accepted only for a source the install serves: bedrock, openai, and openrouter always are, while azure and litellm need their baseUrl.

Which key a call uses#

The organization's own key for the source.
The install-level key, when inference.deploymentKeyFallback is true, which is the default. For native Bedrock, an IAM role counts here too; Bedrock Mantle needs an API key.
Otherwise no key applies, and the proxy refuses the call with 404, stating that the backend is not configured for this organization. Nothing reaches the provider.

Install-level keys are optional. A backend with an endpoint and no install-level key serves only the organizations that have their own. Set inference.deploymentKeyFallback=false where every organization pays with its own key: an organization without one then cannot use that backend, including native Bedrock through an IAM role.

A changed key is in use on every replica within 30 seconds.

Setting a key#

Only an org admin signed in through OIDC can list, set, or remove the organization's keys. API tokens and sandbox tokens are refused, so an agent cannot point its own inference at a key it chose. Any org member can list which backends the organization can use.

In the admin UI, an org admin opens Inference Keys in the sidebar. The page shows each key source the install serves, and the admin can Add, Replace, or Remove the organization's key for it. The change is live on every replica within 30 seconds. The same operations are available over REST:

curl -fsS -X PUT -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"token": "<openrouter-api-key>"}' \
  https://agents.example.com/v1/orgs/<org-id>/inference/keys/openrouter

$TOKEN is the OIDC JWT of a signed-in org admin; an API token is refused. DELETE on the same path removes the key, and the organization falls back to the install-level key, if the fallback allows it. GET /v1/orgs/<org-id>/inference/keys lists the keys that are set.

A key is stored encrypted and never returned: a listing shows only its last four characters. A key shorter than 16 characters is refused. Every set and removal writes an audit trail record, org.inference-key.set or org.inference-key.remove, that names the key source and never carries the key.

Values#

Value	Default	Purpose
`inference.deploymentKeyFallback`	`true`	Let an organization without its own key use the install-level key, or for native Bedrock the AWS SDK default chain. `false` closes the backend to it.
`inference.bedrock.token`	(empty)	Bedrock API key, forwarded as a bearer token. Empty falls back to the AWS SDK default chain.
`inference.bedrock.existingSecret`	(empty)	Read the token from an existing Secret instead.
`inference.bedrock.existingSecretKey`	`NEXUS_BEDROCK_TOKEN`	Key within that Secret.
`inference.azure.baseUrl`	(empty)	Foundry resource root. Empty disables Azure.
`inference.azure.token`	(empty)	Azure API key, forwarded as the `api-key` header. Optional; requires `baseUrl`.
`inference.azure.existingSecret`	(empty)	Read the token from an existing Secret instead.
`inference.azure.existingSecretKey`	`NEXUS_AZURE_TOKEN`	Key within that Secret.
`inference.openai.token`	(empty)	OpenAI API key, forwarded as a bearer token. Empty leaves OpenAI to organizations with their own key.
`inference.openai.existingSecret`	(empty)	Read the token from an existing Secret instead.
`inference.openai.existingSecretKey`	`NEXUS_OPENAI_TOKEN`	Key within that Secret.
`inference.openai.baseUrl`	(empty)	Endpoint root, which must be `https`. Empty targets `https://api.openai.com`.
`inference.openrouter.token`	(empty)	OpenRouter API key, forwarded as a bearer token. Empty leaves OpenRouter to organizations with their own key.
`inference.openrouter.existingSecret`	(empty)	Read the token from an existing Secret instead.
`inference.openrouter.existingSecretKey`	`NEXUS_OPENROUTER_TOKEN`	Key within that Secret.
`inference.litellm.baseUrl`	(empty)	LiteLLM proxy root, which must be `https`. Empty disables LiteLLM.
`inference.litellm.token`	(empty)	LiteLLM virtual key or master key, forwarded as a bearer token. Optional; requires `baseUrl`.
`inference.litellm.existingSecret`	(empty)	Read the token from an existing Secret instead.
`inference.litellm.existingSecretKey`	`NEXUS_LITELLM_TOKEN`	Key within that Secret.
`inference.litellm.forwardTags`	`false`	Tag each call `project:<org-slug>/<project-slug>`. Leave off where LiteLLM routes by tag.

Prefer existingSecret over token in every environment. A --set value lands in your shell history and in the Helm release.

Turning it on for agents#

Configuring a provider is half the job. A policy has to enable managed inference and name the provider:

name: agent-base
managedInference:
  enabled: true
  provider: bedrock

Sandboxes governed by that policy reach the model through the platform. Sandboxes without it have no model access at all.

A policy can enable more than one backend. provider names the primary, which drives agent startup, and providers lists every enabled backend, the primary among them. The standard SDK variables prefer the primary, then the other backends in list order:

managedInference:
  enabled: true
  provider: litellm
  providers: [litellm, openai]

Every enabled backend also gets its own endpoint and placeholder credential, and choosing between backends is the agent's job. See More than one backend at a time.

Three places must agree, or the agent will not answer: a provider the install serves with a key for the organization, the provider named in the policy, and whatever provider the agent image itself is told to use through its environment.

An agent that reads LENS_MANAGED_INFERENCE_PROVIDER gets the third for free — it is told which backend it was given rather than having to be configured to match. See What every backend tells the agent.

What you get from proxying#

Because every call passes through the platform:

Spending is metered per call and enforced against spending limits by org, user, agent, or sandbox.
PII can be masked before the prompt leaves your network, and unmasked in the response. See Privacy and PII controls.
Every call is audited with the actor, project, and cost. See Audit trail.

None of it applies to a call that skipped the endpoint. See Metering follows routing.

Additional providers are configured to customer requirements. Talk to us.

Models — how agents select models at runtime
Spending limits — caps by org, user, agent, and sandbox
Policies — enabling managed inference for a set of sandboxes

Inference providers#

Supported providers#

Endpoints in Helm, keys in Helm or the admin UI#

AWS Bedrock#

On a cluster with an IAM role#

With a Bedrock API key#

Microsoft Foundry#

OpenAI#

Pointing at a compatible endpoint#

What the sandbox receives#

OpenRouter#

Both wire formats, one key#

Choosing a model is the agent's job#

What the sandbox receives#

Billed on what OpenRouter charged#

LiteLLM#

Both wire formats, your model names#

What the sandbox receives#

Billed on what LiteLLM reports#

Spend by project#

Realtime voice sessions#

GPT-Live sessions#

More than one backend at a time#

What the standard variables point at#

Per-backend variables#

No routing, no failover#

What every backend tells the agent#

No model is set#

Metering follows routing#

Per-organization keys#

Which key a call uses#

Setting a key#

Values#

Turning it on for agents#

What you get from proxying#

Related#