Inference providers#

Agents need a model. Lens Agents can proxy every LLM call an agent makes, which is what makes spending limits, PII masking, and per-call audit possible. That is managed inference: the platform holds the provider credential, the agent never sees it, and each call passes through the platform's LLM proxy.

Managed inference is opt-in per policy. Configuring a provider here makes it available; a policy has to select it before any sandbox can use it.

Supported providers#

Provider key	Backend	Model families
`bedrock`	Amazon Bedrock	Claude
`azure`	Microsoft Foundry	Claude (Anthropic Messages API) and GPT (OpenAI Chat Completions and Responses APIs)
`bedrock-mantle`	Amazon Bedrock Mantle	Claude and GPT off the same Bedrock key
`openai`	OpenAI	GPT
`openrouter`	OpenRouter	Roughly 400 models from around 60 vendors, on either wire format

Ask which providers a running install has configured:

curl -fsS -H "Authorization: Bearer $LENS_AGENTS_TOKEN" \
  https://agents.example.com/v1/inference/providers

{ "providers": ["bedrock", "bedrock-mantle"] }

A policy may only select a provider in that list. Selecting anything else has no proxy route behind it and returns 404.

AWS Bedrock#

Bedrock is the default provider. It needs credentials, resolved one of two ways.

On a cluster with an IAM role#

On EKS, or any cluster where the platform's service account can assume a role, attach the role and leave inference.bedrock empty. The AWS SDK default provider chain resolves it.

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=\
"arn:aws:iam::123456789012:role/lens-agents-bedrock" \
  ...

This is the preferred shape: no long-lived key is stored in the cluster at all.

With a Bedrock API key#

Where there is no IRSA or instance role — local clusters, on-prem clusters outside AWS — supply an Amazon Bedrock API key. The platform forwards it upstream as a bearer token.

kubectl create secret generic lens-agents-bedrock \
  --from-literal=NEXUS_BEDROCK_TOKEN="<bedrock-api-key>"

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.bedrock.existingSecret=lens-agents-bedrock \
  ...

Setting a Bedrock token also exposes Bedrock Mantle as a selectable provider. Mantle is AWS's Anthropic- and OpenAI-compatible endpoint, serving both Claude and GPT off the same key. It needs no extra configuration — the host is derived from the request region.

Microsoft Foundry#

One Foundry resource and key serve both model families. A policy selecting azure is authorized for both the Claude and GPT surfaces.

kubectl create secret generic lens-agents-azure \
  --from-literal=NEXUS_AZURE_TOKEN="<azure-api-key>"

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.azure.baseUrl="https://<resource>.services.ai.azure.com" \
  --set inference.azure.existingSecret=lens-agents-azure \
  ...

Use the Foundry resource root, not the Azure OpenAI host

inference.azure.baseUrl is the resource root on the *.services.ai.azure.com host — not the Azure OpenAI *.openai.azure.com host. Leave the surface suffix off: the proxy appends /anthropic for Claude and /openai for GPT itself. Copy the host from a deployment's Target URI in the Foundry portal; the key is that deployment's Key.

inference.azure.anthropic.model sets the default Claude deployment name for the /anthropic surface. GPT takes its model per request, so there is no openai counterpart.

OpenAI#

For an install holding its own OpenAI platform account. An API key is all it needs — the key is what makes the provider selectable.

kubectl create secret generic lens-agents-openai \
  --from-literal=NEXUS_OPENAI_TOKEN="<openai-api-key>"

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.openai.existingSecret=lens-agents-openai \
  ...

This provider serves GPT only. OpenAI hosts no Claude, so a sandbox under a policy selecting openai gets the OpenAI-compatible endpoint and no managed Anthropic one. Installs that need both model families choose Foundry or Bedrock Mantle instead.

Only the paths whose spend can be metered are proxied: /v1/chat/completions, /v1/responses, and /v1/embeddings. Any other path returns 404 naming those three, because batches, fine-tuning, and uploads bill the key without reporting token usage, which would put spend outside the reach of spending limits. Realtime voice runs over a separate WebSocket route, described below.

Pointing at a compatible endpoint#

inference.openai.baseUrl points the backend at any OpenAI-compatible endpoint — a corporate gateway, LiteLLM, a self-hosted server — instead of the vendor's platform.

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.openai.baseUrl="https://gateway.example.com/openai" \
  --set inference.openai.existingSecret=lens-agents-openai \
  ...

The agent's request path is appended to the root unchanged, so a root carrying its own prefix keeps it. The root must be https — the API key travels upstream as a bearer token — and it needs a key beside it: setting baseUrl with no token or existingSecret fails the install rather than registering an endpoint that nothing can reach.

Realtime voice sessions#

OpenAI's realtime speech API is relayed over a WebSocket, so an agent under a policy selecting openai can hold a low-latency speech session. The provider selection is the only switch — there is no separate voice setting.

wss://agents.example.com/v1/projects/<project-id>/llm/openai/v1/realtime

The path, bearer token, project access, policy, and concurrency checks all run before the upstream connection opens, so a refused session never opens a billing connection. Each response writes an audit trail record in the shape the HTTP path writes, attributed per response rather than to the session as a whole.

Three behaviours differ from the HTTP path:

PII masking does not run. Where a policy asks for masking, the session is served unmasked and every audit record carries piiMaskingSkipped. The HTTP path fails closed instead. See Privacy and PII controls.
The spending limit gates the open, then acts as a backstop. A caller already over its limit is refused with 429 before the upstream connection opens. Once the session runs it reports usage only as it goes, so the limit is re-checked per response and closes the session once passed, trailing actual spend by roughly 90 seconds.
Concurrency is capped at eight sessions per project, counted per replica. The cap is not a Helm value.

What the sandbox receives#

A sandbox under a policy selecting openai starts with the variables the OpenAI SDK reads, so an unmodified agent needs no configuration of its own.

Variable	Value
`OPENAI_BASE_URL`	`<platform-url>/v1/projects/<project-id>/llm/openai/v1`
`OPENAI_API_KEY`	An inert placeholder. The sandbox proxy replaces the `Authorization` header with the sandbox's own platform token on the proxy path, and the platform forwards the real OpenAI key upstream.

The trailing /v1 is load-bearing. The OpenAI SDK treats OPENAI_BASE_URL as already version-scoped and appends only /chat/completions, so a base URL that stops short of the version segment makes every managed call return 404.

No OPENAI_MODEL is set, because the SDK takes its model per request. No ANTHROPIC_BASE_URL is set either, because this backend hosts no Claude. The platform sets each variable only where the policy's own env does not already carry one, so a value set in the policy wins. A value baked into the agent image does not — the platform layers its env over the image's.

OpenRouter#

OpenRouter is a reseller in front of roughly 60 vendors. One API key reaches roughly 400 models — Anthropic, OpenAI, Google, Meta, DeepSeek, Mistral, xAI — with no cloud account, IAM role, or region to decide first. It is the shortest path from an empty cluster to an agent that answers.

kubectl create secret generic lens-agents-openrouter \
  --from-literal=NEXUS_OPENROUTER_TOKEN="<openrouter-api-key>"

helm upgrade --install lens-agents oci://ghcr.io/lensapp/lens-agents \
  --set inference.openrouter.existingSecret=lens-agents-openrouter \
  ...

Setting the token is what makes the provider selectable. There is nothing else to configure.

Both wire formats, one key#

Foundry and Bedrock Mantle also serve two surfaces, but there each surface is a model family: the Anthropic surface serves Claude and the OpenAI surface serves GPT. OpenRouter is different. It normalizes whichever model it routes to into the format that asked for it, so the surface an agent speaks does not constrain the model it can drive.

An agent built on the Anthropic SDK — Claude Code included — can therefore drive a Gemini or Llama model, and an agent built on the OpenAI SDK can drive Claude. A sandbox on this backend is seeded with both endpoints and the agent's own SDK picks.

Choosing a model is the agent's job#

A policy grants access to the backend. It does not name a model.

inference.openrouter.model seeds a starting model into each sandbox as vendor/model — google/gemini-3-pro, deepseek/deepseek-v3.2. Empty uses the built-in default, anthropic/claude-sonnet-5. The seed exists for one narrow reason: the Anthropic SDK refuses to send a request without a model, so a sandbox with none is a dead sandbox. An agent overrides it freely, and the OpenAI SDK ignores it entirely because it takes its model per request.

To pin a model for a whole project, set ANTHROPIC_MODEL in the policy's env, where every other agent-facing variable already lives. The platform only seeds a variable the policy has not already set, so the policy wins.

What the sandbox receives#

Variable	Value
`ANTHROPIC_BASE_URL`	`<platform-url>/v1/projects/<project-id>/llm/openrouter/anthropic`
`OPENAI_BASE_URL`	`<platform-url>/v1/projects/<project-id>/llm/openrouter/openai/v1`
`OPENROUTER_BASE_URL`	The same OpenAI-compatible root, under the name a client that supports OpenRouter natively looks for.
`OPENROUTER_API_KEY`	A credential placeholder the sandbox arms on the proxy path, replacing the `Authorization` header with the sandbox's own platform token.
`ANTHROPIC_API_KEY`	The literal `nexus-managed`. An ordinary string rather than a placeholder, satisfying a client that insists on the variable being set.
`ANTHROPIC_MODEL`	`inference.openrouter.model`, or `anthropic/claude-sonnet-5`.

The trailing /v1 on the OpenAI-shaped roots is load-bearing, and its absence on ANTHROPIC_BASE_URL equally so: the Anthropic SDK appends /v1/messages to its base, while the OpenAI SDK treats its base as already version-scoped and appends only /chat/completions.

ANTHROPIC_API_KEY is inert by construction: the proxy overwrites Authorization with the real credential, and these surfaces drop a client-supplied x-api-key outright, so the value authenticates to nothing.

Nothing here speaks AWS, so no CLAUDE_CODE_USE_BEDROCK and no SigV4 placeholder keys are set. No OPENAI_MODEL either — that SDK has no such convention.

A client that supports OpenRouter natively is best configured as OpenRouter, using OPENROUTER_BASE_URL and OPENROUTER_API_KEY. Configured that way it lists the catalogue under OpenRouter and leaves its Anthropic slot for real Anthropic. It also sends the key as Authorization: Bearer, which is the header this credential's injection arms — a key sent as x-api-key stays an unarmed placeholder the sandbox refuses to forward.

Billed on what OpenRouter charged#

OpenRouter states its cost on every response, on both surfaces, streamed or not. That reported figure is what cost aggregation and spending limits use for this backend, rather than a token count priced against a rate card.

This matters because the model most likely to be missing from a rate card is the newest one, which is exactly what someone evaluating the platform reaches for. A model the table cannot price meters as zero and counts toward no cap; a reported cost has no such gap. openrouter/auto and openrouter/fusion — where a request naming no model routes — are among the ids this covers.

Where a request reports no cost, its tokens are still priced from the table, so the other backends behave exactly as they did. Costing is per request rather than per reporting period, so a period spanning the upgrade mixes reported and priced requests without billing either twice. The per-dimension cost breakdown describes only the priced remainder, which is why a fully reported period shows its total with zeros across the breakdown rather than a split the provider never gave.

Deny openrouter.ai in policy

Metering happens because a request travels through the managed endpoint, not because the platform intercepts outbound TLS. One key reaching 60 vendors makes a direct dial to openrouter.ai the cheapest way for an agent to spend money nothing meters. Keep the host denied — which is the default — so the agent can reach it only through the managed endpoint. See Metering follows routing.

What every backend tells the agent#

Beyond the provider-shaped variables above, a sandbox with managed inference enabled receives the selected backend by name.

Variable	Value
`LENS_MANAGED_INFERENCE_PROVIDER`	The provider key the policy selected: `bedrock`, `azure`, `bedrock-mantle`, `openai`, or `openrouter`.
`LLM_PROVIDER`	The same value, under the name a standalone agent already reads.
`LENS_MANAGED_INFERENCE_MODEL`	The model the sandbox resolved to, under a provider-neutral name.

The endpoints alone are ambiguous — azure, bedrock-mantle, and openrouter all seed ANTHROPIC_BASE_URL, yet their model-id conventions differ (claude-opus-4-8 on a Foundry deployment, anthropic.claude-sonnet-4-6-v1 on Mantle, vendor/model on OpenRouter). An agent that guessed the backend from the URL shape would send an id the backend does not recognize.

Naming it explicitly also survives a policy edit. A sandbox template is authored before anyone selects a provider and its env is static, so these are resolved per sandbox rather than baked in: change managedInference.provider and the sandbox follows, instead of pinning whatever the template's author assumed.

LENS_MANAGED_INFERENCE_MODEL is set for every backend that resolves a portable model id. Native bedrock is excluded, because its ANTHROPIC_MODEL is Claude Code's opus alias, which no other client resolves; openai is excluded because it seeds no model at all. As everywhere else, a value the policy's env already carries wins.

Metering follows routing#

Managed inference is metered, budgeted, and PII-masked because the request travels through the platform's inference endpoint — not because the platform transparently intercepts a sandbox's outbound TLS. The forward proxy authenticates the connection and pipes it; it does not read the body.

The consequence is worth stating plainly. If a policy grants a sandbox direct network access to a provider's runtime — an allow entry for bedrock-runtime.*.amazonaws.com, bedrock-mantle.*.api.aws, an Azure Foundry host, or openrouter.ai — that traffic is neither metered nor gated by a spending limit. It is governed only by the network policy that let it out, and it writes no usage record, so a cap set on that sandbox reads as a cap on its spend and is not.

This is deliberate: what a sandbox may reach directly is a policy decision, and metering is a consequence of choosing the managed endpoint. To guarantee coverage, leave the provider host denied, which is the default posture. The guardrail is a deny rule, not silent interception.

Values#

Value	Default	Purpose
`inference.bedrock.token`	(empty)	Bedrock API key, forwarded as a bearer token. Empty falls back to the AWS SDK default chain.
`inference.bedrock.existingSecret`	(empty)	Read the token from an existing Secret instead.
`inference.bedrock.existingSecretKey`	`NEXUS_BEDROCK_TOKEN`	Key within that Secret.
`inference.azure.baseUrl`	(empty)	Foundry resource root. Empty disables Azure.
`inference.azure.token`	(empty)	Azure API key, forwarded as the `api-key` header.
`inference.azure.existingSecret`	(empty)	Read the token from an existing Secret instead.
`inference.azure.existingSecretKey`	`NEXUS_AZURE_TOKEN`	Key within that Secret.
`inference.azure.anthropic.model`	(empty)	Default Claude deployment name.
`inference.openai.token`	(empty)	OpenAI API key, forwarded as a bearer token. Empty disables OpenAI.
`inference.openai.existingSecret`	(empty)	Read the token from an existing Secret instead.
`inference.openai.existingSecretKey`	`NEXUS_OPENAI_TOKEN`	Key within that Secret.
`inference.openai.baseUrl`	(empty)	Endpoint root, which must be `https`. Empty targets `https://api.openai.com`. Requires a token beside it.
`inference.openrouter.token`	(empty)	OpenRouter API key, forwarded as a bearer token. Empty disables OpenRouter.
`inference.openrouter.existingSecret`	(empty)	Read the token from an existing Secret instead.
`inference.openrouter.existingSecretKey`	`NEXUS_OPENROUTER_TOKEN`	Key within that Secret.
`inference.openrouter.model`	(empty)	Starting model seeded into sandboxes, as `vendor/model`. Empty uses `anthropic/claude-sonnet-5`.

Prefer existingSecret over token in every environment. A --set value lands in your shell history and in the Helm release.

Turning it on for agents#

Configuring a provider is half the job. A policy has to enable managed inference and name the provider:

name: agent-base
managedInference:
  enabled: true
  provider: bedrock

Sandboxes governed by that policy reach the model through the platform. Sandboxes without it have no model access at all.

Three places must agree, or the agent will not answer: the provider configured on the install, the provider named in the policy, and whatever provider the agent image itself is told to use through its environment.

An agent that reads LENS_MANAGED_INFERENCE_PROVIDER or LLM_PROVIDER gets the third for free — it is told which backend it was given rather than having to be configured to match. See What every backend tells the agent.

What you get from proxying#

Because every call passes through the platform:

Spending is metered per call and enforced against spending limits by org, user, agent, or sandbox.
PII can be masked before the prompt leaves your network, and unmasked in the response. See Privacy and PII controls.
Every call is audited with the actor, project, and cost. See Audit trail.

None of it applies to a call that skipped the endpoint. See Metering follows routing.

Additional providers are configured to customer requirements. Talk to us.

Models — how agents select models at runtime
Spending limits — caps by org, user, agent, and sandbox
Policies — enabling managed inference for a set of sandboxes

Inference providers#

Supported providers#

AWS Bedrock#

On a cluster with an IAM role#

With a Bedrock API key#

Microsoft Foundry#

OpenAI#

Pointing at a compatible endpoint#

Realtime voice sessions#

What the sandbox receives#

OpenRouter#

Both wire formats, one key#

Choosing a model is the agent's job#

What the sandbox receives#

Billed on what OpenRouter charged#

What every backend tells the agent#

Metering follows routing#

Values#

Turning it on for agents#

What you get from proxying#

Related#