Configure AI Gateway v2

AI Gateway v2 is a single gateway process configured entirely through platform chart values under the ai-gateway-v2 key. There are no custom resources to apply. In prose, value paths on this page are relative to that key, except those that start with global. or another chart's key, such as enterprise-manager.. See AI Gateway v2 for providers, routing, screening, and budgets.

Prerequisites

A corporate identity provider that issues the tokens your clients send. The gateway trusts the issuers you list under auth.issuers. See Configure identity.
The Enterprise Manager with the directory enabled. Budgets, pricing, recorded spend, virtual API keys, and model access live there.
cert-manager, a ClusterIssuer, and a Secret holding its CA bundle. The Enterprise Manager's budget receiver and the optional screening adapters are served with certificates from that issuer, and the gateway verifies them with the bundle.
A Gateway API Gateway served by Envoy Gateway. AI Gateway v2 attaches to it with an HTTPRoute and has no ingress of its own.
Redis or Valkey, only if you intend to enable the PII scan result cache. See PCI/PII controls.

Enable it

AI Gateway v2 and the Envoy-based AI Gateway are alternative implementations of the same model gateway. A platform runs one of them. Turn AI Gateway v2 on and the AI Gateway off in the same change that adds the budget and directory values from Bring it up in this order:

values.yaml
global:
  stacklok:
    aiGatewayV2:
      enabled: true
    aiGateway:
      enabled: false

The chart refuses to render with both set to true, and the error names both keys. To run both during a migration, see Migrate from the AI Gateway.

The chart also refuses to render until AI Gateway v2 has a budget receiver configured, as described in Bring it up in this order. To run a gateway that records no spend, set budgets.acknowledgeUnmetered: true instead.

Trust an identity provider

auth.issuers lists the identity providers whose JWTs the gateway accepts. Each entry takes these fields:

Field	Purpose
`issuer`	The exact `iss` value
`audience`	The `aud` value the token must carry
`bindingClaim`	The claim that identifies the caller, such as `uid` for Okta or `oid` for Microsoft Entra ID. Never a mutable claim such as `email`
`tokenUse`	Optional. The `token_use` value the token must carry
`requiredScope`	Optional. The scope the token must carry, written exactly as it appears in the token's scope claim
`jwksUri`	Where the issuer publishes its signing keys

values.yaml
ai-gateway-v2:
  auth:
    issuers:
      - issuer: 'https://login.microsoftonline.com/<TENANT_ID>/v2.0'
        audience: '<APPLICATION_CLIENT_ID>'
        bindingClaim: oid
        requiredScope: LLM.Invoke
        jwksUri: 'https://login.microsoftonline.com/<TENANT_ID>/discovery/v2.0/keys'

Set requiredScope whenever your identity provider gives ID tokens and access tokens the same audience, so an ID token can't serve as an API credential. A token without the scope is refused with 403. For Entra ID, use the short name the token's scp claim carries, such as LLM.Invoke. Tokens issued through the client credentials flow carry no scope, so requiredScope refuses them.

Credentials that aren't JWTs are treated as virtual API keys and checked against the directory, so virtual keys need no issuer entry.

Bring it up in this order

Complete the following sequence before sending production traffic:

Create budgets that cover every caller. A caller with no applicable budget is refused. Set an organization default for resolved directory users, or create budgets for individual users and groups. See Budgets and pricing.
Connect the gateway to the Enterprise Manager. Serve the Enterprise Manager's webhook listener, point the gateway's budget receiver and directory connection at it, and upgrade the release:
values.yaml
```
global:
  webhooks:
    issuerRef:
      name: <WEBHOOK_CLUSTER_ISSUER>
      kind: ClusterIssuer
    caBundleSecret: <WEBHOOK_CA_BUNDLE_SECRET>

enterprise-manager:
  webhookTLS:
    enabled: true
  webhookAuth:
    audience: budgets-webhook
  grpc:
    callerAuth:
      clusterOidcIssuer: '<CLUSTER_OIDC_ISSUER>'

ai-gateway-v2:
  budgets:
    endpoint: 'https://<ENTERPRISE_MANAGER_SERVICE>.<NAMESPACE>.svc:443'
    tokenProjection:
      enabled: true
  directory:
    endpoint: '<DIRECTORY_GRPC_ADDRESS>'
    groupsResolutionIssuer: '<IDP_ISSUER_URL>'
    tokenProjection:
      enabled: true
```
budgets.endpoint must be an https:// URL. Its token audience, budgets.tokenProjection.audience, defaults to budgets-webhook and must equal enterprise-manager.webhookAuth.audience. directory.endpoint is the host:port of the directory's gRPC listener, without a scheme.

Set directory.groupsResolutionIssuer to the issuer the directory knows your users under, usually your corporate identity provider's issuer. Requests that use a virtual API key resolve their owner's groups through it. Without it, only user budgets apply to those requests.

Set enterprise-manager.grpc.callerAuth.clusterOidcIssuer to your cluster's ServiceAccount token issuer. The Enterprise Manager verifies the gateway's tokens against it, and while it's empty, refuses the gateway's directory calls and connectivity polls. This command prints the issuer:
```
kubectl get --raw /.well-known/openid-configuration | jq -r .issuer
```
On managed clusters such as Amazon EKS, the issuer is a public URL, so allow egress to it from the Enterprise Manager pods.

The platform chart adds the gateway's ServiceAccount to enterprise-manager.grpc.callerAuth.saSubjectAllowlist by default. If you set that list yourself, include system:serviceaccount:<NAMESPACE>:<RELEASE_NAME>-ai-gateway-v2.
Add providers and models. See Connect model providers.
Optionally turn on model access, so callers can use only the models their directory grants cover. See Route models and control access.
Verify that the gateway pods are ready:
```
kubectl get pods -n <NAMESPACE> \
  -l app.kubernetes.io/name=ai-gateway-v2,app.kubernetes.io/component=gateway
```
Then confirm that budget enforcement is wired, as described in Verify the gateway reaches the budgets service.

Publish the endpoint

AI Gateway v2 serves inference on a ClusterIP Service, port 8080. Publish it by attaching an HTTPRoute to your Envoy Gateway Gateway:

values.yaml
ai-gateway-v2:
  httpRoute:
    enabled: true
    parentRefs:
      - name: <GATEWAY_NAME>
        namespace: <GATEWAY_NAMESPACE>
        sectionName: <LISTENER_NAME>
    hostname: '<AI_GATEWAY_HOST>'

parentRefs is required when the route is enabled. Set hostname: without it, the route matches every hostname the listener accepts. The route's request timeout follows lifecycle.maxStreamDuration, 15 minutes by default, so a supported long stream is never cut short at the route.

Check that the route has Accepted and ResolvedRefs set to True:

kubectl get httproute -n <NAMESPACE> -o yaml

If you place another proxy in front of the Gateway, preserve the request paths and authentication headers, pass streamed responses without buffering, and allow your longest model requests.

To show this endpoint to users in the console and in the generated thv llm setup command, set toolhive-cloud-ui.gatewayProxyUrl to https://<AI_GATEWAY_HOST>.

Verify authenticated use

Send a request from a client machine using a model you configured and a token accepted by one of your issuers. The caller needs a directory identity, an applicable budget, and a grant for the model if model access is on:

curl --fail-with-body --silent --show-error \
  https://<AI_GATEWAY_HOST>/v1/chat/completions \
  -H 'Authorization: Bearer <ACCESS_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{"model":"<MODEL_NAME>","messages":[{"role":"user","content":"Reply with pong"}]}'

This example uses an OpenAI-compatible model. For an Anthropic model, send the same check to /anthropic/v1/messages in the Anthropic Messages format. A successful response verifies DNS, TLS, authentication, budget admission, and the provider connection. For a private CA, use --cacert <CA_FILE> and configure your clients to trust that CA. Repeat with "stream": true and curl --no-buffer to confirm chunks arrive before the response completes.

Client endpoints

AI Gateway v2 passes each request to the provider in the API format the client sent, so each client uses the base path for its own API:

Client	Base URL	Credential header
OpenAI-compatible clients	`https://<AI_GATEWAY_HOST>/v1`	`Authorization: Bearer`
Claude Code and Anthropic clients	`https://<AI_GATEWAY_HOST>/anthropic`	`x-api-key` or `Authorization: Bearer`
Codex (Responses API)	`https://<AI_GATEWAY_HOST>/v1`, which serves `/v1/responses`	`Authorization: Bearer`
Gemini CLI and Gemini clients	`https://<AI_GATEWAY_HOST>`, which serves `/v1beta`	`x-goog-api-key`

Each credential header accepts either a JWT or a virtual API key. The thv llm setup command that the console builds configures each client with the right path. See Roll out gateway clients.

Restrict network access

networkPolicy.enabled renders a NetworkPolicy for the gateway pods. It is off by default. Enable it when your cluster applies a default-deny policy:

values.yaml
ai-gateway-v2:
  networkPolicy:
    enabled: true
    ingress:
      gatewayNamespaces:
        - envoy-gateway-system
      monitoringNamespaces:
        - monitoring
    allowExternalEgress: false

Ingress admits only the Envoy pods in ingress.gatewayNamespaces, which must not be empty. The metrics port is admitted from ingress.monitoringNamespaces when Prometheus metrics are on.
Egress with allowExternalEgress: false admits DNS, outbound 443 for providers, the Enterprise Manager, the enabled screening adapters, and the OTLP ports in telemetry.otlp.endpoint and telemetry.otlp.tracesEndpoint. If the Enterprise Manager runs in a different namespace, set networkPolicy.governanceEgress.namespaces. Add anything else the gateway reaches under networkPolicy.egress.

A missing egress rule fails closed: governance calls can't complete, so every request is refused with 503 or 504.

Disconnect the gateway

An administrator can stop all model traffic without a redeploy by setting the AI Gateway's connectivity state through the Enterprise Manager API:

curl -X PUT \
  https://<ENTERPRISE_MANAGER_URL>/api/v1/platform-settings/gateways/ai-gateway/connectivity \
  -H "Authorization: Bearer <ADMIN_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"state": "disconnected"}'

While disconnected, the gateway refuses every inference request with 503 and the error code administratively_disconnected. Send {"state": "connected"} to restore traffic. Pods stay ready, so a disconnect never restarts them.

The console's Emergency Disconnect control, on the System tab of Platform Settings, lists the AI Gateway only while global.stacklok.aiGateway.enabled is true, such as during a migration window. Its Disconnect AI Gateway button stops both gateways.

Disconnection applies when global.stacklok.enterpriseManager.url is set. Each replica polls the state every 5 seconds by default. Until its first poll succeeds, a replica refuses traffic, and a later failed poll keeps the last known state. The platform chart allows the gateway's ServiceAccount to poll. If you override enterprise-manager.grpc.callerAuth.httpSaSubjectAllowlist, or rename the gateway with fullnameOverride or serviceAccount.name, include its actual subject or the gateway refuses all traffic.

Content screening posture

Screening failures deny requests by default. PII scanning blocks a request when Presidio is unreachable or slow, and prompt injection screening blocks a request when its adapter is unreachable, slow, or returns an error. To admit traffic during a rollout or incident, set scanning.ner.failureAction: fail-open or guardrails.failurePolicy: Ignore. Requests admitted this way reach the provider unscreened. See PCI/PII controls and Screen prompts for injection.

How configuration changes apply

The gateway reads its configuration at startup, so every change applies through a rollout. The chart renders the configuration into a ConfigMap, and a changed value changes the pod template. A credential Secret edited in place rolls the pods on the next helm upgrade, because the chart hashes Secret contents. With Argo CD or another renderer without cluster access, run kubectl rollout restart after editing a Secret.

Existing pods keep serving until new pods are ready, so a change that leaves the new pods unready stalls the rollout and leaves the gateway up. Check kubectl rollout status after every upgrade.

Next steps

AI Gateway v2 for providers, routing, screening, and budgets.
Roll out gateway clients to distribute deployment-specific setup instructions.

Prerequisites​

Enable it​

Trust an identity provider​

Bring it up in this order​

Publish the endpoint​

Verify authenticated use​

Client endpoints​

Restrict network access​

Disconnect the gateway​

Content screening posture​

How configuration changes apply​

Next steps​

Related information​