# Agent Router (formerly Envoy AI Gateway) > The open source AI gateway and MCP gateway built on Envoy and Envoy Gateway. An Agentic AI Foundation project. Agent Router controls; Envoy carries. Agent Router gives application teams one OpenAI-compatible API for every model and MCP tool, and platform teams one place to enforce credentials, routing, quotas, failover and usage attribution. Repository: https://github.com/theagentrouter/agent-router · License: Apache-2.0 --- # Home Source: https://theagentrouter.ai/docs/ Welcome to the **Agent Router** documentation! This open-source project, built on **Envoy Proxy**, aims to simplify how application clients interact with **Generative AI (GenAI)** services. It provides a secure, scalable, and efficient way to manage LLM/AI traffic, with backend rate limiting and policy control. **Formerly Envoy AI Gateway:** Agent Router is the new name for **Envoy AI Gateway**, now an **Agentic AI Foundation** project. Same code, same maintainers, new home — **no CRD, API, or CLI names have changed** (`aigw`, `AIGatewayRoute`, and the `aigateway.envoyproxy.io` API group are all unchanged), and there is nothing to migrate. ## Is Agent Router for you? Agent Router is worth exploring if: - Multiple applications or teams share access to models or tools. - Application teams should not have to integrate with every provider separately. - Provider credentials should not live inside application workloads. - You need consistent permissions, quotas, usage attribution, or failover. - You want to use hosted and self-managed models within a single platform boundary. - You are exposing MCP servers and need to control what each identity can discover and invoke. Agent Router is not an agent framework and does not tell an agent how to reason. Its value is clearest when AI traffic crosses provider, team, security, cost, or operational boundaries — when a prototype becomes something the business depends on. The fastest way to find out is one command on your laptop: ```shell OPENAI_API_KEY=sk-your-key aigw run ``` That starts an OpenAI-compatible router at `localhost:1975` — see [Run locally in 60 seconds](getting-started/index.md#run-locally-in-60-seconds). ## **Project Overview** The project — created as **Envoy AI Gateway** — addresses the complexity of connecting applications to GenAI services by leveraging Envoy's flexibility and Kubernetes-native features. It has evolved through contributions from the Envoy community, fostering a collaborative approach to solving real-world challenges. ### **Key Objectives** - Provide a unified layer for routing and managing LLM/AI traffic. - Support automatic failover mechanisms to ensure service reliability. - Ensure end-to-end security, including upstream authorization for LLM/AI traffic. - Implement a policy framework to support usage limiting use cases. - Foster an open-source community to address GenAI-specific routing and quality of service needs. ## **Project Goals** The Agent Router project is designed to address the critical challenges of AI/LLM integration in enterprise environments through the following core goals: - **Resilient Connectivity Across Providers and Self-Hosted Models**: Create robust, fault-tolerant connections that integrate with LLM providers (such as OpenAI, Anthropic, AWS Bedrock, etc.) and self-hosted models, ensuring high availability through intelligent routing and automatic failover. - **Comprehensive Observability for Performance and Cost Management**: Provide visibility into traffic performance, usage patterns, and cost analytics, enabling organizations to optimize their GenAI usage and monitor service quality. - **Enterprise-Grade Security Features**: Implement security controls including fine-grained access control, authorization policies, rate limiting, to access services, as well as ensure secure egress connection to external providers from the gateway via [Upstream Authentication](capabilities/security/upstream-auth.mdx). - **Extensible Architecture**: Leverage Envoy's proven extensibility framework to enable rapid development of custom features, allowing organizations to adapt the gateway to their specific AI infrastructure needs. - **Reliable Foundation with Envoy Proxy**: Build upon Envoy's battle-tested proxy technology to provide a stable, performant, and production-ready foundation that enterprises can rely on for their traffic handling needs. These goals guide the development of features and capabilities that make AI/LLM integration secure, scalable, and operationally excellent for enterprise environments. Documentation for installation, setup, and contribution guidelines is included to help new users and contributors get started easily. ## **Community Collaboration** [Weekly community meetings][meeting-notes] are held every Monday to discuss updates, address issues, and review contributions. ## **Architecture Overview** Agent Router is the control plane; Envoy is the data plane that carries the traffic. See the [architecture documentation](concepts/architecture/) for how the pieces fit together. ## **Get Involved** We welcome community contributions! Here's how you can participate: - Attend the [weekly community meetings][meeting-notes] to stay updated and share ideas. - Submit feature requests and pull requests via the GitHub repository. - Join the conversation on the [community Discord][discord]. Refer to [this contributing guide][contributing.md] for detailed instructions on setting up your environment and contributing. --- The **Agent Router** addresses the growing demand for secure, scalable, and efficient AI/LLM traffic management. Your contributions and feedback are key to its success and to advancing the future of AI service integration. [meeting-notes]: https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w [discord]: https://discord.gg/xuxtPq43gZ [contributing.md]: https://github.com/theagentrouter/agent-router/blob/main/CONTRIBUTING.md --- # Getting Started Source: https://theagentrouter.ai/docs/getting-started/ Agent Router runs two ways. The fastest is one command on your laptop; the same configuration then ships to Kubernetes for production. ## Run locally in 60 seconds The standalone CLI starts an OpenAI-compatible router on your machine — no Kubernetes, no Docker required (Linux and macOS): ```shell OPENAI_API_KEY=sk-your-key aigw run ``` Then point any OpenAI-compatible client or SDK at `http://localhost:1975/v1`: ```shell curl http://localhost:1975/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "gpt-5", "messages": [{"role": "user", "content": "Say this is a test!"}]}' ``` `aigw` auto-configures from the same environment variables as the OpenAI SDK, and can also front self-hosted models (Ollama, vLLM) and MCP servers. See [Installation](../cli/installation.md) for how to get the `aigw` binary, and [aigw run](../cli/run.md) for provider auto-configuration, MCP gateway mode, and custom configuration files. ## Deploy on Kubernetes For production, Agent Router runs as a control plane on Envoy Gateway in your Kubernetes cluster. It uses the same configuration API you test locally with `aigw run`, so what works on your laptop deploys unchanged. This guide walks through the Kubernetes path: 1. [Prerequisites](./prerequisites.md) - Setting up your Kubernetes cluster - Installing required tools - Setting up Envoy Gateway 2. [Installation](./installation.md) - Installing Agent Router - Configuring the gateway - Verifying the installation 3. [Basic Usage](./basic-usage.md) - Deploying a basic configuration - Making your first request - Understanding the response format 4. [Connect Providers](./connect-providers) - Setting up OpenAI integration - Configuring AWS Bedrock - Managing credentials securely For local Docker or non-Kubernetes deployments, start with the [Agent Router CLI](../cli/) instead. ## Need Help? If you run into any issues: - Join our [Community Discord](https://discord.gg/xuxtPq43gZ) - File an issue on [GitHub](https://github.com/theagentrouter/agent-router/issues) --- # Prerequisites Source: https://theagentrouter.ai/docs/getting-started/prerequisites Before you begin using Agent Router, you'll need to ensure you have the following prerequisites in place: ## Required Tools Make sure you have the following tools installed: - `kubectl` - The Kubernetes command-line tool - `helm` - The package manager for Kubernetes - `curl` - For testing API endpoints (installed by default on most systems) **Verify Installation:** Run these commands to verify your tools are properly installed: Verify kubectl installation: ```shell kubectl version --client ``` Verify helm installation: ```shell helm version ``` Verify curl installation: ```shell curl --version ``` ## Kubernetes Cluster **Version Requirements:** Agent Router requires Kubernetes version 1.32 or higher. We recommend using a recent stable version of Kubernetes for the best experience. You need a running Kubernetes cluster with your kubeconfig properly configured. You have several options: **Existing Cluster** If you already have a Kubernetes cluster, ensure your kubeconfig is properly configured to access it. Verify your cluster meets the version requirements by running: ```shell kubectl version --output=json ``` The server version in the output should show version 1.32 or higher: ```json { "serverVersion": { "major": "1", "minor": "32+", ... } } ``` **Caution:** If your cluster is running a version lower than 1.32, our recommendation is to upgrade your cluster to meet the minimum requirements. The recommended Kubernetes versions are those that are actively tested and supported by the Agent Router team. **Docker Desktop** If you're using Docker Desktop, you can enable its built-in Kubernetes cluster: 1. Open Docker Desktop 2. Click on the gear icon (Settings) 3. Select "Kubernetes" from the left sidebar 4. Check "Enable Kubernetes" 5. Click "Apply & Restart" Wait for Docker Desktop to restart and for the Kubernetes cluster to be ready. You can verify the setup by running: ```shell kubectl config use-context docker-desktop kubectl cluster-info ``` The output should show that the Kubernetes control plane is running. **Tip:** Docker Desktop's Kubernetes is a great choice for local development as it: - Comes pre-installed with Docker Desktop - Runs locally on your machine - Integrates well with Docker Desktop's UI - Requires minimal setup **Local Kind Cluster** If you don't have a Kubernetes cluster, you can quickly create a local one using [kind](https://kind.sigs.k8s.io/). First, install kind if you haven't already (on macOS with Homebrew): ```shell brew install kind ``` Then create a cluster: ```shell kind create cluster ``` ## Envoy Gateway **Important:** Ensure you're using a clean Envoy Gateway deployment. If you have an existing Envoy Gateway installation with custom configurations, it may conflict with AI Gateway's requirements. We recommend: - Using a fresh Kubernetes cluster, or - Uninstalling any existing Envoy Gateway deployments before proceeding: ```shell helm uninstall eg -n envoy-gateway-system kubectl delete namespace envoy-gateway-system ``` **Version Requirements:** Agent Router requires Envoy Gateway version 1.9.2 or higher. For the best experience while trying out AI Gateway, we recommend using the latest version as shown in the commands below. Agent Router is built on top of Envoy Gateway. Install it using Helm and wait for the deployment to be ready. ```shell helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \ --version v1.9.2 \ --namespace envoy-gateway-system \ --create-namespace \ -f https://raw.githubusercontent.com/theagentrouter/agent-router/main/manifests/envoy-gateway-values.yaml kubectl wait --timeout=2m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available ``` > If you are experiencing network issues with `docker.io` , you can follow the [Install from Source Code](https://github.com/envoyproxy/gateway/tree/main/charts/gateway-helm#install-from-source-code) instructions. Note that we have included the AI Gateway-specific helm values file via the `-f` flag. This file contains the necessary configuration for AI Gateway integration. ### Additional Features (Rate Limiting, InferencePool, etc.) Depending on the additional features you want (like rate limiting or InferencePool), you need to pass additional addon values files to modify the Envoy Gateway installation. Currently, supported addons are: - [**Rate Limiting**](../capabilities/traffic/usage-based-ratelimiting.md): https://github.com/theagentrouter/agent-router/blob/main/examples/token_ratelimit/envoy-gateway-values-addon.yaml - [**InferencePool**](../capabilities/inference/index.md): https://github.com/theagentrouter/agent-router/blob/main/examples/inference-pool/envoy-gateway-values-addon.yaml For example, to install with all addons enabled, run: ```shell helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \ --version v1.9.2 \ --namespace envoy-gateway-system \ --create-namespace \ -f https://raw.githubusercontent.com/theagentrouter/agent-router/main/manifests/envoy-gateway-values.yaml \ -f https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/token_ratelimit/envoy-gateway-values-addon.yaml \ -f https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/inference-pool/envoy-gateway-values-addon.yaml ``` --- # Installation Source: https://theagentrouter.ai/docs/getting-started/installation This guide will walk you through installing Agent Router and its required components. ## Installing Agent Router The easiest way to install Agent Router on Kubernetes is using the Helm charts. You need to install the CRDs first, followed by the AI Gateway controller. If you want to run Agent Router as a standalone proxy outside Kubernetes, use [`aigw run`](../cli/run.md). The CLI can run directly on Linux and macOS, or from the Docker Compose examples in [`cmd/aigw`](https://github.com/theagentrouter/agent-router/tree/main/cmd/aigw). Those examples include an Ollama-backed stack and an OpenTelemetry stack for local traces and metrics. ### Step 1: Install AI Gateway CRDs First, install the CRD Helm chart (`ai-gateway-crds-helm`) which manages all Custom Resource Definitions: ```shell helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \ --version v1.2.0 \ --namespace envoy-ai-gateway-system \ --create-namespace ``` ### Step 2: Install AI Gateway Resources After the CRDs are installed, install the AI Gateway Helm chart: ```shell helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \ --version v1.2.0 \ --namespace envoy-ai-gateway-system \ --create-namespace kubectl wait --timeout=2m -n envoy-ai-gateway-system deployment/ai-gateway-controller --for=condition=Available ``` **Tip:** Note that you are browsing the documentation for the main branch version of Agent Router, which is not a stable release. We highly recommend you replace `v0.0.0-latest` with `v0.0.0-${commit hash of https://github.com/theagentrouter/agent-router}` to pin to a specific version. Otherwise, the controller will be installed with the latest version at the time of installation, which can be unstable over time due to ongoing development (the latest container tags are overwritten). > If you are experiencing network issues with `docker.io`, you can install the helm charts from the code repo [ai-gateway-crds-helm](https://github.com/theagentrouter/agent-router/tree/main/manifests/charts/ai-gateway-crds-helm) and [ai-gateway-helm](https://github.com/theagentrouter/agent-router/tree/main/manifests/charts/ai-gateway-helm) instead. **Verify Installation:** Check the status of the pods. All pods should be in the `Running` state with `Ready` status. Check AI Gateway pods: ```shell kubectl get pods -n envoy-ai-gateway-system ``` **Upgrading from Previous Versions:** If you installed AI Gateway with only `ai-gateway-helm` previously, first install the CRD chart with `--take-ownership` to transfer CRD ownership, then upgrade the main chart: ```shell helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm --version v1.2.0 --namespace envoy-ai-gateway-system --take-ownership helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm --version v1.2.0 --namespace envoy-ai-gateway-system ``` ## Next Steps After completing the installation: - Continue to [Basic Usage](./basic-usage.md) to learn how to make your first request - Or jump to [Connect Providers](./connect-providers) to set up OpenAI and AWS Bedrock integration --- # Basic Usage Source: https://theagentrouter.ai/docs/getting-started/basic-usage This guide will help you set up a basic AI Gateway configuration and make your first request. For Windows users, note that you are able to use Windows Subsystem for Linux (WSL) to run the commands below if they do not work on the Windows command prompt. ## Setting Up Your Environment ### Deploy Basic Configuration Let's start by deploying a basic AI Gateway setup that includes a test backend: ```shell kubectl apply -f https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/basic.yaml ``` Wait for the Gateway pod to be ready: ```shell kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system \ --for=condition=Ready ``` **Tip:** In the `basic.yaml` file, you will find "ClientTrafficPolicy" resource that configures the buffer limit on the client to the gateway connection. By default, Envoy Gateway configures 32KB as the buffer limit, which is not enough for most AI model responses, hence it's configured to 50MB in the example. Envoy Gateway provides more advanced policies, not limited to the buffer limit. Please refer to the [Client Traffic Policies](https://gateway.envoyproxy.io/latest/tasks/traffic/client-traffic-policy/) documentation for more details. **HTTP/2 window size settings:** If you are sending requests over HTTP/2, Envoy imposes additional flow-control limits via `initialStreamWindowSize` and `initialConnectionWindowSize`. These act as a secondary cap on request body size, independent of `bufferLimit`, and can cause `413 request_payload_too_large` errors even after the buffer limit is raised. To avoid this, set both values to the Envoy defaults (`16Mi` and `24Mi`) in the `ClientTrafficPolicy`: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: ClientTrafficPolicy metadata: name: client-buffer-limit namespace: default spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: envoy-ai-gateway-basic connection: bufferLimit: 50Mi http2: initialStreamWindowSize: 16Mi initialConnectionWindowSize: 24Mi ``` See the Envoy API reference for [`initial_stream_window_size`](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field-config-core-v3-http2protocoloptions-initial-stream-window-size) and [`initial_connection_window_size`](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field-config-core-v3-http2protocoloptions-initial-connection-window-size) for more details. ## Configure `$GATEWAY_URL` First, check if your Gateway has an external IP address assigned: ```shell kubectl get svc -n envoy-gateway-system \ --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic ``` You'll see output similar to this: ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) eg-envoy-ai-gateway LoadBalancer 10.96.61.234 80:31234/TCP ``` Choose one of these options based on the EXTERNAL-IP status: **Using External IP** If the EXTERNAL-IP shows an actual IP address (not ``), you can access the gateway directly: First, save the external IP and set the gateway URL: ```shell export GATEWAY_URL=$(kubectl get gateway/envoy-ai-gateway-basic -o jsonpath='{.status.addresses[0].value}') ``` Verify the URL is available: ```shell echo $GATEWAY_URL ``` **Using Port Forwarding** If the EXTERNAL-IP shows `` or your cluster doesn't support LoadBalancer services, use port forwarding. First, set the gateway URL: ```shell export GATEWAY_URL="http://localhost:8080" ``` Then set up port forwarding (this will block the terminal): ```shell export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system \ --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -o jsonpath='{.items[0].metadata.name}') kubectl port-forward -n envoy-gateway-system svc/$ENVOY_SERVICE 8080:80 ``` ## Testing the Gateway ### Making Test Requests Open a new terminal and send test requests to the AI Gateway using the `GATEWAY_URL` we set up: #### Test Chat Completions ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "some-cool-self-hosted-model", "messages": [ { "role": "system", "content": "Hi." } ] }' \ $GATEWAY_URL/v1/chat/completions ``` #### Test Completions (Legacy) ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "some-cool-self-hosted-model", "prompt": "def fib(n):\n if n <= 1:\n return n\n else:\n return fib(n-1) + fib(n-2)", "max_tokens": 25, "temperature": 0.4, "top_p": 0.9 }' \ $GATEWAY_URL/v1/completions ``` #### Test Embeddings ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "some-cool-self-hosted-model", "input": "Hello, world!" }' \ $GATEWAY_URL/v1/embeddings ``` #### Test Rerank ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "some-cool-self-hosted-model", "query": "What is the capital of France?", "documents": [ "Paris is the capital of France.", "Berlin is the capital of Germany." ] }' \ $GATEWAY_URL/cohere/v2/rerank ``` **Tip:** If you're opening a new terminal, you'll need to set the `GATEWAY_URL` variable again. ### Expected Responses #### Chat Completions Response You should receive a response like: ```json { "choices": [ { "message": { "content": "I'll be back." } } ] } ``` #### Completions Response For legacy completions requests, you should receive a response like: ```json { "choices": [ { "text": " + fib(n-3) + fib(n-4)\n\ndef fib(n):\n if n <= 1:\n return", "index": 0, "finish_reason": "length" } ] } ``` #### Embeddings Response For embeddings requests, you should receive a response like: ```json { "object": "list", "data": [ { "object": "embedding", "embedding": [0.1, 0.2, 0.3, 0.4, 0.5], "index": 0 } ], "model": "some-cool-self-hosted-model", "usage": { "prompt_tokens": 3, "total_tokens": 3 } } ``` #### Rerank Response You should receive a response like: ```json { "results": [ { "index": 0, "relevance_score": 0.98 }, { "index": 1, "relevance_score": 0.12 } ] } ``` **Note:** These responses come from a mock backend. The model `some-cool-self-hosted-model` is configured to return test responses. For real AI model responses, see the [Connect Providers](./connect-providers) guide. ### Understanding the Response Formats The basic setup includes a mock backend that demonstrates the API structure but doesn't provide real AI responses. The response formats follow the standard OpenAI API formats: **Chat Completions:** - A `choices` array containing responses - Each message having a `role` and `content` **Embeddings:** - A `data` array containing embedding objects - Each embedding having a vector array and index - Usage information showing token consumption **Rerank:** - A `results` array containing ranked results for the provided documents - Each result includes the original document `index` and a `relevance_score` - Higher `relevance_score` indicates higher relevance to the query ## Next Steps Now that you've tested the basic setup, you can: - Configure [real AI model backends](./connect-providers) like OpenAI or AWS Bedrock - Explore the [API Reference](../api/) for more details about available endpoints --- # Connect Providers Source: https://theagentrouter.ai/docs/getting-started/connect-providers/ After setting up the basic AI Gateway with the mock backend, you can configure it to work with real AI model providers. This section will guide you through connecting different AI providers to your gateway. ## Example Providers In this getting started guide you'll find quickstart setups to connect to the following providers: - [Tetrate Agent Router Service (TARS)](./tars.md) - Connect to Tetrate Agent Router Service's models - [Bitdeer AI](./bitdeer.md) - Connect to Bitdeer AI's OpenAI-compatible inference models - [OpenAI](./openai.md) - Connect to OpenAI's GPT models - [Anthropic](./anthropic.md) - Connect to Anthropic's Claude models - [AWS Bedrock](./aws-bedrock.md) - Access AWS Bedrock's suite of foundation models - [Azure OpenAI](./azure-openai.md) - Access Azure OpenAI's suite of foundation models - [GCP VertexAI](./gcp-vertexai.md) - Access GCP Gemini and Anthropic models on VertexAI **Tip:** To learn how to connect to providers see [Connecting to AI Providers](https://theagentrouter.ai/docs/capabilities/llm-integrations/connect-providers) and you can view all [Supported Providers](https://theagentrouter.ai/docs/capabilities/llm-integrations/supported-providers). ## Before You Begin Before configuring any provider, complete the [Basic Usage](../basic-usage.md) guide. ## Security Best Practices When configuring AI providers, keep these security considerations in mind: - Store credentials securely using Kubernetes secrets - Never commit API keys or credentials to version control - Regularly rotate your credentials - Use the principle of least privilege when setting up access - Monitor usage and set up appropriate rate limits ## Next Steps Choose your provider to get started: - [Connect Tetrate Agent Router Service (TARS)](./tars.md) - [Connect Bitdeer AI](./bitdeer.md) - [Connect OpenAI](./openai.md) - [Connect Anthropic](./anthropic.md) - [Connect AWS Bedrock](./aws-bedrock.md) - [Connect Azure OpenAI](./azure-openai.md) - [Connect GCP VertexAI](./gcp-vertexai.md) --- # Connect OpenAI Source: https://theagentrouter.ai/docs/getting-started/connect-providers/openai This guide will help you configure Agent Router to work with OpenAI's models. ## Prerequisites Before you begin, you'll need: - An OpenAI API key from [OpenAI's platform](https://platform.openai.com) - Basic setup completed from the [Basic Usage](../basic-usage.md) guide - Basic configuration removed as described in the [Advanced Configuration](./index.md) overview ## Configuration Steps **Ready to proceed?:** Ensure you have followed the steps in [Connect Providers](../connect-providers/) ### 1. Download configuration template ```shell curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/openai.yaml ``` ### 2. Configure OpenAI Credentials Edit the `openai.yaml` file to replace the OpenAI placeholder value: - Find the section containing `OPENAI_API_KEY` - Replace it with your actual OpenAI API key **Security Note:** Make sure to keep your API key secure and never commit it to version control. The key will be stored in a Kubernetes secret. ### 3. Apply Configuration Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, then the secret credential update will be picked up automatically in a few seconds. ```shell kubectl apply -f openai.yaml kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system \ --for=condition=Ready ``` ### 4. Test the Configuration You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. See the [Basic Usage](../basic-usage.md) page for instructions. #### Test Chat Completions ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o-mini", "messages": [ { "role": "user", "content": "Hi." } ] }' \ $GATEWAY_URL/v1/chat/completions ``` #### Test Completions (Legacy) OpenAI supports the legacy completions endpoint with specific models: ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "babbage-002", "prompt": "def fib(n):\n if n <= 1:\n return n\n else:\n return fib(n-1) + fib(n-2)", "max_tokens": 25, "temperature": 0.4, "top_p": 0.9 }' \ $GATEWAY_URL/v1/completions ``` ## Troubleshooting If you encounter issues: 1. Verify your API key is correct and active 2. Check pod status: ```shell kubectl get pods -n envoy-gateway-system ``` 3. View controller logs: ```shell kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller ``` 4. View External Processor Logs ```shell kubectl logs -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic -c ai-gateway-extproc ``` 5. Common errors: - 401: Invalid API key - 429: Rate limit exceeded - 503: OpenAI service unavailable ## Configuring More Models To use more models, add more [AIGatewayRouteRule]s to the `openai.yaml` file with the [model alias] in the `value` field. For example, let's add [o1] as a chat completion model, and [text-embedding-ada-002](https://platform.openai.com/docs/models/text-embedding-ada-002) as embedding models: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: envoy-ai-gateway-basic-openai namespace: default spec: parentRefs: - name: envoy-ai-gateway-basic kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: o1 - matches: - headers: - type: Exact name: x-ai-eg-model value: text-embedding-ada-002 backendRefs: - name: envoy-ai-gateway-basic-openai ``` ## Next Steps After configuring OpenAI: - [Connect AWS Bedrock](./aws-bedrock.md) to add another provider [AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule [model alias]: https://platform.openai.com/docs/models#current-model-aliases [o1]: https://platform.openai.com/docs/models#o1 --- # Connect AWS Bedrock Source: https://theagentrouter.ai/docs/getting-started/connect-providers/aws-bedrock This guide will help you configure Agent Router to work with AWS Bedrock's foundation models, including Llama, Anthropic Claude, and other models available on AWS Bedrock. ## Prerequisites - AWS credentials with access to Bedrock - Basic setup completed from the [Basic Usage](../basic-usage.md) guide - Basic configuration removed as described in the [Advanced Configuration](./index.md) overview - Enabled model access to "Llama 3.2 1B Instruct" in the `us-east-1` region (see [AWS Bedrock Model Access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html)) ## Authentication Methods Agent Router supports the AWS SDK default credential chain, which includes: 1. **EKS Pod Identity** - Recommended for production on EKS (v1.24+) 2. **IRSA (IAM Roles for Service Accounts)** - Recommended for production on EKS (all versions) 3. **Static Credentials** - For development, testing, or non-EKS environments ## Setup Instructions ### Option 1: EKS Pod Identity (Recommended for EKS 1.24+) EKS Pod Identity is the simplest way to grant AWS permissions to your pods without managing static credentials. **Step 1: Set up AWS IAM** Follow the official AWS documentation to configure EKS Pod Identity: - [EKS Pod Identity Agent installation](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html) - [Create IAM role and policy](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html) Your IAM policy needs these permissions: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream", "bedrock:ListFoundationModels", "aws-marketplace:ViewSubscriptions" ], "Resource": "*" } ] } ``` The Pod Identity association should reference: - **Namespace**: `envoy-gateway-system` - **ServiceAccount**: `ai-gateway-dataplane-aws` **Step 2: Apply AI Gateway configuration** ```shell kubectl apply -f https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/aws-pod-identity.yaml kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system --for=condition=Ready ``` **Step 3: Test** ```shell export GATEWAY_URL=$(kubectl get gateway envoy-ai-gateway-basic -n default -o jsonpath='{.status.addresses[0].value}') curl -H "Content-Type: application/json" -d '{ "model": "us.meta.llama3-2-1b-instruct-v1:0", "messages": [{"role": "user", "content": "Hello!"}] }' http://$GATEWAY_URL/v1/chat/completions ``` --- ### Option 2: IRSA (IAM Roles for Service Accounts) IRSA works on all EKS versions and uses OIDC federation for authentication. **Step 1: Set up AWS IAM** Follow the official AWS documentation to configure IRSA: - [Enable OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) - [Create IAM role with trust policy](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) Your IAM policy needs the same Bedrock permissions as Pod Identity above. The trust policy should allow the ServiceAccount `system:serviceaccount:envoy-gateway-system:ai-gateway-dataplane-aws`. **Step 2: Download and configure** ```shell curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/aws-irsa.yaml ``` Edit `aws-irsa.yaml` and replace `ACCOUNT_ID` with your AWS account ID in the ServiceAccount annotation: ```yaml eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ACCOUNT_ID:role/AIGatewayBedrockRole ``` **Step 3: Apply and test** ```shell kubectl apply -f aws-irsa.yaml kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system --for=condition=Ready # Test (use same curl command as Pod Identity above) export GATEWAY_URL=$(kubectl get gateway envoy-ai-gateway-basic -n default -o jsonpath='{.status.addresses[0].value}') curl -H "Content-Type: application/json" -d '{ "model": "us.meta.llama3-2-1b-instruct-v1:0", "messages": [{"role": "user", "content": "Hello!"}] }' http://$GATEWAY_URL/v1/chat/completions ``` --- ### Option 3: Static Credentials Use AWS access key ID and secret for authentication. Suitable for development, testing, or non-EKS environments. **Production Warning:** Static credentials are not recommended for production. Use EKS Pod Identity or IRSA instead. **Step 1: Download and configure** ```shell curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/aws.yaml ``` Edit `aws.yaml` and replace the credential placeholders: - `AWS_ACCESS_KEY_ID`: Your AWS access key ID - `AWS_SECRET_ACCESS_KEY`: Your AWS secret access key **Step 2: Apply and test** ```shell kubectl apply -f aws.yaml kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system --for=condition=Ready # Test (use same curl command as above) export GATEWAY_URL=$(kubectl get gateway envoy-ai-gateway-basic -n default -o jsonpath='{.status.addresses[0].value}') curl -H "Content-Type: application/json" -d '{ "model": "us.meta.llama3-2-1b-instruct-v1:0", "messages": [{"role": "user", "content": "Hello!"}] }' http://$GATEWAY_URL/v1/chat/completions ``` You can also access an Anthropic model with native Anthropic messages endpoint: ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", "messages": [ { "role": "user", "content": "What is capital of France?" } ], "max_tokens": 100 }' \ $GATEWAY_URL/anthropic/v1/messages ``` Expected output: ```json { "id": "msg_01XFDUDYJgAACzvnptvVoYEL", "type": "message", "role": "assistant", "content": [ { "type": "text", "text": "The capital of France is Paris." } ], "model": "claude-3-5-sonnet-20241022", "stop_reason": "end_turn", "usage": { "input_tokens": 13, "output_tokens": 8 } } ``` ## Troubleshooting If you encounter issues: 1. **Verify authentication is configured correctly** - For **EKS Pod Identity**: Check Pod Identity association exists ```shell aws eks list-pod-identity-associations --cluster-name YOUR_CLUSTER_NAME ``` - For **IRSA**: Check ServiceAccount annotation ```shell kubectl get sa ai-gateway-dataplane-aws -n envoy-gateway-system -o yaml ``` - For **Static Credentials**: Verify secret exists ```shell kubectl get secret -n default ``` 2. **Check pod logs** ```shell POD_NAME=$(kubectl get pod -n envoy-gateway-system \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -o jsonpath='{.items[0].metadata.name}') kubectl logs -n envoy-gateway-system $POD_NAME -c extproc ``` 3. **Common errors** - **401/403**: Invalid credentials or insufficient IAM permissions - **404**: Model not found or not available in the specified region - **AssumeRole errors**: Check IAM role trust policy For more details on AWS authentication, see: - [AWS SDK credential chain](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html) - [EKS Pod Identity documentation](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) - [IRSA documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) ## Configuring More Models To use additional models, add more [AIGatewayRouteRule]s to your configuration with the [model ID] in the `value` field. For example, to use [Claude 3 Sonnet]: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: envoy-ai-gateway-basic-aws namespace: default spec: parentRefs: - name: envoy-ai-gateway-basic kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: anthropic.claude-3-sonnet-20240229-v1:0 backendRefs: - name: envoy-ai-gateway-basic-aws ``` ## Using Anthropic Native API When using Anthropic models on AWS Bedrock, you have two options: 1. **OpenAI-compatible format** (`/v1/chat/completions`) - Works with most models but may not support all Anthropic-specific features. Note that `max_tokens` is required by the Anthropic API — if your clients do not include it, you must inject it via a request body mutation. 2. **Native Anthropic API** (`/anthropic/v1/messages`) - Provides full access to Anthropic-specific features (only for Anthropic models) ### Streaming with Native Anthropic API The native Anthropic API also supports streaming responses: ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", "messages": [ { "role": "user", "content": "Count from 1 to 5." } ], "max_tokens": 100, "stream": true }' \ $GATEWAY_URL/anthropic/v1/messages ``` ## Advanced Features with Anthropic Models Since the gateway supports the native Anthropic API, you have full access to Anthropic-specific features: ### Extended Thinking ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", "messages": [ { "role": "user", "content": "Solve this puzzle: A farmer needs to cross a river with a fox, chicken, and bag of grain. The boat can only hold the farmer and one item. How does the farmer get everything across safely?" } ], "max_tokens": 1000, "thinking": { "type": "enabled", "budget_tokens": 5000 } }' \ $GATEWAY_URL/anthropic/v1/messages ``` ### Prompt Caching ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", "system": [ { "type": "text", "text": "You are an AI assistant specialized in Python programming. You help users write clean, efficient Python code.", "cache_control": {"type": "ephemeral"} } ], "messages": [ { "role": "user", "content": "Write a function to calculate fibonacci numbers." } ], "max_tokens": 500 }' \ $GATEWAY_URL/anthropic/v1/messages ``` ### Tool Use (Function Calling) ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", "messages": [ { "role": "user", "content": "What is the weather in San Francisco?" } ], "max_tokens": 500, "tools": [ { "name": "get_weather", "description": "Get the current weather in a given location", "input_schema": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" } }, "required": ["location"] } } ] }' \ $GATEWAY_URL/anthropic/v1/messages ``` [AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule [model ID]: https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html [Claude 3 Sonnet]: https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table --- # Connect Azure OpenAI Source: https://theagentrouter.ai/docs/getting-started/connect-providers/azure-openai This guide will help you configure Agent Router to work with Azure OpenAI's foundation models. There are two ways to do the [Azure OpenAI authentication](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#authentication): Microsoft Entra ID and API Key. We will use Microsoft Entra ID to authenticate an application to use the Azure OpenAI service. You can obtain an access token using the OAuth 2.0 client credentials grant flow. This process involves registering the application in Microsoft Entra ID (formerly Azure Active Directory), configuring the appropriate permissions, and acquiring a token from the Microsoft identity platform. The access token is then used as proof of authorization in API requests to the Azure OpenAI endpoint. For detailed steps, refer to the official [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow#get-a-token). API Key authentication is not supported yet. ## Prerequisites Before you begin, you'll need: - Azure credentials with access to OpenAI service. - Basic setup completed from the [Basic Usage](../basic-usage.md) guide - Basic configuration removed as described in the [Advanced Configuration](./index.md) overview ## Azure Credential Setup 1. An Azure account with OpenAI service access enabled 2. Your Azure tenant ID, client ID, and client secret key 3. Enabled model access to "GPT-4o" ## Configuration Steps ### 1. Download configuration template ```shell curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/azure_openai.yaml ``` ### 2. Configure Azure Credentials Edit the `azure_openai.yaml` file to replace these placeholder values: - `AZURE_TENANT_ID`: Your Azure tenant ID - `AZURE_CLIENT_ID`: Your Azure client ID - `AZURE_CLIENT_SECRET`: Your Azure client secret **Security Note:** Keep your Azure credentials secure and never commit them to version control. The credentials will be stored in Kubernetes secrets. ### 3. Apply Configuration Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, then the secret credential update will be picked up automatically in a few seconds. ```shell kubectl apply -f azure_openai.yaml kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system \ --for=condition=Ready ``` ### 4. Test the Configuration You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. See the [Basic Usage](../basic-usage.md) page for instructions. ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [ { "role": "user", "content": "Hi." } ] }' \ $GATEWAY_URL/v1/chat/completions ``` ## Troubleshoot If you encounter issues: 1. Verify your Azure credentials are correct and active 2. Check pod status ```shell kubectl get pods -n envoy-gateway-system ``` 3. View controller logs: ```shell kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller ``` 4. Common errors: - 401/403: Invalid credentials or insufficient permissions - 404: Model not found or not available in region - 429: Rate limit exceeded ## Configuring More Models To use more models, add more [AIGatewayRouteRule]s to the `azure_openai.yaml` file with the [model ID] in the `value` field. For example, to use [GPT-4.5 Preview] ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: envoy-ai-gateway-basic-azure namespace: default spec: parentRefs: - name: envoy-ai-gateway-basic kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: gpt-4.5-preview backendRefs: - name: envoy-ai-gateway-basic-azure ``` [AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule [model ID]: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models [GPT-4.5 Preview]: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=global-standard%2Cstandard-chat-completions#gpt-45-preview --- # Connect GCP VertexAI Source: https://theagentrouter.ai/docs/getting-started/connect-providers/gcp-vertexai This guide will help you configure Agent Router to work with GCP VertexAI's Gemini and Anthropic models. ## Prerequisites Before you begin, you'll need: - GCP credentials with access to GCP VertexAI - Basic setup completed from the [Basic Usage](../basic-usage.md) guide - Basic configuration removed as described in the [Advanced Configuration](./index.md) overview ## Authentication Options Agent Router supports three authentication methods for GCP VertexAI: 1. **Application Default Credentials (ADC)** - Recommended for GKE with Workload Identity 2. **Service Account Key Files** - For explicit JSON credentials 3. **Workload Identity Federation** - For cross-cloud authentication ### Option 1: Application Default Credentials (Recommended for GKE) When running on GKE with Workload Identity, configure only `projectName` and `region`: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: gcp-credentials spec: targetRefs: - group: aigateway.envoyproxy.io kind: AIServiceBackend name: your-backend type: GCPCredentials gcpCredentials: projectName: YOUR_PROJECT_NAME region: us-central1 ``` ADC automatically handles credential rotation via GKE Workload Identity. ### Option 2: Service Account Key Files For non-GKE environments, use a service account key file: 1. Your GCP project id and name. 2. In your GCP project, enable VertexAI API access. 3. Create a GCP service account and generate the JSON key file. **Security Note:** Service account key files should be avoided in production when possible. Use ADC/Workload Identity instead. ## Configuration Steps ### 1. Download configuration template ```shell curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/gcp_vertex.yaml ``` ### 2. Configure GCP Credentials Edit the `gcp_vertex.yaml` file to replace these placeholder values: - `GCP_PROJECT_NAME`: Your GCP project name - `GCP_REGION`: GCP region - Update the generated service account key JSON string in the secret **Security Note:** Make sure to keep your GCP service account credentials secure and never commit them to version control. The credentials will be stored in Kubernetes secrets. ### 3. Apply Configuration Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, then the secret credential update will be picked up automatically in a few seconds. ```shell kubectl apply -f gcp_vertex.yaml kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system \ --for=condition=Ready ``` ### 4. Test the Configuration You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. See the [Basic Usage](../basic-usage.md) page for instructions. To access a Gemini model with chat completion endpoint: ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "gemini-2.5-flash", "messages": [ { "role": "user", "content": "Hi." } ] }' \ $GATEWAY_URL/v1/chat/completions ``` To access an Anthropic model with chat completion endpoint: ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "claude-3-7-sonnet@20250219", "messages": [ { "role": "user", "content": "What is capital of France?" } ], "max_completion_tokens": 100 }' \ $GATEWAY_URL/v1/chat/completions ``` Expected output: ```json { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "The capital of France is Paris. Paris is not only the capital city but also the largest city in France, known for its cultural significance, historic landmarks like the Eiffel Tower and the Louvre Museum, and its influence in fashion, art, and cuisine.", "role": "assistant" } } ], "object": "chat.completion", "usage": { "completion_tokens": 58, "prompt_tokens": 13, "total_tokens": 71 } } ``` You can also access an Anthropic model with native Anthropic messages endpoint: ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "claude-3-7-sonnet@20250219", "messages": [ { "role": "user", "content": "What is capital of France?" } ], "max_tokens": 100 }' \ $GATEWAY_URL/anthropic/v1/messages ``` ## Troubleshooting If you encounter issues: 1. Verify your GCP credentials are correct and active 2. Check pod status: ```shell kubectl get pods -n envoy-gateway-system ``` 3. View controller logs: ```shell kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller ``` 4. Common errors: - 401/403: Invalid credentials or insufficient permissions - 404: Model not found or not available in a region - 429: Rate limit exceeded ## Configuring More Models To use more models, add more [AIGatewayRouteRule]s to the `gcp_vertex.yaml` file with the [model ID] in the `value` field. For example, to use [Claude 3 Sonnet] ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: envoy-ai-gateway-basic-gcp-gemini namespace: default spec: schema: name: OpenAI parentRefs: - name: envoy-ai-gateway-basic kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: gemini-2.5-flash-pro backendRefs: - name: envoy-ai-gateway-basic-gcp ``` [AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule [model ID]: https://cloud.google.com/vertex-ai/generative-ai/docs/models [Anthropic Claude]: https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude --- # Connect Tetrate Agent Router Service (TARS) Source: https://theagentrouter.ai/docs/getting-started/connect-providers/tars This guide will help you configure Agent Router to work with Tetrate Agent Router Service (TARS)'s models. ## Prerequisites Before you begin, you'll need: - An API key from [Tetrate Agent Router Service's platform](https://router.tetrate.ai/) - Basic setup completed from the [Basic Usage](../basic-usage.md) guide - Basic configuration removed as described in the [Advanced Configuration](./index.md) overview ## Configuration Steps **Ready to proceed?:** Ensure you have followed the steps in [Connect Providers](../connect-providers/) ### 1. Download configuration template ```shell curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/tars.yaml ``` ### 2. Configure Tetrate Agent Router Service (TARS) Credentials Edit the `tars.yaml` file to replace the TARS placeholder value: - Find the section containing `TARS_API_KEY` - Replace it with your actual TARS API key **Security Note:** Make sure to keep your API key secure and never commit it to version control. The key will be stored in a Kubernetes secret. ### 3. Apply Configuration Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, then the secret credential update will be picked up automatically in a few seconds. ```shell kubectl apply -f tars.yaml kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system \ --for=condition=Ready ``` ### 4. Test the Configuration You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. See the [Basic Usage](../basic-usage.md) page for instructions. #### Test Chat Completions ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o-mini", "messages": [ { "role": "user", "content": "Hi." } ] }' \ $GATEWAY_URL/v1/chat/completions ``` #### Test Completions (Legacy) TARS fully supports the legacy completions endpoint: ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "babbage-002", "prompt": "def fib(n):\n if n <= 1:\n return n\n else:\n return fib(n-1) + fib(n-2)", "max_tokens": 25, "temperature": 0.4, "top_p": 0.9 }' \ $GATEWAY_URL/v1/completions ``` ## Troubleshooting If you encounter issues: 1. Verify your API key is correct and active 2. Check pod status: ```shell kubectl get pods -n envoy-gateway-system ``` 3. View controller logs: ```shell kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller ``` 4. View External Processor Logs ```shell kubectl logs -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic -c ai-gateway-extproc ``` 5. Common errors: - 401: Invalid API key - 429: Rate limit exceeded - 503: TARS service unavailable --- # Connect Anthropic Source: https://theagentrouter.ai/docs/getting-started/connect-providers/anthropic This guide will help you configure Agent Router to work with Anthropic's models. ## Prerequisites Before you begin, you'll need: - An Anthropic API key from [Anthropic's console](https://console.anthropic.com) - Basic setup completed from the [Basic Usage](../basic-usage.md) guide - Basic configuration removed as described in the [Advanced Configuration](./index.md) overview ## Configuration Steps **Ready to proceed?:** Ensure you have followed the steps in [Connect Providers](../connect-providers/) ### 1. Download configuration template ```shell curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/anthropic.yaml ``` ### 2. Configure Anthropic Credentials Edit the `anthropic.yaml` file to replace the Anthropic placeholder value: - Find the section containing `ANTHROPIC_API_KEY` - Replace it with your actual Anthropic API key **Security Note:** Make sure to keep your API key secure and never commit it to version control. The key will be stored in a Kubernetes secret. ### 3. Apply Configuration Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, then the secret credential update will be picked up automatically in a few seconds. ```shell kubectl apply -f anthropic.yaml kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system \ --for=condition=Ready ``` ### 4. Test the Configuration You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. See the [Basic Usage](../basic-usage.md) page for instructions. #### Test Chat Completions ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "claude-sonnet-4-5", "messages": [ { "role": "user", "content": "Hi." } ] }' \ $GATEWAY_URL/v1/chat/completions ``` ## Troubleshooting If you encounter issues: 1. Verify your API key is correct and active 2. Check pod status: ```shell kubectl get pods -n envoy-gateway-system ``` 3. View controller logs: ```shell kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller ``` 4. View External Processor Logs ```shell kubectl logs -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic -c ai-gateway-extproc ``` 5. Common errors: - 401: Invalid API key - 429: Rate limit exceeded - 503: Anthropic service unavailable ## Configuring More Models To use more models, add more [AIGatewayRouteRule]s to the `anthropic.yaml` file with the model name in the `value` field. For example, let's add [claude-haiku-3-5](https://docs.anthropic.com/en/docs/about-claude/models#claude-3.5-haiku) as a chat completion model: ```yaml apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: AIGatewayRoute metadata: name: envoy-ai-gateway-basic-anthropic namespace: default spec: parentRefs: - name: envoy-ai-gateway-basic kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: claude-sonnet-4-5 - matches: - headers: - type: Exact name: x-ai-eg-model value: claude-haiku-3-5 backendRefs: - name: envoy-ai-gateway-basic-anthropic ``` ## Next Steps After configuring Anthropic: - [Connect OpenAI](./openai.md) to add another provider - [Connect AWS Bedrock](./aws-bedrock.md) to add another provider [AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule --- # Connect Bitdeer AI Source: https://theagentrouter.ai/docs/getting-started/connect-providers/bitdeer This guide will help you configure Agent Router to work with Bitdeer AI's serverless models through its OpenAI-compatible API. ## Prerequisites Before you begin, you'll need: - An API key from the [Bitdeer AI Console](https://www.bitdeer.ai/model/apikeys) - Basic setup completed from the [Basic Usage](../basic-usage.md) guide - Basic configuration removed as described in the [Advanced Configuration](./index.md) overview ## Configuration Steps **Info:** Ready to proceed? Ensure you have followed the steps in [Connect Providers](./index.md). ### 1. Download configuration template ```shell curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/basic/bitdeer.yaml ``` ### 2. Configure Bitdeer AI Credentials Edit the `bitdeer.yaml` file to replace the Bitdeer placeholder value: - Find the section containing `BITDEER_API_KEY` - Replace it with your actual Bitdeer AI API key **Security Note:** Make sure to keep your API key secure and never commit it to version control. The key will be stored in a Kubernetes secret. ### 3. Apply Configuration Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, the secret credential update will be picked up automatically in a few seconds. ```shell kubectl apply -f bitdeer.yaml kubectl wait pods --timeout=2m \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -n envoy-gateway-system \ --for=condition=Ready ``` The template sends requests to Bitdeer AI's OpenAI-compatible endpoint at `https://api-inference.bitdeer.ai/v1`. ### 4. Test the Configuration You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. See the [Basic Usage](../basic-usage.md) page for instructions. #### Test Chat Completions ```shell curl -H "Content-Type: application/json" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Pro", "messages": [ { "role": "user", "content": "Explain serverless inference in one sentence." } ] }' \ $GATEWAY_URL/v1/chat/completions ``` #### Test Streaming ```shell curl -N -H "Content-Type: application/json" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Pro", "messages": [ { "role": "user", "content": "Write a short poem about inference." } ], "stream": true }' \ $GATEWAY_URL/v1/chat/completions ``` ## Supported Models The template configures `deepseek-ai/DeepSeek-V4-Pro`. Other model IDs include `Qwen/Qwen3.5-397B-A17B` and `moonshotai/Kimi-K2.6`. See the [Bitdeer AI Model Catalog](https://developers.bitdeer.ai/docs/api/models-catalog) for the current list and each model's supported capabilities. ## Troubleshooting If you encounter issues: - Verify your API key is correct and active - Confirm the requested model ID appears in the Bitdeer AI Model Catalog - Check pod status: ```shell kubectl get pods -n envoy-gateway-system ``` - View controller logs: ```shell kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller ``` - View External Processor logs: ```shell kubectl logs -n envoy-gateway-system \ -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ -c ai-gateway-extproc ``` Common errors: - `401`: Invalid API key - `429`: Rate limit exceeded - `503`: Bitdeer AI service unavailable ## Configuring More Models To use more models, add more matching rules to the `AIGatewayRoute` in `bitdeer.yaml`. Set each rule's `x-ai-eg-model` header value to a model ID from the [Bitdeer AI Model Catalog](https://developers.bitdeer.ai/docs/api/models-catalog). For example, add Qwen and Kimi models alongside the default DeepSeek model: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: envoy-ai-gateway-basic-bitdeer namespace: default spec: parentRefs: - name: envoy-ai-gateway-basic kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: deepseek-ai/DeepSeek-V4-Pro - headers: - type: Exact name: x-ai-eg-model value: Qwen/Qwen3.5-397B-A17B - headers: - type: Exact name: x-ai-eg-model value: moonshotai/Kimi-K2.6 backendRefs: - name: envoy-ai-gateway-basic-bitdeer ``` ## Next Steps After configuring Bitdeer AI, return to [Connect Providers](./index.md) to add another provider. --- # Agent Router CLI Source: https://theagentrouter.ai/docs/cli/ The Agent Router CLI, `aigw`, is a command-line interface that provides a set of useful tools for using the gateway. **Warning:** The CLI is experimental and currently under active development. Currently, you can do the following with the `aigw` CLI: - **Run**: Run the Agent Router locally as a standalone proxy with a given configuration file without any dependencies such as docker or Kubernetes. --- # Installation Source: https://theagentrouter.ai/docs/cli/aigwinstall ## Official CLI binaries Each release includes the binaries for the `aigw` CLI build for different platforms. They can be downloaded directly from the corresponding release in the [GitHub releases page](https://github.com/theagentrouter/agent-router/releases). ## Using the Docker image You can also use the official Docker images to run the CLI without installing it locally. The CLI images are available at: https://hub.docker.com/r/envoyproxy/ai-gateway-cli/tags To run the CLI using Docker, you only need to expose the port where the standalone `aigw` listens to and configure the environment variables for the credentials. If you want to use a custom configuration file, you can mount it as a volume. The following example runs the AI Gateway with the default configuration for the [OpenAI provider](../getting-started/connect-providers/openai.md): ```shell docker run --rm -p 1975:1975 -e OPENAI_API_KEY=OPENAI_API_KEY envoyproxy/ai-gateway-cli run ``` ## Building the latest version To use the latest version, you can use the following commands to clone the repo and build the CLI: ```shell git clone https://github.com/theagentrouter/agent-router.git cd agent-router go install ./cmd/aigw ``` **Tip:** `go install` command installs a binary in the `$(go env GOPATH)/bin` directory. Make sure that the `$(go env GOPATH)/bin` directory is in your `PATH` environment variable. For example, you can add the following line to your shell profile (e.g., `~/.bashrc`, `~/.zshrc`, etc.): ```sh export PATH=$PATH:$(go env GOPATH)/bin ``` Now, you can check if the installation was successful by running the following command: ```sh aigw --help ``` This will display the help message for the `aigw` CLI like this: ``` Usage: aigw [flags] Envoy AI Gateway CLI Flags: -h, --help Show context-sensitive help. Commands: version [flags] Show version. run [] [flags] Run the AI Gateway locally for given configuration. Run "aigw --help" for more information on a command. ``` ## Configuration The [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) defines standard locations for user-specific files: - **Config files**: User-specific configuration (persistent, shared) - **Data files**: Downloaded binaries (persistent, shared) - **State files**: Logs and configs per run (persistent, debugging) - **Runtime files**: Ephemeral files like sockets (deleted on reboot) `aigw` adopts these conventions to separate configuration, downloaded Envoy binaries, logs, and ephemeral runtime files. | Environment Variable | Default Path | CLI Flag | | -------------------- | --------------------- | --------------- | | `AIGW_CONFIG_HOME` | `~/.config/aigw` | `--config-home` | | `AIGW_DATA_HOME` | `~/.local/share/aigw` | `--data-home` | | `AIGW_STATE_HOME` | `~/.local/state/aigw` | `--state-home` | | `AIGW_RUNTIME_DIR` | `/tmp/aigw-${UID}` | `--runtime-dir` | **Priority**: CLI flags > Environment variables > Defaults ## What's next? The following sections provide more information about each of the CLI commands: - [aigw run](./run.md): Run the AI Gateway locally for a given configuration. --- # aigw run Source: https://theagentrouter.ai/docs/cli/aigwrun ## Overview This command runs the Agent Router locally as a standalone proxy with a given configuration file without any dependencies such as docker or Kubernetes. Since the project is primarily focused on the Kubernetes environment, this command is useful for testing the configuration locally before deploying it to a Kubernetes cluster. Not only does it help in testing the configuration, but it is also useful in a local development environment of the provider-agnostic AI applications. **Warning:** Currently, `aigw run` supports Linux and macOS. ## Quick Start The simplest way to get started is to have `aigw` generate a configuration for you using environment variables. `aigw` supports auto-configuration using the same environment variables as the OpenAI SDK, making it easy to integrate with existing tooling. Here are some examples: ```bash # OpenAI OPENAI_API_KEY=sk-your-key aigw run # Azure OpenAI AZURE_OPENAI_ENDPOINT=https://example.openai.azure.com \ AZURE_OPENAI_API_KEY=your-key \ OPENAI_API_VERSION=2024-12-01-preview \ aigw run # Tetrate Agent Router Service (TARS) OPENAI_BASE_URL=https://api.router.tetrate.ai/v1 OPENAI_API_KEY=sk-your-key aigw run # Ollama running locally OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_API_KEY=unused aigw run ``` Now, the AI Gateway is running locally with the default configuration serving at `localhost:1975`. Then, open a new terminal and run the following curl commands to test the AI Gateway. For example, use `qwen2.5:0.5b` model to route to Ollama, assuming Ollama is running locally and `ollama pull qwen2.5:0.5b` is executed to pull the model. ```shell curl -H "Content-Type: application/json" -XPOST http://localhost:1975/v1/chat/completions \ -d '{"model": "qwen2.5:0.5b","messages": [{"role": "user", "content": "Say this is a test!"}]}' ``` ### Supported Environment Variables The following environment variables are compatible with the OpenAI SDK: **OpenAI / OpenAI-compatible backends:** When `OPENAI_API_KEY` is set, the following environment variables are read: | Variable | Required | Example | Description | | ----------------- | -------- | --------------------------- | ---------------------------------------------------- | | `OPENAI_API_KEY` | Yes | `sk-proj-...` | API key for authentication (use "unused" for Ollama) | | `OPENAI_BASE_URL` | No | `https://api.openai.com/v1` | Base URL of your OpenAI-compatible backend | **Azure OpenAI:** When `AZURE_OPENAI_API_KEY` is set, the following environment variables are read: | Variable | Required | Example | Description | | ----------------------- | -------- | ---------------------------------- | ------------------------------------------- | | `AZURE_OPENAI_ENDPOINT` | Yes | `https://example.openai.azure.com` | Your Azure endpoint, including the resource | | `AZURE_OPENAI_API_KEY` | Yes | `abc123...` | API key for authentication | | `OPENAI_API_VERSION` | Yes | `2024-12-01-preview` | Azure OpenAI API version | **Optional headers (both OpenAI and Azure OpenAI):** | Variable | Example | Description | | ------------------- | ---------- | ---------------------------------------------------------------------------------------------- | | `OPENAI_ORG_ID` | `org-...` | Organization ID - adds `OpenAI-Organization` request header for billing and access control | | `OPENAI_PROJECT_ID` | `proj_...` | Project ID - adds `OpenAI-Project` request header for project-level billing and access control | ## Custom Configuration To run the AI Gateway with a custom configuration, provide the path to the configuration file as an argument to the `aigw run` command. For example, to run the AI Gateway with a custom configuration file named `config.yaml`, run the following command: ```shell aigw run config.yaml ``` The configuration uses the same API as the Agent Router custom resources definitions. See [API Reference](../api/) for more information. The best way to start customizing the configuration is to start with an [example configuration](https://github.com/theagentrouter/agent-router/tree/main/examples) and modify it as needed. ### Modify an Example Configuration First, download an example configuration to use as a starting point: ```shell curl -o ollama.yaml https://raw.githubusercontent.com/theagentrouter/agent-router/refs/heads//examples/aigw/ollama.yaml ``` Next, let's say change the model matcher from `.*` (match all) to specifically match `deepseek-r1:1.5b` and save the configuration to `custom.yaml`: ```diff --- ollama.yaml +++ custom.yaml @@ -88,7 +88,7 @@ - headers: - type: RegularExpression name: x-ai-eg-model - value: .* + value: deepseek-r1:1.5b ``` You can also use environment variable substitution (`envsubst`) to allow small changes without needing to copy a file. For example, you could use this syntax instead, to default the model to the `CHAT_MODEL` variable. ```diff --- ollama.yaml +++ custom.yaml @@ -88,7 +88,7 @@ - headers: - type: RegularExpression name: x-ai-eg-model - value: .* + value: ${CHAT_MODEL:=deepseek-r1:1.5b} ``` ### Run with the Custom Configuration Now, run the AI Gateway with the custom configuration by running the following command: ```shell aigw run custom.yaml ``` Now, the AI Gateway is running locally with the custom configuration serving at `localhost:1975`. ```shell curl -H "Content-Type: application/json" -XPOST http://localhost:1975/v1/chat/completions \ -d '{"model": "deepseek-r1:1.5b","messages": [{"role": "user", "content": "Say this is a test!"}]}' ``` ## MCP Configuration `aigw run` supports running as an [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) Gateway, allowing AI agents to connect to multiple MCP servers through a unified endpoint. The gateway aggregates tools from multiple backends, applies security policies, and provides observability for MCP traffic. ### Using MCP Servers Configuration File Use the `--mcp-config` flag to load MCP server configuration from a JSON file. The format follows the canonical [MCP servers configuration](https://modelcontextprotocol.io/docs/getting-started/intro#configuring-mcp-servers) used by clients like Claude Desktop, Cursor, and VS Code: ```json { "mcpServers": { "context7": { "type": "http", "url": "https://mcp.context7.com/mcp" }, "github": { "type": "http", "url": "https://api.githubcopilot.com/mcp/readonly", "headers": { "Authorization": "Bearer ${GITHUB_ACCESS_TOKEN}" }, "includeTools": ["issue_read", "list_issues"] } } } ``` Start the MCP Gateway: ```shell aigw run --mcp-config mcp-servers.json ``` The gateway will be available at `http://localhost:1975/mcp`. Configure your MCP client to connect to this URL. ### Using Inline JSON Configuration For quick testing or when the configuration is provided programmatically, use the `--mcp-json` flag: ```shell aigw run --mcp-json '{"mcpServers":{"context7":{"type":"http","url":"https://mcp.context7.com/mcp"}}}' ``` ### MCP Servers Configuration Format Each server configuration in the `mcpServers` object supports the following properties: **`type`** (string, required) The transport protocol for the MCP server. Supported values: `"http"`, `"streamable-http"`, `"streamable_http"`, or `"streamableHttp"`. **`url`** (string, required) The full endpoint URL for the MCP server, including protocol, hostname, and path. **`headers`** (object, optional) HTTP headers to send with requests to the MCP server. Authorization headers with Bearer tokens are automatically extracted and injected as API keys. Example: ```json "headers": { "Authorization": "Bearer ${GITHUB_TOKEN}", "X-Custom-Header": "value" } ``` Header values support environment variable substitution using `${VAR}` syntax. For example, `${GITHUB_TOKEN}` will be replaced at runtime with the value of the `GITHUB_TOKEN` environment variable. **`includeTools`** (array of strings, optional) List of specific tool names to expose from this server. If not specified, all tools from the server are exposed. This follows the same convention as [Gemini CLI's tool filtering](https://google-gemini.github.io/gemini-cli/docs/tools/mcp-server.html#optional). Example: ```json "includeTools": ["issue_read", "list_issues", "search_issues"] ``` ### Testing the MCP Gateway Use the [MCP Inspector](https://github.com/modelcontextprotocol/inspector) to test your gateway: ```shell # List available tools npx @modelcontextprotocol/inspector@0.16.8 --cli http://localhost:1975/mcp --method tools/list # Call a tool npx @modelcontextprotocol/inspector@0.16.8 \ --cli http://localhost:1975/mcp \ --method tools/call \ --tool-name context7__resolve-library-id \ --tool-arg libraryName="react" ``` **Note:** Tool names are automatically prefixed with the server name (e.g., `github__issue_read`, `context7__resolve-library-id`) to route calls to the correct backend. ### Advanced MCP Configuration For production deployments with advanced features like OAuth authentication, fine-grained access control, and Kubernetes integration, use the [`MCPRoute` API](https://theagentrouter.ai/docs/capabilities/mcp/). The same configuration can be used in both `aigw run` standalone mode and Kubernetes environments. ### Admin Endpoints While running, `aigw` serves admin endpoints on port `1064` by default: - `/metrics`: Prometheus metrics endpoint for [LLM/AI metrics](../capabilities/observability/metrics.md) - `/health`: Health check endpoint that verifies the external processor is healthy ## OpenTelemetry Agent Router's router joins and records distributed traces when supplied with an [OpenTelemetry](https://opentelemetry.io/) collector endpoint. Requests to the OpenAI Chat Completions and Embeddings endpoints are recorded as Spans which include typical timing and request details. In addition, there are GenAI attributes representing the LLM or Embeddings call including full request and response details, defined by [OpenInference semantic conventions][openinference]. OpenInference attributes default to include full request and response data for both chat completions and embeddings. This can be toggled with configuration, but when enabled allows systems like [Arize Phoenix][phoenix] to perform evaluations of production requests captured in OpenTelemetry spans. For chat completions, this includes traditional LLM metrics such as correctness and hallucination detection. For embeddings, it enables agentic RAG evaluations focused on retrieval and semantic analysis. ### OpenTelemetry configuration `aigw run` supports OpenTelemetry tracing via environment variables: - **[OTEL SDK][otel-env]**: OTLP exporter configuration that controls span and metrics export such as: - `OTEL_EXPORTER_OTLP_ENDPOINT`: Collector endpoint (e.g., `http://phoenix:6006`) - `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`: Override traces endpoint separately - `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`: Override metrics endpoint separately - `OTEL_TRACES_EXPORTER`: Traces exporter type (e.g., `console`, `otlp`, `none`) - `OTEL_METRICS_EXPORTER`: Metrics exporter type (e.g., `console`, `otlp`, `none`) - `OTEL_BSP_SCHEDULE_DELAY`: Batch span processor delay (default: 5000ms) - `OTEL_METRIC_EXPORT_INTERVAL`: Metrics export interval (default: 60000ms) - **Semantic Convention**: Select which span attributes are recorded. See [Tracing][tracing] for the attribute differences. - `AI_GATEWAY_TRACING_SEMCONV`: `openinference` (default) or `gen_ai` for the OpenTelemetry GenAI conventions. An unrecognized value fails startup. - `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`: Capture message content when `gen_ai` is selected (default: `false`). The GenAI conventions treat content as opt-in, unlike OpenInference. - This also selects the vocabulary for MCP spans: the default keeps the gateway-specific `mcp.*` attributes, `gen_ai` opts into the OpenTelemetry MCP conventions. - **[OpenInference][openinference-config]**: Control sensitive data redaction, such as below. These apply only when `AI_GATEWAY_TRACING_SEMCONV` is `openinference`. There is [similar config][openinference-embeddings] for embeddings: - `OPENINFERENCE_HIDE_INPUTS`: Hide input messages/prompts (default: `false`) - `OPENINFERENCE_HIDE_OUTPUTS`: Hide output messages/completions (default: `false`) - `OPENINFERENCE_HIDE_EMBEDDINGS_TEXT`: Hide embeddings input (default: `false`) - `OPENINFERENCE_HIDE_EMBEDDINGS_VECTORS`: Hide embeddings output (default: `false`) - **Header Mapping**: Map HTTP request headers to span attributes, metric labels, and access log attributes. See [Session Tracking][session-tracking] for more details. - `OTEL_AIGW_REQUEST_HEADER_ATTRIBUTES`: Base mapping applied to metrics, spans, and access logs. Example: `x-tenant-id:tenant.id` - `OTEL_AIGW_SPAN_REQUEST_HEADER_ATTRIBUTES`: Span-only mapping (merged with base). Default: `agent-session-id:session.id` when unset (set to empty to disable). Example: `x-forwarded-proto:url.scheme` - `OTEL_AIGW_METRICS_REQUEST_HEADER_ATTRIBUTES`: Metrics-only mapping (merged with base). Example: `x-tenant-id:tenant.id` (metrics never default to `session.id`) - `OTEL_AIGW_LOG_REQUEST_HEADER_ATTRIBUTES`: Access-log-only mapping (merged with base). Default: `agent-session-id:session.id` when unset (set to empty to disable). Example: `x-forwarded-proto:url.scheme` See [docker-compose-otel.yaml][docker-compose-otel.yaml] for a complete example configuration. ## Configuration ### File Locations `aigw run` uses the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) to organize files: | Environment Variable | Default Path | Purpose | | -------------------- | --------------------- | --------------------------- | | `AIGW_CONFIG_HOME` | `~/.config/aigw` | User configuration files | | `AIGW_DATA_HOME` | `~/.local/share/aigw` | Downloaded Envoy binaries | | `AIGW_STATE_HOME` | `~/.local/state/aigw` | Persistent logs and configs | | `AIGW_RUNTIME_DIR` | `/tmp/aigw-${UID}` | Ephemeral runtime files | See [Installation - Configuration](./installation.md#configuration) for more details about XDG directories. ### File Mappings Each invocation creates a unique run identifier (`runID`) in format `YYYYMMDD_HHMMSS_UUU` to isolate concurrent runs: | File Type | Purpose | Path | Type | | ------------------------- | ---------------------------------------- | ---------------------------------------------------------------- | ------- | | Default Config | Configuration file location | `${AIGW_CONFIG_HOME}/config.yaml` | CONFIG | | Envoy Version Preference | Selected Envoy version (via func-e) | `${AIGW_CONFIG_HOME}/envoy-version` | CONFIG | | Envoy Binaries | Downloaded executables (via func-e) | `${AIGW_DATA_HOME}/envoy-versions/{version}/bin/envoy` | DATA | | AIGW Logs | Gateway logs and stderr output | `${AIGW_STATE_HOME}/runs/{runID}/aigw.log` | STATE | | Envoy Gateway Config | Generated EG configuration | `${AIGW_STATE_HOME}/runs/{runID}/envoy-gateway-config.yaml` | STATE | | Envoy Gateway Resources | Generated EG resources (Gateway, Routes) | `${AIGW_STATE_HOME}/runs/{runID}/envoy-ai-gateway-resources/...` | STATE | | External Processor Config | Generated extproc configuration | `${AIGW_STATE_HOME}/runs/{runID}/extproc-config.yaml` | STATE | | Envoy Run Logs (func-e) | Envoy stdout/stderr (via func-e) | `${AIGW_STATE_HOME}/envoy-runs/{runID}/stdout.log,stderr.log` | STATE | | UDS Socket | Unix domain socket for extproc | `${AIGW_RUNTIME_DIR}/{runID}/uds.sock` | RUNTIME | | Admin Address (func-e) | Envoy admin endpoint (via func-e) | `${AIGW_RUNTIME_DIR}/{runID}/admin-address.txt` | RUNTIME | **File Categories:** - **CONFIG**: User-specific configuration (persistent, shared across runs) - **DATA**: Downloaded binaries (persistent, shared across runs) - **STATE**: Per-run logs and configs (persistent for debugging) - **RUNTIME**: Ephemeral files like sockets (cleaned on reboot) ### `runID` By default, `aigw run` generates a timestamp-based `runID` for each invocation. You can customize this for predictable paths: ```shell # Use run ID "0" for Docker/Kubernetes deployments aigw run --run-id=0 # Or via environment variable AIGW_RUN_ID=production aigw run ``` Custom run IDs: - Enable predictable file paths in containers - Allow correlation across multiple runs with the same ID - Must not contain path separators (`/` or `\`) --- [openinference]: https://github.com/Arize-ai/openinference/tree/main/spec [phoenix]: https://docs.arize.com/phoenix [otel-env]: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/ [openinference-config]: https://github.com/Arize-ai/openinference/blob/main/spec/configuration.md [openinference-embeddings]: https://github.com/Arize-ai/openinference/blob/main/spec/embedding_spans.md [docker-compose-otel.yaml]: https://github.com/theagentrouter/agent-router/blob/main/cmd/aigw/docker-compose-otel.yaml [session-tracking]: ../capabilities/observability/tracing.md#session-tracking [tracing]: ../capabilities/observability/tracing.md#semantic-conventions --- # Concepts Source: https://theagentrouter.ai/docs/concepts/ Welcome to the Concepts section of Agent Router documentation. This section will help you understand the core components and how they work together to manage AI traffic effectively. ## Architecture - [System Architecture Overview](./architecture/system-architecture.md) - High-level overview of control and data planes - [Control Plane](./architecture/control-plane.md) - How configuration and management works - [Data Plane](./architecture/data-plane.md) - Request processing and traffic flow ## Resources - [Resources Overview](./resources.md) - Understanding the AI Gateway resources and their relationships --- # Architecture Source: https://theagentrouter.ai/docs/concepts/architecture/ This section provides a detailed look at the architectural components of Agent Router. Understanding the architecture will help you better deploy, configure, and maintain your gateway installation. ## Overview Agent Router follows a modern cloud-native architecture with distinct control and data planes. This separation of concerns allows for better scalability, maintainability, and flexibility in deployment options. Agent Router integrates with Envoy Gateway for the control plane and Envoy Proxy for the data plane. ## Key Concepts ### Control Plane A control plane is a component that manages the configuration of the data plane. We utilize Envoy Gateway as a central control plane, and Agent Router works in conjunction with it to manage the data plane configuration. ### Data Plane The data plane is the component that sits in the request path and processes the requests. In the context of Agent Router, the data plane consists of Envoy Proxy and the AI Gateway external processor that processes the AI requests. ## In This Section 1. [System Architecture Overview](./system-architecture.md) - High-level architecture overview - Control and data plane separation 2. [Control Plane Explained](./control-plane.md) - AI Gateway Controller - Envoy Gateway Controller - Configuration management - Resource orchestration 3. [Data Plane and Traffic Flow](./data-plane.md) - External Processor functionality - Request processing flow ## What's Next After understanding the architecture: - Check out our [Getting Started](../../getting-started/index.md) guide for hands-on experience --- # System Architecture Overview Source: https://theagentrouter.ai/docs/concepts/architecture/system-architecture Agent Router follows a modern cloud-native architecture pattern with distinct control and data planes. Let's explore how these components work together to manage AI traffic effectively. The architecture is divided into two main planes: 1. **Control Plane**: Responsible for configuring and managing the system - Kubernetes API Server serves as the configuration interface - AI Gateway Controller manages AI-specific resources and configurations - Envoy Gateway Controller handles core proxy configuration and xDS 2. **Data Plane**: Handles the actual request traffic - Envoy Proxy and External Processor work together to handle AI traffic - Envoy Proxy routes and manages all incoming requests - External Processor handles AI-specific transformations and validations - Rate Limit Service manages token-based rate limiting The Control Plane configures the Data Plane through several paths: - AI Gateway Controller configures the External Processor as well as fine-tuning xDS configurations via Envoy Gateway extension server mechanism. - Envoy Gateway Controller configures Envoy Proxy through xDS as well as Rate Limit Service. ```mermaid graph TB subgraph "Control Plane" K8s[Kubernetes API Server] Controller[AI Gateway Controller] EGController[Envoy Gateway Controller] Controller --> |GWAPI/EG resources|K8s K8s -->|AI Gateway resources| Controller K8s -->|GWAPI/EG resources| EGController Controller <-->|xDS fine-tune| EGController style Controller fill:#f9f,stroke:#333,stroke-width:2px style EGController fill:#ddf,stroke:#333 end subgraph "Data Plane" Envoy[Envoy Proxy] ExtProc[External Processor] RateLimit[Rate Limit Service] Envoy --> RateLimit Envoy --> ExtProc style Envoy fill:#f9f,stroke:#333,stroke-width:2px style ExtProc fill:#f9f,stroke:#333,stroke-width:2px style RateLimit fill:#dfd,stroke:#333 end Controller -.->|Configures| ExtProc EGController -.->|Configures| RateLimit EGController -.->|Updates Config| Envoy ``` For detailed information about specific components: - Learn more about the [Control Plane](./control-plane.md) components - Understand the [Data Plane](./data-plane.md) in detail --- # Control Plane Explained Source: https://theagentrouter.ai/docs/concepts/architecture/control-plane The control plane is responsible for configuring and managing the system. It consists of several key components working together to manage the AI Gateway configuration. ```mermaid graph TB %% Define styles for different component categories classDef aiGatewayStyle fill:#f9f,stroke:#E74C3C,color:#000 classDef envoyGatewayStyle fill:#ddf,stroke:#none,color:#000 classDef aiGatewayResourceStyle fill:#f9f,stroke:#E74C3C,color:#000 classDef envoyGatewayResourceStyle fill:#ddf,stroke:#none,color:#000 classDef generatedResourceStyle fill:#ddf,stroke:#none,color:#000 classDef envoyProxyStyle fill:#F5CBA7,stroke:#D35400,color:#000 classDef aiGatewayExtProcStyle fill:#f9f,stroke:#E74C3C,color:#000 subgraph "UserAppliedResources"["User Applied Resources"] subgraph "AI Gateway Resources" AIGatewayRoute:::aiGatewayResourceStyle AIServiceBackend:::aiGatewayResourceStyle BackendSecurityPolicy:::aiGatewayResourceStyle end subgraph "Envoy Gateway resources" Gateway:::envoyGatewayResourceStyle Backend:::envoyGatewayResourceStyle EGDots[...]:::envoyGatewayResourceStyle end end style UserAppliedResources fill:none,stroke-dasharray:5 AIGateway((Agent Router Controller)):::aiGatewayStyle EnvoyGateway((Envoy Gateway Controller)):::envoyGatewayStyle subgraph "Generated Resources" HTTPRoute:::generatedResourceStyle HTTPRouteFilter:::generatedResourceStyle ExtProcConfigSecret[AI Gateway ExtProc Secret]:::aiGatewayExtProcStyle end subgraph "EnvoyProxyPod"[Envoy Proxy Pod] EnvoyProxy[Envoy Proxy]:::envoyProxyStyle AIExtProc[AI Gateway ExtProc]:::aiGatewayExtProcStyle end style EnvoyProxyPod fill:none,stroke:5 AIGatewayRoute -.->AIGateway AIServiceBackend -.->AIGateway BackendSecurityPolicy -.->AIGateway Gateway -.->|Watching|AIGateway Gateway -.->EnvoyGateway Backend -.->EnvoyGateway EGDots -.->EnvoyGateway AIGateway -.->HTTPRoute AIGateway -.->HTTPRouteFilter AIGateway -.->ExtProcConfigSecret HTTPRoute -.->EnvoyGateway HTTPRouteFilter -.->EnvoyGateway ExtProcConfigSecret -.->AIExtProc EnvoyGateway -.->|xDS/Envoy config|EnvoyProxy AIGateway <-.->|Extension Server protocol|EnvoyGateway AIGateway -.->|Inserting ExtProc as Sidecar|AIExtProc ``` ## How It Works The control plane operates through a chain of components that work together to manage the configuration: 1. The Agent Router controller watches AI Gateway Custom Resources (CRs) 2. When changes are detected, it updates/generates the Envoy Gateway configuration 3. The Envoy Gateway communicates with the Agent Router controller via the [Envoy Gateway Extension server] protocol, and the Agent Router controller then fine-tunes the xDS configuration before the Envoy Gateway applies it to the Envoy Proxy. 4. The data plane (Envoy Proxy) processes AI traffic based on this configuration where the AI Gateway ExtProc runs as a sidecar to handle AI-specific processing. The sidecar container is inserted by the AI Gateway controller into the Envoy Proxy Pod. This architecture ensures a clear separation of concerns, where the AI Gateway controller focuses on AI-specific configuration while leveraging Envoy Gateway for general proxy management. ## Components ### 1. Agent Router Controller The AI Gateway Controller manages AI-specific components and configurations: #### ExtProc Management - Creates and updates ExtProc Secrets with processing rules as well as credentials - Inserts the AI Gateway ExtProc as a [sidecar container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) in the Envoy Proxy Pod via the [Kubernetes Admission Webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/). The container mounts the ExtProc config secret and communicates with the Envoy Proxy to process AI traffic. #### Resource Management - Watches AI Gateway Custom Resources (CRs) - Creates and manages `HTTPRoute` and `HTTPRouteFilter` resources - Manages backend security policies and authentication, including the credentials rotation #### Integration with Envoy Gateway - Works alongside Envoy Gateway Controller (not directly configuring Envoy) - Creates resources, such as `HTTPRoute`, that Envoy Gateway translates into Envoy configuration with AI-specific processing rules - Serves the [Envoy Gateway Extension Server] to fine-tune the Envoy configuration (xDS) - Enables token-based rate limiting through metadata ### 2. Envoy Gateway Controller - Manages the core Envoy configuration through xDS - Handles service discovery and load balancing - Manages TLS certificates - Translates Gateway API resources into Envoy configuration ## Notable Rationale - As explained, [Envoy Gateway Extension server] is used for fine-tuning the xDS configuration to implement our features. This allows us to leverage Envoy Gateway for core proxy management while still customizing the configuration for AI-specific needs. - For example, upstream filters (HTTP filter attached per Backend) is not supported in Envoy Gateway, so the AI Gateway controller uses the extension server to achieve this, which is essential to implement per-model priority routing. - Another example is that backendRef level priority configuration is not supported in Gateway API or Envoy Gateway, so the AI Gateway controller uses the extension server to insert this configuration into the xDS. - Delegating core proxy management to Envoy Gateway allows us to focus on AI-specific features and configurations, reducing duplication of effort and leveraging the strengths of both controllers. - For example, we don't need to re-implement Gateway API resource translation, service discovery, load balancing, and TLS management. - Inserting the ExtProc as a sidecar container brings the following benefits: - Performance: Running the ExtProc as a sidecar within the same Pod as Envoy Proxy reduces network latency and overhead, leading to faster processing of AI traffic. Notably, the communication between Envoy Proxy and the ExtProc happens over a local Unix Domain Socket (UDS), not over the network. - Simplified Deployment: Managing the ExtProc as part of the Envoy Proxy Pod simplifies deployment and scaling, as both components can be managed together. - Session Affinity: Running as a sidecar naturally ensures the session affinity between Envoy Proxy and the ExtProc which is crucial for implementing provider fallbacks. As we explain in [Data Plane and Traffic Flow], we have two phases in the request processing: Router-level and Upstream-level processing to handle provider fallbacks. Since Envoy does not support session affinity for External Processor access as of this writing, without the session affinity, the Upstream-level ExtProc might end up talking to a different ExtProc instance which does not have the context of the router level processing (mainly the original request body and headers). ## Next Steps To learn more: - Understand the [Data Plane and Traffic Flow] [Envoy Gateway Extension server]: https://gateway.envoyproxy.io/docs/tasks/extensibility/extension-server/ [Data Plane and Traffic Flow]: ./data-plane.md --- # Data Plane and Traffic Flow Source: https://theagentrouter.ai/docs/concepts/architecture/data-plane The data plane handles the actual request traffic, with the External Processor (ExtProc) playing a central role in managing AI-specific processing. ## Components The data plane consists of several key components: ### 1. Envoy Proxy The core proxy that handles all incoming traffic and integrates with: - External Processor for AI-specific processing - Rate Limit Service for token-based rate limiting - Various AI providers as backends ### 2. AI Gateway External Processor A specialized extension service of Envoy Proxy that handles AI-specific processing needs. It performs three main functions: 1. **Request Processing** - Routes requests to appropriate AI providers - Handles model selection and validation - Manages provider-specific authentication - Supports different API formats (OpenAI, AWS Bedrock) 2. **Token Management** - Tracks token usage from AI providers - Handles both streaming and non-streaming responses - Provides usage data for rate limiting decisions 3. **Provider Integration** - Transforms requests between different AI provider formats - Normalizes responses to a consistent format - Manages provider-specific requirements ### 3. Rate Limit Service Handles token-based rate limiting by: - Tracking token usage across requests - Enforcing rate limits based on token consumption - Managing rate limit budgets ## Request Processing Flow ```mermaid sequenceDiagram participant Client as Client (OpenAI SDK) participant Envoy as Envoy Proxy participant RLS as Rate Limit Service participant Processor as AI Gateway External Processor participant Provider as AI Provider / Upstream Client->>Envoy: Request Envoy->>Processor: Router-level ExtProc Request Note over Processor: Extract Model Name Processor-->>Envoy: ClearRouteCache; Envoy->>RLS: Check Rate Limit RLS-->>Envoy: ; loop Retry/Fallback loop Note over Envoy: Select Upstream/Endpoint Envoy->>Processor: Upstream level ExtProc Request Note over Processor: Request-Transform & Upstream Authnz Processor-->>Envoy: ; Envoy->>Provider: Forward Request Provider-->>Envoy: Response end Envoy->>Processor: Process Response Note over Processor: Response Transform & Extract Token Usage Processor-->>Envoy: Add Usage Metadata Envoy->>RLS: Reduce Rate Limit budget RLS-->>Envoy: ; Envoy->>Client: Response ``` The data plane processes requests through several key steps: ### 1. Request Path 1. **Routing**: Calculates the destination AI provider based on: - Request path - Headers - Model name extracted from the request body 2. **Request Transformation**: Prepares the request for the provider: - Request body transformation - Request path modification - Format adaptation 3. **Upstream Authorization**: Handles provider authentication: - API key management - Header modifications - Authentication token handling 4. **Token Rate Limiting Check**: Checks the request against the Rate Limit Service: - Validates token usage - Enforces rate limits based on configured budgets ### 2. Response Path 1. **Response Transformation**: - Transforms provider response for client compatibility - Normalizes response format - Handles streaming responses 2. **Token Usage Management**: - Extracts token usage from responses - Calculates usage based on configuration - Stores usage in per-request dynamic metadata - Enables rate limiting based on token consumption ## Notable Rationale - Why the External Processor is separated into two phases (Router-level and Upstream-level): - In Envoy, retry/fallback happens after the router filter at the upstream level. For example, when the upstream server returns 5xx, Envoy does not invoke the router level filter again. Instead, it invokes only the upstream level filters. In our case, retry/fallback will make the requests to totally different AI providers. For example, on the first try, it goes to OpenAI, and on the second try, it goes to AWS Bedrock. In this case, we need to do different request transformations and upstream authorizations. So, this logic needs to be in the upstream level filter. - Why the External Processor? - The External Processor is the most powerful battle-tested and production-ready extension point in Envoy. It allows us to implement complex logic without modifying Envoy's core codebase. - [Dynamic Modules](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/dynamic_modules) could be a future alternative as it offers better performance as well as less complexity in the overall architecture. The work is tracked in [envoyproxy/ai-gateway#90](https://github.com/theagentrouter/agent-router/issues/90). ## Next Steps To learn more: - Explore the [System Architecture](./system-architecture.md) - Check out our [Getting Started](../../getting-started/index.md) guide for hands-on experience --- # Resources Source: https://theagentrouter.ai/docs/concepts/resources The Agent Router uses several custom resources to manage AI traffic. Here's an overview of the key resources and how they relate to each other: ## Resource Reference | Resource | Purpose | API Reference | | --------------------- | ---------------------------------------------------- | ------------------------------------------------------------- | | AIGatewayRoute | Defines unified API and routing rules for AI traffic | [AIGatewayRoute](../api/api.mdx#aigatewayroute) | | AIServiceBackend | Represents individual AI service backends | [AIServiceBackend](../api/api.mdx#aiservicebackend) | | BackendSecurityPolicy | Configures authentication for backend access | [BackendSecurityPolicy](../api/api.mdx#backendsecuritypolicy) | ## Core Resources ### AIGatewayRoute A resource that defines a unified AI API for a Gateway, allowing clients to interact with multiple AI backends using a single schema. - Specifies the input API schema for client requests - Contains routing rules to direct traffic to appropriate backends - Manages request/response transformations between different API schemas - Can track LLM request costs (like token usage) ### AIServiceBackend Represents a single AI service backend that handles traffic with a specific API schema. - Defines the output API schema the backend expects - References a Kubernetes Service or Envoy Gateway Backend - Can reference a BackendSecurityPolicy for authentication ### BackendSecurityPolicy Configures authentication and authorization rules for backend access. - API Key authentication - AWS credentials authentication ## Resource Relationships ```mermaid graph TD A[AIGatewayRoute] -->|references| B[AIServiceBackend] B -->|references| C[K8s Service/Backend] B -->|references| D[BackendSecurityPolicy] D -->|contains| E[API Key/AWS Credentials] ``` The AIGatewayRoute acts as the entry point, defining how client requests are processed and routed to one or more AIServiceBackends. Each AIServiceBackend can reference a BackendSecurityPolicy, which provides the necessary credentials for accessing the underlying AI service. --- # Capabilities Source: https://theagentrouter.ai/docs/capabilities/ Welcome to the Agent Router capabilities documentation! This section provides detailed information about the various features and capabilities that Agent Router offers to help you manage and optimize your AI/LLM traffic. ## LLM Providers Integrations Support for various Large Language Model providers: - **[Connecting to AI Providers](./llm-integrations/connect-providers.md)**: Learn how to establish connectivity with any supported AI provider - **[Supported Providers](./llm-integrations/supported-providers.md)**: Compatible AI/LLM service providers - **[Supported Endpoints](./llm-integrations/supported-endpoints.md)**: Available API endpoints and operations - **[Vendor-Specific Fields](./llm-integrations/vendor-specific-fields.md)**: Use backend-specific parameters and access provider-unique capabilities in your OpenAI-compatible requests - **[Prompt Caching](./llm-integrations/prompt-caching.md)**: Provider-agnostic prompt caching using unified cache_control API ## Inference Optimization Advanced inference optimization capabilities for AI/LLM workloads: - **[InferencePool Support](./inference/inferencepool-support.md)**: Intelligent routing and load balancing for inference endpoints - **[HTTPRoute + InferencePool](./inference/httproute-inferencepool.md)**: Basic inference routing with standard Gateway API - **[AIGatewayRoute + InferencePool](./inference/aigatewayroute-inferencepool.md)**: Advanced AI-specific routing with enhanced features ## Gateway Configuration - **[GatewayConfig](./gateway-config.md)**: Gateway-scoped configuration for the external processor (env vars, resources, shared settings) - **[Scaling](./scaling.md)**: Configure multiple controller replicas and horizontal pod autoscaling for production deployments ## Traffic Management Comprehensive traffic handling and routing capabilities: - **[Model Virtualization](./traffic/model-virtualization.md)**: Abstract and virtualize AI models - **[Provider Fallback](./traffic/provider-fallback.md)**: Automatic failover between AI providers - **[Quota Policy](./traffic/quota-policy.md)**: Token-based quota management for controlling total consumption budgets - **[Usage-based Rate Limiting](./traffic/usage-based-ratelimiting.md)**: Token-aware rate limiting for AI workloads - **[Header and Body Mutations](./traffic/header-body-mutations.md)**: Customize HTTP headers and JSON body fields per backend or route ## Security Robust security features for AI gateway deployments: - **[Upstream Authentication](./security/upstream-auth.mdx)**: Secure authentication to upstream AI services ## Model Context Protocol (MCP) Connect AI agents to external tools and data sources: - **[MCP Gateway](./mcp/)**: Server multiplexing, tool routing, OAuth authentication, and observability for MCP workloads ## Agent2Agent (A2A) Route traffic between agents: - **[A2A Traffic (preview)](./a2a/)**: Route A2A agents through the gateway today using Envoy's native A2A filter, ahead of a first-class A2A route type ## Observability Monitoring and observability tools for AI workloads: - **[Metrics](./observability/metrics.md)**: Comprehensive metrics collection and monitoring --- # Gateway Configuration Source: https://theagentrouter.ai/docs/capabilities/gateway-config The `GatewayConfig` CRD provides gateway-scoped configuration for the AI Gateway external processor container. This allows you to configure environment variables and resource requirements at the Gateway level, rather than at the route level. ## Overview Use `GatewayConfig` when you need to: - Configure per-gateway OpenTelemetry tracing settings - Set resource requirements (CPU/memory) for the external processor - Share configuration across multiple Gateways - Configure environment variables for different gateway instances without affecting others ## Usage ### Creating a GatewayConfig Create a `GatewayConfig` resource with your desired configuration: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: GatewayConfig metadata: name: my-gateway-config namespace: default spec: extProc: kubernetes: env: - name: OTEL_EXPORTER_OTLP_ENDPOINT value: "http://otel-collector:4317" - name: OTEL_SERVICE_NAME value: "my-ai-gateway" resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "512Mi" ``` ### Referencing from a Gateway Reference the `GatewayConfig` from your Gateway using an annotation: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: my-gateway namespace: default annotations: aigateway.envoyproxy.io/gateway-config: my-gateway-config spec: gatewayClassName: envoy-gateway listeners: - name: http protocol: HTTP port: 8080 ``` **Note:** The `GatewayConfig` must be in the same namespace as the Gateway that references it. ## Configuration Options ### Environment Variables The `spec.extProc.kubernetes.env` field accepts a list of Kubernetes `EnvVar` objects: ```yaml spec: extProc: kubernetes: env: - name: OTEL_EXPORTER_OTLP_ENDPOINT value: "http://otel-collector:4317" - name: OTEL_EXPORTER_OTLP_HEADERS value: "api-key=your-secret" - name: LOG_LEVEL value: "debug" ``` ### Resource Requirements The `spec.extProc.kubernetes.resources` field configures compute resources for the external processor container: ```yaml spec: extProc: kubernetes: resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "512Mi" ``` If not specified, Kubernetes default resource allocations are used. ### Metadata Forwarding Namespaces The `spec.extProc.metadataForwardingNamespaces` field lists the untyped dynamic metadata namespaces Envoy forwards to the external processor on this Gateway's AI traffic. It is required for [per-request credential override from dynamic metadata](./security/upstream-auth.mdx#from-dynamic-metadata-recommended): a `BackendSecurityPolicy` using `credentialOverride.fromDynamicMetadata` only receives its metadata when the namespace it reads from is declared here. ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: GatewayConfig metadata: name: my-gateway-config namespace: default spec: extProc: metadataForwardingNamespaces: - envoy.filters.http.ext_authz ``` These namespaces travel into the processor. The processor writes `io.envoy.ai_gateway` back to Envoy, which receives that namespace for token-based rate limiting and access logs. The list belongs to the Gateway that declares it and governs that Gateway's AI traffic, so attaching a route to a Gateway cannot widen what a different Gateway forwards. Under Envoy Gateway's `mergeGateways`, every Gateway in the class shares one Envoy, so a namespace any one of them declares is forwarded for all of their AI traffic. The per-Gateway isolation described above does not hold in that setup. ### Forward Proxy (Egress) The `spec.forwardProxy` field routes all upstream AI/LLM traffic from Gateways referencing this `GatewayConfig` through an HTTP CONNECT forward proxy. This is intended for data planes deployed in locked-down networks that have no direct outbound access to providers and require all egress to traverse a proxy. ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: GatewayConfig metadata: name: my-gateway-config namespace: default spec: forwardProxy: # host:port of the HTTP CONNECT proxy (host may be a hostname or IP; port is required). address: proxy.corp:3128 ``` Under the hood, each upstream cluster's transport socket is wrapped in Envoy's `http_11_proxy` transport socket, which opens an HTTP/1.1 `CONNECT` tunnel to the proxy and then establishes the upstream connection through it. The upstream TLS session is preserved end-to-end inside the tunnel, so the proxy sees only the `CONNECT` and the encrypted bytes — not the request contents. **Note:** The forward proxy applies to all AI Gateway upstream (LLM) clusters for Gateways that reference this `GatewayConfig`. InferencePool backends, which use in-cluster endpoints, are not proxied. **Warning:** Proxy authentication (for example, `Proxy-Authorization`) and a TLS connection to the proxy itself are not currently supported. The proxy is expected to be reachable over plaintext HTTP and to authorize the gateway by network policy (for example, an IP allowlist). ## Environment Variable Precedence Environment variables can be configured at multiple levels. The precedence order is (highest to lowest): 1. **GatewayConfig.spec.extProc.kubernetes.env** - Highest priority 2. **Global controller flags** (`--extproc-extra-env-vars`) - Lower priority When the same environment variable is defined at multiple levels, the higher precedence value is used. ### Example If the controller is started with: ``` --extproc-extra-env-vars="LOG_LEVEL=info;GLOBAL_VAR=global" ``` And a GatewayConfig defines: ```yaml env: - name: LOG_LEVEL value: "debug" - name: CONFIG_VAR value: "config" ``` The resulting environment variables will be: - `LOG_LEVEL=debug` (GatewayConfig overrides global) - `GLOBAL_VAR=global` (from global) - `CONFIG_VAR=config` (from GatewayConfig) ## Shared Configuration Multiple Gateways can reference the same `GatewayConfig`: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: GatewayConfig metadata: name: shared-config spec: extProc: kubernetes: env: - name: OTEL_SERVICE_NAME value: "ai-gateway-cluster" --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: gateway-1 annotations: aigateway.envoyproxy.io/gateway-config: shared-config spec: # ... --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: gateway-2 annotations: aigateway.envoyproxy.io/gateway-config: shared-config spec: # ... ``` ## Migration from Route-Level Configuration The route-level resource configuration (`AIGatewayRoute.spec.filterConfig.externalProcessor.resources`) is deprecated. Migrate to `GatewayConfig`: ### Before (deprecated) ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: my-route spec: filterConfig: type: ExternalProcessor externalProcessor: resources: requests: cpu: "100m" memory: "128Mi" ``` ### After (recommended) ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: GatewayConfig metadata: name: my-config spec: extProc: kubernetes: resources: requests: cpu: "100m" memory: "128Mi" --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: my-gateway annotations: aigateway.envoyproxy.io/gateway-config: my-config spec: # ... ``` ## Status The `GatewayConfig` status reports the validity of the configuration: ```yaml status: conditions: - type: Accepted status: "True" reason: Accepted message: "GatewayConfig reconciled successfully" ``` Possible condition types: - `Accepted`: The configuration is valid and applied - `NotAccepted`: The configuration has validation errors ## See Also - [Tracing](./observability/tracing.md) - Configure distributed tracing for AI Gateway - [Metrics](./observability/metrics.md) - Configure metrics collection - [Examples](https://github.com/theagentrouter/agent-router/tree/main/examples/gateway-config) - Example YAML files --- # Scaling the AI Gateway Controller Source: https://theagentrouter.ai/docs/capabilities/scaling The AI Gateway controller has two independent components with different scaling characteristics: 1. **Kubernetes controller** — reconciles CRDs and performs mutable operations against the Kubernetes API (requires leader election). 2. **Extension server** — a gRPC server that handles Envoy Gateway extension calls. It is read-only with respect to the Kubernetes API and scales horizontally. Because Envoy Gateway's xDS server communicates with all Envoy instances, and each EG replica in turn calls the AIGW extension server, the extension server can become a bottleneck under load. Running multiple AIGW controller replicas distributes this load across pods. **Note:** Leader election applies only to the Kubernetes controller portion. The extension server starts and serves traffic on every replica, including non-leader pods, so all replicas handle extension server requests from the moment they are ready. ## Recommended Configuration ### Helm Set `controller.replicaCount` and ensure `controller.leaderElection.enabled` is `true` (the default): ```yaml controller: replicaCount: 2 leaderElection: enabled: true ``` Apply with: ```shell helm upgrade --install ai-gateway-helm \ oci://docker.io/envoyproxy/ai-gateway-helm \ --namespace envoy-ai-gateway-system \ --values values.yaml ``` ### Kubernetes Deployment (without Helm) Edit the controller `Deployment` directly: ```shell kubectl -n envoy-ai-gateway-system scale deployment ai-gateway-controller \ --replicas=2 ``` Or patch the `Deployment` manifest: ```yaml spec: replicas: 2 ``` ## Horizontal Pod Autoscaler For dynamic workloads you can pair the above with an HPA. The extension server is CPU-bound, so CPU utilization is a reasonable metric: ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: ai-gateway-controller namespace: envoy-ai-gateway-system spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: ai-gateway-controller minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 ``` ## Resource Requests and Limits Size each replica according to the number of Envoy instances it will serve. A reasonable starting point: ```yaml controller: resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi" ``` Adjust based on observed usage in your environment. ## See Also - [Gateway Configuration](./gateway-config.md) — per-gateway ext_proc settings - [Observability](./observability/) — metrics to monitor controller load --- # Model Context Protocol (MCP) Gateway Source: https://theagentrouter.ai/docs/capabilities/mcp/ Agent Router provides first-class support for [Model Context Protocol](https://modelcontextprotocol.io/) (MCP), enabling AI agents to securely connect to external tools and data sources. This guide provides an overview of the MCP Gateway capabilities and how to configure routing to MCP servers using the `MCPRoute` API. ## Overview Agent Router's MCP support allows you to: - **Aggregate multiple MCP servers** into a single unified endpoint - **Apply security policies** including OAuth authentication, fine-grained access control over the tool access, and upstream API key injection - **Filter tools** to control which capabilities are exposed to clients - **Leverage Envoy's networking** for load balancing, rate limiting, circuit breaking, and observability The MCP Gateway acts as a transparent proxy between MCP clients (AI agents like Claude, Goose, etc.) and backend MCP servers, providing the same production-grade features available for LLM traffic. ## Key Features | Feature | Description | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Streamable HTTP Transport** | Full support for MCP's streamable HTTP transport, aligning with the [June 2025 MCP spec](https://modelcontextprotocol.io/specification/2025-06-18). Efficient handling of stateful sessions and multi-part JSON-RPC messaging over persistent HTTP connections. | | **Fine-Grained Authorization** | Native enforcement of [OAuth authentication flows](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization). Implement granular access control using JWT claims, scopes, and CEL expressions. | | **Server Multiplexing & Tool Routing** | Route tool calls to the right MCP backends, aggregating and filtering available tools based on gateway policy. Dynamically merge streaming notifications from multiple MCP servers into a unified interface. | | **Upstream Authentication** | Built-in upstream authentication primitives to securely connect to external MCP servers using API keys and header injection. | | **Full MCP Spec Coverage** | Complete [June 2025 MCP spec](https://modelcontextprotocol.io/specification/2025-06-18) compliance, including support for tool calls, notifications, prompts, resources, and bi-directional server-to-client requests. | | **Built-in Observability** | OpenTelemetry tracing and Prometheus metrics for all MCP requests, using the same observability stack as LLM traffic. | ## Architecture The MCP Gateway is implemented as a lightweight proxy component within the Agent Router sidecar, leveraging Envoy's battle-tested networking stack for all connection handling. ```mermaid sequenceDiagram participant Client as MCP Client
(AI Agent) participant Gateway as Agent Router participant MCP1 as MCP Server 1 participant MCP2 as MCP Server 2 Client->>Gateway: Initialize session Gateway->>MCP1: Initialize Gateway->>MCP2: Initialize MCP1-->>Gateway: Session ID 1 MCP2-->>Gateway: Session ID 2 Gateway-->>Client: Unified Session ID Client->>Gateway: List tools Gateway->>MCP1: List tools Gateway->>MCP2: List tools MCP1-->>Gateway: Tools A, B MCP2-->>Gateway: Tools X, Y Gateway-->>Client: Filtered & merged tools Client->>Gateway: Call tool (server1__toolA) Gateway->>MCP1: Call toolA MCP1-->>Gateway: Result Gateway-->>Client: Result ``` **Key architectural aspects:** - **Session Management**: The gateway creates unified sessions by encoding multiple backend session IDs, handling reconnection with `Last-Event-ID` support for SSE streams. - **Notification Handling**: Long-lived SSE streams from multiple MCP servers are merged into a single stream for clients, with proper event ID reconstruction. - **Request Routing**: Tool names are automatically prefixed with the backend name (e.g., `github__issue_read`) to route calls to the correct upstream server. For detailed architecture and design decisions, see the [MCP Gateway proposal](https://github.com/theagentrouter/agent-router/tree/main/docs/proposals/006-mcp-gateway). ## Trying it out Before you begin, you'll need to complete the basic setup from the [Basic Usage](https://theagentrouter.ai/docs/getting-started/basic-usage) guide, which includes installing Envoy Gateway and AI Gateway. ### Basic MCPRoute Configuration The following example demonstrates a basic `MCPRoute` that proxies the GitHub MCP server: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: MCPRoute metadata: name: mcp-route namespace: default spec: parentRefs: - name: aigw-run kind: Gateway group: gateway.networking.k8s.io path: "/mcp" # Clients connect to http://gateway-address/mcp backendRefs: - name: github kind: Backend group: gateway.envoyproxy.io path: "/mcp/x/issues/readonly" securityPolicy: apiKey: secretRef: name: github-token --- apiVersion: gateway.envoyproxy.io/v1alpha1 kind: Backend metadata: name: github namespace: default spec: endpoints: - fqdn: hostname: api.githubcopilot.com port: 443 --- apiVersion: v1 kind: Secret metadata: name: github-token namespace: default type: Opaque stringData: apiKey: ghp_your_token_here ``` Apply this configuration: ```shell kubectl apply -f mcp-route.yaml ``` Now clients can connect to `http:///mcp` and access GitHub tools. ### Tool Filtering Control which tools are exposed using the `toolSelector` field. You can use exact matches or regular expressions: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: MCPRoute metadata: name: mcp-route namespace: default spec: parentRefs: - name: aigw-run kind: Gateway group: gateway.networking.k8s.io backendRefs: # GitHub: only expose issue-related tools - name: github kind: Backend group: gateway.envoyproxy.io path: "/mcp/x/issues/readonly" toolSelector: includeRegex: - .*issues?.* # Matches issue_read, list_issues, etc. securityPolicy: apiKey: secretRef: name: github-token # Context7: expose specific tools by exact name - name: context7 kind: Backend group: gateway.envoyproxy.io path: "/mcp" toolSelector: include: - resolve-library-id - query-docs ``` **Note:** The `toolSelector` field requires exactly one of `include` or `includeRegex` to be specified. If not specified, all tools from the MCP server are exposed. ### Server Multiplexing The gateway automatically aggregates tools from multiple MCP servers into a single unified interface: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: MCPRoute metadata: name: mcp-unified namespace: default spec: parentRefs: - name: aigw-run kind: Gateway group: gateway.networking.k8s.io path: "/mcp" backendRefs: - name: github kind: Backend group: gateway.envoyproxy.io path: "/mcp/x/issues/readonly" securityPolicy: apiKey: secretRef: name: github-token - name: context7 kind: Backend group: gateway.envoyproxy.io path: "/mcp" ``` Clients will see all tools with prefixed names: - `github__issue_read` - `github__list_issues` - `context7__resolve-library-id` - `context7__query-docs` ### Header Forwarding Forward HTTP headers from the client request to specific backend MCP servers. This enables per-user authentication passthrough (e.g., personal access tokens) without requiring OAuth: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: MCPRoute metadata: name: mcp-unified namespace: default spec: parentRefs: - name: aigw-run kind: Gateway group: gateway.networking.k8s.io path: "/mcp" backendRefs: - name: atlassian kind: Backend group: gateway.envoyproxy.io path: "/mcp" forwardHeaders: - name: X-Atlassian-Jira-Personal-Token - name: X-Atlassian-Jira-Url - name: Authorization backendHeader: X-Original-Auth # optional: rename the header - name: github kind: Backend group: gateway.envoyproxy.io path: "/mcp" securityPolicy: apiKey: secretRef: name: github-token ``` Each `forwardHeaders` entry specifies: - `name` (required): The header to extract from the incoming client request. - `backendHeader` (optional): A different header name to use when forwarding to the backend. If omitted, the original header name is used. Header **values** are forwarded verbatim. The gateway does not add an auth scheme such as `Bearer `. Headers are scoped per-backend — during fan-out operations like `tools/list`, only the backends with explicit `forwardHeaders` configuration receive the forwarded headers. Other backends in the same route are unaffected. To keep a default least-privilege service-account token on a backend while letting callers override it with a personal access token, set `securityPolicy.apiKey.injectionPolicy: IfNotPresent` and map the client token onto the same header with `forwardHeaders`. When the client omits that header, the gateway injects the configured API key (and prefixes it with `Bearer ` when the target is `Authorization`). When the client sends it, the forwarded value is preserved as-is. The default is `injectionPolicy: Always`, which always injects the configured credential. ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: MCPRoute metadata: name: mcp-unified namespace: default spec: parentRefs: - name: aigw-run kind: Gateway group: gateway.networking.k8s.io backendRefs: - name: github kind: Backend group: gateway.envoyproxy.io securityPolicy: apiKey: secretRef: name: github-sa-token # default least-privilege service account injectionPolicy: IfNotPresent forwardHeaders: - name: X-GitHub-PAT backendHeader: Authorization ``` For that example, send the full header value the backend expects, including the scheme: ``` X-GitHub-PAT: Bearer ghp_... ``` A raw token (`X-GitHub-PAT: ghp_...`) is copied onto `Authorization` unchanged, so the backend typically rejects it. Omit `X-GitHub-PAT` to use the injected service-account key instead. `injectionPolicy` applies only to header injection. Do not combine `injectionPolicy: IfNotPresent` with `queryParam`. If the MCPRoute itself uses OAuth or API-key client authentication, do not forward inbound `Authorization` (that is the gateway token). Use a dedicated client header and `backendHeader` to map it onto the backend credential header. ### OAuth Authentication Protect your MCP Gateway with OAuth authentication following the [MCP Authorization specification](https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization): ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: MCPRoute metadata: name: mcp-route namespace: default spec: parentRefs: - name: aigw-run kind: Gateway group: gateway.networking.k8s.io backendRefs: - name: github kind: Backend group: gateway.envoyproxy.io path: "/mcp/readonly" securityPolicy: oauth: issuer: "https://keycloak.example.com/realms/master" audiences: - "https://api.example.com/mcp" protectedResourceMetadata: resource: "https://api.example.com/mcp" scopesSupported: - "profile" - "email" ``` #### Authorization servers with non-standard metadata locations By default the gateway discovers the authorization server's endpoints by probing the well-known URIs derived from `issuer`, as described in [RFC 8414](https://datatracker.ietf.org/doc/html/rfc8414#section-3). Some authorization servers publish their metadata somewhere the issuer does not lead to, for example at a versioned path. Point `authorizationServerMetadataUrl` at the document in that case: ```yaml securityPolicy: oauth: issuer: "https://example.com/api/idp/authn" authorizationServerMetadataUrl: "https://example.com/api/idp/v4/authn/.well-known/openid-configuration" ``` `issuer` is unaffected by this field: it still identifies the authorization server in the protected resource metadata the gateway publishes. When `jwks` is not set, the JWKS URI is also discovered from the document fetched here. If that document cannot be fetched, the MCPRoute is not accepted and the reason appears in its status conditions. The OAuth flow follows the MCP specification's authorization code flow with PKCE: ```mermaid sequenceDiagram participant C as MCP Client participant G as MCP Gateway participant A as Authorization Server C->>G: MCP request without token G->>C: HTTP 401 with WWW-Authenticate header C->>G: Request Protected Resource Metadata G->>C: Return OAuth metadata C->>A: GET /.well-known/oauth-authorization-server A->>C: Authorization server metadata Note over C: Generate PKCE parameters C->>A: Authorization request + PKCE Note over A: User authorizes A->>C: Authorization code C->>A: Token request + code_verifier A->>C: Access token C->>G: MCP request with access token G->>G: Verify token (JWKS) Note over G: Token validated G->>C: MCP response ``` ### Authorization Policies Agent Router supports fine-grained access control over tool access using a combination of: - **JWT Scopes & Claims**: Validate standard OAuth2 scopes and custom claims - **Tool Selection**: Restrict access to specific tools - **CEL Expressions**: Flexible, advanced matching using Common Expression Language (CEL) #### Configuration Structure Authorization is configured in the `MCPRoute` resource under `spec.securityPolicy.authorization`. ```yaml spec: securityPolicy: authorization: rules: - source: jwt: scopes: ["read"] target: tools: - backend: "github" tool: "list_issues" ``` #### Rule Evaluation Rules are evaluated in order. The first rule that matches the request (Target, Source, and CEL) determines the action (Allow/Deny). If no rules match, the `defaultAction` is applied. #### Matchers | Matcher | Description | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Target** | Matches specific tools. Can filter by `backend` and `tool` name. | | **Source** | Matches JWT properties. `scopes`: List of required scopes (all must be present). `claims`: Key-value pairs. Arrays in claims match if _any_ value matches. | | **CEL** | Advanced expression evaluated against the request context. | #### CEL Context The following variables are available in CEL expressions: | Variable | Description | | --------------------- | ---------------------------------------------- | | `request.method` | HTTP method (e.g., "POST") | | `request.host` | Host header value | | `request.path` | URL path | | `request.headers` | Map of headers (lowercased keys, single value) | | `request.auth.jwt` | Parsed JWT `{claims: ..., scopes: [...]}` | | `request.mcp.method` | MCP JSON-RPC method (e.g., "tools/call") | | `request.mcp.backend` | Target backend name | | `request.mcp.tool` | Target tool name (for tool calls) | | `request.mcp.params` | Parsed JSON-RPC parameters | #### Examples **Comprehensive Policy Example** This example demonstrates various matching strategies including token scopes, claims, tool targeting, and CEL expressions. ```yaml authorization: rules: - source: jwt: scopes: - echo target: tools: - backend: mcp-backend tool: echo cel: request.mcp.params.arguments.text.matches("^Hello, .*!$") && request.headers["x-tenant-id"] == "t-123" - source: jwt: scopes: - sum claims: - name: tenant valueType: String values: - acme - globex - name: org.departments valueType: StringArray values: - engineering - development target: tools: - backend: mcp-backend tool: sum ``` ## See Also - [MCP Gateway Proposal](https://github.com/theagentrouter/agent-router/tree/main/docs/proposals/006-mcp-gateway) - Detailed architecture and design decisions - [MCP Specification](https://modelcontextprotocol.io/specification/2025-06-18) - Official Model Context Protocol documentation - [MCP Example](https://github.com/theagentrouter/agent-router/tree/main/examples/mcp) - Complete working example - [CLI MCP Configuration](https://theagentrouter.ai/docs/cli/aigwrun#mcp-configuration) - Using MCP with `aigw run` standalone mode --- # Agent2Agent (A2A) Traffic Source: https://theagentrouter.ai/docs/capabilities/a2a/ **Preview:** Agent Router does not yet have a dedicated A2A route type. This page shows how to route [A2A](https://a2a-protocol.org/) agents through the gateway **today**, using only Envoy Gateway resources plus the `envoy.filters.http.a2a` filter that ships inside Envoy Proxy. The filter is alpha in Envoy and the recipe relies on `EnvoyPatchPolicy`, so treat this as a preview. A first-class `A2ARoute` API is being discussed in [issue #2070](https://github.com/theagentrouter/agent-router/issues/2070). **Help shape A2A support:** Early adopters of A2A can already configure Envoy through Agent Router to handle A2A traffic with the approach on this page. The community is working on first-class A2A features and CRDs. If you want to help, or just want to tell us what your agents need, join the discussion on [GitHub](https://github.com/theagentrouter/agent-router/issues/2070), the [community Discord](https://discord.gg/xuxtPq43gZ), or the [weekly community meeting](https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w) held every Monday. ## What you get Following this page, an A2A agent behind the gateway gets: - **Routing** for the agent card (`GET /.well-known/agent-card.json`) and the JSON-RPC endpoint, with hostnames, timeouts long enough for Server-Sent Events, and optional path rewriting. - **Protocol validation** by Envoy's A2A filter: every POST on the route must be a JSON-RPC 2.0 envelope, and oversized bodies are rejected. - **A2A fields in access logs and metrics**: the JSON-RPC `method` and `id` (and, for A2A 0.3 clients, task and context IDs) land in Envoy dynamic metadata; the filter also exposes `a2a.requests_rejected`, `a2a.invalid_json` and `a2a.body_too_large` counters. - **Client authentication** (JWT, API key, external auth) on the RPC endpoint via `SecurityPolicy`, while the public card stays open. - Optionally, a **gateway-served agent card** so clients discover the gateway URL rather than the agent's internal address. What you do **not** get yet: aggregation of several agents behind one card, rewriting of the backend's own card, method-level authorization, or support for the gRPC and HTTP+JSON bindings. See [Limitations](#limitations). ## Prerequisites - Agent Router installed following [Prerequisites](../../getting-started/prerequisites.md). The Envoy Gateway values file used there already sets `extensionApis.enableEnvoyPatchPolicy: true`. - Envoy Gateway v1.8 or newer. The A2A filter exists since Envoy Proxy v1.38.0, which Envoy Gateway v1.8 is the first release to ship; see the [compatibility matrix](https://gateway.envoyproxy.io/news/releases/matrix/) for later pairings. On an older proxy image the patched listener is rejected by Envoy, which takes every route on that listener down, so check the image before you start. - An A2A agent speaking the JSON-RPC binding. The example below assumes it runs as `Service/research-agent` on port 8080 in namespace `default`. **Warning:** `EnvoyPatchPolicy` lets anyone with permission to create it inject arbitrary proxy configuration. Restrict it with RBAC as described in the [Envoy Gateway security note](https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/#security-warning). The A2A filter itself is marked alpha with an unknown security posture in Envoy: use it only where both clients and agents are trusted. ## Step 1: Give A2A agents their own Gateway The A2A filter is applied to a whole Envoy listener. Once enabled, every JSON `POST` on that listener is buffered and parsed as JSON-RPC, so an LLM route sharing the listener would have its chat-completion bodies parsed too and could hit the body size limit. Keep A2A traffic on a dedicated Gateway (or at least a dedicated listener): ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: a2a-gateway namespace: default spec: gatewayClassName: envoy-ai-gateway-basic listeners: - name: http protocol: HTTP port: 80 ``` If you must share a listener with other traffic, see [Advanced: enabling the filter per route](#advanced-enabling-the-filter-per-route). ## Step 2: Route the agent card and the RPC endpoint One `HTTPRoute` with two **named** rules. The names matter: `SecurityPolicy` targets the `rpc` rule in Step 4 so the card stays public. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: research-agent namespace: default spec: parentRefs: - name: a2a-gateway hostnames: - research.agents.example.com rules: # A2A discovery: the card is host-rooted, so one agent per hostname. - name: agent-card matches: - method: GET path: type: Exact value: /.well-known/agent-card.json backendRefs: - name: research-agent port: 8080 # JSON-RPC endpoint. Streaming methods return text/event-stream and can run # for minutes, so raise the request timeouts (Envoy's default is 15s). - name: rpc matches: - method: POST path: type: Exact value: /a2a timeouts: request: 30m backendRequest: 30m backendRefs: - name: research-agent port: 8080 ``` If the agent serves its RPC endpoint on a different path, add a `URLRewrite` filter to the `rpc` rule. Do not attach request buffering (`BackendTrafficPolicy.requestBuffer`) to these routes; it stalls streaming responses. ## Step 3: Enable Envoy's A2A filter `EnvoyPatchPolicy` inserts the filter into the listener's HTTP filter chain immediately before the router, so it runs after any authentication filters added by `SecurityPolicy`: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: EnvoyPatchPolicy metadata: name: a2a-filter namespace: default spec: targetRef: group: gateway.networking.k8s.io kind: Gateway name: a2a-gateway type: JSONPatch jsonPatches: - type: type.googleapis.com/envoy.config.listener.v3.Listener # //. With the XDSNameSchemeV2 # runtime flag enabled in Envoy Gateway this becomes "tcp-80". name: default/a2a-gateway/http operation: op: add # Selects the router filter; "add" at an array index inserts before it. jsonPath: "$..http_filters[?(@.name=='envoy.filters.http.router')]" value: name: envoy.filters.http.a2a typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.a2a.v3.A2a # PASS_THROUGH forwards non-A2A requests untouched. # REJECT answers them with 400; use it only on listeners that carry # nothing but JSON-RPC A2A traffic (see Limitations). traffic_mode: PASS_THROUGH # Bytes buffered while parsing the JSON-RPC envelope. Envoy's default # is 8192, which is too small for real messages; the maximum is 10485760. # Requests that exceed it get a 413 in both traffic modes. max_request_body_size: 1048576 ``` Check that the patch was applied: ```shell kubectl get envoypatchpolicy a2a-filter -o yaml | grep -A3 'type: Programmed' ``` Both `Accepted` and `Programmed` must be `True`. If the Gateway was created with `mergeGateways`, target the `GatewayClass` instead of the `Gateway`, as in the [Envoy Gateway task](https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/). ## Step 4: Protect the RPC endpoint, keep the card public The card must be fetchable without credentials; the RPC endpoint usually must not. Target the `rpc` rule by `sectionName`: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: SecurityPolicy metadata: name: research-agent-rpc namespace: default spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: research-agent sectionName: rpc jwt: providers: - name: corp-idp issuer: https://idp.example.com audiences: - research-agent remoteJWKS: uri: https://idp.example.com/.well-known/jwks.json ``` `apiKeyAuth` and `extAuth` work the same way. Advertise the matching scheme in the agent card's `securitySchemes` so clients know what to send. ## Step 5: Log A2A fields The filter writes its findings as dynamic metadata under the filter name. Add them to the access log in your `EnvoyProxy` resource, alongside the MCP fields from [Access Logs](../observability/accesslogs.md): ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: EnvoyProxy metadata: name: a2a-access-logs namespace: default spec: telemetry: accessLog: settings: - sinks: - type: File file: path: /dev/stdout format: type: JSON json: a2a.method: "%DYNAMIC_METADATA(envoy.filters.http.a2a:method)%" jsonrpc.request.id: "%DYNAMIC_METADATA(envoy.filters.http.a2a:id)%" a2a.version: "%REQ(A2A-VERSION)%" # Populated for A2A 0.3 clients only; see Limitations. a2a.task.id: "%DYNAMIC_METADATA(envoy.filters.http.a2a:params:taskId)%" a2a.context.id: "%DYNAMIC_METADATA(envoy.filters.http.a2a:params:message:contextId)%" # Common fields start_time: "%START_TIME%" method: "%REQ(:METHOD)%" path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" response_code: "%RESPONSE_CODE%" response_code_details: "%RESPONSE_CODE_DETAILS%" duration: "%DURATION%" upstream_cluster: "%UPSTREAM_CLUSTER%" route_name: "%ROUTE_NAME%" ``` Attach it to the Gateway through `spec.infrastructure.parametersRef`. Do not log the whole `envoy.filters.http.a2a` namespace: for 0.3 clients it contains the full message content. Rejections show up in `response_code_details` as `a2a_filter_reject`, `a2a_filter_not_valid_jsonrpc`, `a2a_filter_parse_error` or `a2a_filter_body_too_large`, and in the proxy stats as `http.http-.a2a.requests_rejected`, `.invalid_json` and `.body_too_large` (for example `http.http-10080.a2a.body_too_large`). ## Step 6: Verify ```shell export GATEWAY_HOST=$(kubectl get gateway/a2a-gateway -o jsonpath='{.status.addresses[0].value}') # Discovery curl -s -H "Host: research.agents.example.com" \ http://$GATEWAY_HOST/.well-known/agent-card.json | jq .name # A2A 1.0 SendMessage (add the Authorization header if you applied Step 4) curl -s -H "Host: research.agents.example.com" \ -H "Content-Type: application/a2a+json" -H "A2A-Version: 1.0" \ -d '{"jsonrpc":"2.0","id":"1","method":"SendMessage","params":{"message":{"messageId":"m1","role":"ROLE_USER","parts":[{"text":"hello"}]}}}' \ http://$GATEWAY_HOST/a2a | jq . # Protocol validation: a JSON body that is not JSON-RPC is forwarded in PASS_THROUGH # mode and rejected with 400 in REJECT mode. curl -s -o /dev/null -w '%{http_code}\n' -H "Host: research.agents.example.com" \ -H "Content-Type: application/json" -d '{"hello":"world"}' http://$GATEWAY_HOST/a2a ``` The access log line for the second request carries `"a2a.method": "SendMessage"`. `kubectl port-forward` is fine for these one-shot calls, but it drops long-lived streams and any connection Envoy resets (for example after a 413). Test `SendStreamingMessage` from a pod inside the cluster or through a real load balancer; through the gateway a stream that the agent keeps open for 18 seconds arrives intact. ## Optional: serve the agent card from the gateway A2A clients connect to whatever URL the card advertises. If the agent only knows its in-cluster address, its card sends clients around the gateway. Rather than rewriting the backend's card, serve a static one from the gateway with Envoy Gateway's direct response filter and point the `agent-card` rule at it: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: HTTPRouteFilter metadata: name: research-agent-card namespace: default spec: directResponse: statusCode: 200 contentType: application/json body: type: Inline inline: | { "name": "Research Agent", "description": "Summarises research papers", "supportedInterfaces": [ {"url": "https://research.agents.example.com/a2a", "protocolBinding": "JSONRPC", "protocolVersion": "1.0"} ], "capabilities": {"streaming": true}, "defaultInputModes": ["text/plain"], "defaultOutputModes": ["text/plain"], "securitySchemes": {"bearer": {"type": "http", "scheme": "bearer"}}, "skills": [] } ``` ```yaml - name: agent-card matches: - method: GET path: type: Exact value: /.well-known/agent-card.json filters: - type: ExtensionRef extensionRef: group: gateway.envoyproxy.io kind: HTTPRouteFilter name: research-agent-card ``` ## Advanced: enabling the filter per route If the A2A route must share a listener with other traffic, insert the filter **disabled** and enable it only on the A2A routes. This is the same mechanism Agent Router uses for its own filters. 1. In the listener patch above, add `disabled: true` next to `typed_config`. 2. Add a second patch against the route configuration. Envoy Gateway names xDS routes `httproute///rule//match//`, so a regular expression on the route name selects the rules you want. Patch the filter's own key under `typed_per_filter_config`: Envoy Gateway creates the map when the route has none and leaves other entries (for example the `SecurityPolicy` API-key config) untouched. ```yaml - type: type.googleapis.com/envoy.config.route.v3.RouteConfiguration name: default/a2a-gateway/http operation: op: add # Only the rpc rule needs the filter; the card is a bodyless GET anyway. jsonPath: "$..routes[?(@.name =~ 'httproute/default/research-agent/rule/1/')]" path: /typed_per_filter_config/envoy.filters.http.a2a value: "@type": type.googleapis.com/envoy.config.route.v3.FilterConfig # An empty config enables a filter that is disabled at the listener. config: {} ``` With this in place the filter parses bodies on the `rpc` route only: a 1.5 MiB non-A2A body sent to another `HTTPRoute` on the same listener is forwarded, while the same body on `/a2a` gets a 413. Rule indexes follow the order of `rules` in the `HTTPRoute`, so keep the patch and the route in the same file. Confirm the result with `kubectl get envoypatchpolicy` (both conditions `True`) and the Envoy admin `config_dump`. This bookkeeping is exactly what a native `A2ARoute` will remove. ## Keeping up with Envoy The `EnvoyPatchPolicy` above is plain Envoy configuration, so as the A2A filter grows you can amend the `typed_config` yourself without waiting for an Agent Router release. Check these sources against the Envoy version your gateway actually runs: - [A2A filter configuration overview](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/a2a_filter) and the [A2A proto API reference](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/a2a/v3/a2a.proto), which lists every field the filter accepts. Replace `latest` in the URL with your proxy's version tag, for example `v1.38.0`, to see exactly what your proxy supports. - [Envoy release notes](https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history) for `a2a` entries in new releases, and the [filter source](https://github.com/envoyproxy/envoy/tree/main/source/extensions/filters/http/a2a) for behaviour that has not been documented yet. - [Envoy Gateway compatibility matrix](https://gateway.envoyproxy.io/news/releases/matrix/) to see which Envoy Proxy version each Envoy Gateway release ships, and the [EnvoyPatchPolicy task](https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/) for the xDS resource names the patch must target, which change when the `XDSNameSchemeV2` runtime flag is on. To find the running version, read the image tag of the proxy pod: ```shell kubectl get pods -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=a2a-gateway \ -o jsonpath='{.items[0].spec.containers[?(@.name=="envoy")].image}{"\n"}' ``` To try a newer Envoy Proxy before Envoy Gateway ships it, override the proxy image in the `EnvoyProxy` resource as described in [Customize EnvoyProxy](https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/). Keep in mind that the A2A filter is alpha, so its config and behaviour can change between releases; re-read the release notes before bumping. ## Limitations These reflect the filter as it ships today and will shrink as Envoy and Agent Router evolve. Check [Keeping up with Envoy](#keeping-up-with-envoy) for what your proxy version supports. - **A2A 1.0 method names are only partly modelled.** The filter validates every request and logs `method` and `id`, but `taskId` and `contextId` extraction currently keys on the A2A 0.3 names (`message/send`, `tasks/get`, ...). - **`parser_config` and `storage_mode` are accepted but not yet wired up.** Leave them unset until a release note says otherwise. - **`max_request_body_size` applies in both modes.** A JSON-RPC envelope that is not complete within the limit gets a 413 even in `PASS_THROUGH`. - **`REJECT` suits JSON-RPC-only listeners.** Route the gRPC and HTTP+JSON bindings on a listener without the filter. - **Agent cards are served as-is, one per hostname.** URL rewriting, aggregation and `aigw run` support are part of the first-class A2A work in [issue #2070](https://github.com/theagentrouter/agent-router/issues/2070). --- # Inference Optimization Source: https://theagentrouter.ai/docs/capabilities/inference/ Agent Router offers smart ways to improve the speed and reliability of your AI/LLM tasks. This section explains how it uses intelligent routing and load balancing to manage inference requests across different backend endpoints efficiently. ![](https://theagentrouter.ai/img/resource-model.png) ## Overview The inference optimization capabilities in Agent Router enable: - **Intelligent Endpoint Selection**: Automatically route requests to the most suitable inference endpoints based on real-time metrics and availability - **Dynamic Load Balancing**: Distribute inference workloads across multiple backend instances for optimal resource utilization - **Seamless Integration**: Work with both standard HTTPRoute and AI Gateway's enhanced AIGatewayRoute configurations - **Extensible Architecture**: Support for custom endpoint picker providers (EPP) to implement domain-specific routing logic ## Getting Started To get started with InferencePool support in Agent Router: 1. **[Learn about InferencePool Support](./inferencepool-support.md)**: Understand the core concepts and benefits 2. **[Try HTTPRoute + InferencePool](./httproute-inferencepool.md)**: Start with basic inference routing 3. **[Explore AIGatewayRoute + InferencePool](./aigatewayroute-inferencepool.md)**: Leverage advanced AI-specific features --- # InferencePool Support Source: https://theagentrouter.ai/docs/capabilities/inference/inferencepool-support InferencePool is a powerful feature that enables intelligent routing and load balancing for AI inference workloads. It integrates with the [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) to provide dynamic endpoint selection based on real-time metrics and availability. Real-time metrics refer to data points such as KVCache Usage, Queued Inference Requests, LoRA Adapters info and etc, which are continuously monitored to make informed routing decisions. ![](https://theagentrouter.ai/img/request-flow.png) ## What is InferencePool? InferencePool is a Kubernetes custom resource that defines a pool of inference endpoints and specifies how requests should be routed among them. It solves several key challenges in AI inference deployments: - **Dynamic Load Balancing**: Automatically distribute requests across multiple inference instances - **Intelligent Routing**: Route requests to the most suitable endpoints based on current load, latency, and more self-defined routing algorithms. - **Automatic Failover**: Handle endpoint failures gracefully with built-in retry mechanisms - **Resource Optimization**: Maximize utilization of inference compute resources ## Core Problems Solved Traditional load balancing treats all backends equally, but inference workloads have unique characteristics: - Variable processing times based on model complexity and input size - Different resource requirements for different models - Need for intelligent routing based on real-time metrics ## Request Flow InferencePool supports two integration modes with different request flows: ### AIGatewayRoute + InferencePool Flow ```mermaid sequenceDiagram participant Client as Client (OpenAI SDK) participant Envoy as Envoy Proxy participant RLS as Rate Limit Service participant Processor as AI Gateway External Processor participant EPP as Endpoint Picker Provider participant Backend as Inference Backend Client->>Envoy: Request with model header Envoy->>RLS: Check Rate Limit RLS-->>Envoy: Rate limit OK Envoy->>Processor: Router-level ExtProc Request Note over Processor: Extract Model Name Processor-->>Envoy: ClearRouteCache Envoy->>EPP: Router-level ExtProc Request Note over EPP: Pick Optimal Endpoint EPP-->>Envoy: Add x-gateway-destination-endpoint header loop Retry/Fallback loop Note over Envoy: Forward to Selected Endpoint Envoy->>Processor: Upstream-level ExtProc Request Note over Processor: Request Transform Processor-->>Envoy: Transformed Request Envoy->>Backend: Forward Request Backend-->>Envoy: Response end Envoy->>Processor: Process Response Note over Processor: Extract Token Usage & Metrics Processor-->>Envoy: Add Usage Metadata Envoy->>RLS: Update Rate Limit Budget RLS-->>Envoy: Budget Updated Envoy->>Client: Response ``` ### HTTPRoute + InferencePool Flow ```mermaid sequenceDiagram participant Client as Client participant Envoy as Envoy Proxy participant EPP as Endpoint Picker Provider participant Backend as Inference Backend Client->>Envoy: Request Envoy->>EPP: Router-level ExtProc Request Note over EPP: Pick Optimal Endpoint EPP-->>Envoy: Add x-gateway-destination-endpoint header loop Retry/Fallback loop Note over Envoy: Forward to Selected Endpoint Envoy->>Backend: Forward Request Backend-->>Envoy: Response end Envoy->>Client: Response ``` ## Integration Approaches Agent Router supports two ways to use InferencePool: ### 1. HTTPRoute + InferencePool Direct integration with standard Gateway API HTTPRoute for simple inference routing scenarios. **Use Cases:** - Simple inference workloads without complex AI-specific requirements - Direct OpenAI-compatible API forwarding ### 2. AIGatewayRoute + InferencePool Enhanced integration with AI Gateway's custom route type for advanced AI-specific features. **Use Cases:** - Multi-model routing based on request content - Token-based rate limiting - Advanced observability and metrics - Request/response transformation ## AIGatewayRoute Advantages AIGatewayRoute provides several advantages over standard HTTPRoute when used with InferencePool: ### Advanced OpenAI Routing - Built-in OpenAI API schema validation - Seamless integration with OpenAI SDKs - Route multiple models in a single listener - Mix InferencePool and traditional backends - Automatic model extraction from request body ### AI-Specific Features - Token-based rate limiting - Model performance metrics - Cost tracking and management - Request/response transformation ## Getting Started To use InferencePool with Agent Router, you'll need to: 1. **Install Prerequisites**: - Deploy Gateway API Inference Extension CRD. - Configure Envoy Gateway with the InferencePool support. See [Envoy Gateway Installation Guide](../../getting-started/prerequisites.md#additional-features-rate-limiting-inferencepool-etc) for details. 2. **Configure InferencePool**: Define your inference endpoints and selection criteria 3. **Set up Routes**: Configure either HTTPRoute or AIGatewayRoute to use the InferencePool 4. **Deploy Endpoint Picker**: Install and configure the endpoint picker provider The following sections provide detailed guides for both integration approaches with complete examples and step-by-step instructions. --- # HTTPRoute + InferencePool Guide Source: https://theagentrouter.ai/docs/capabilities/inference/httproute-inferencepool This guide shows how to use InferencePool with the standard Gateway API HTTPRoute for intelligent inference routing. This approach provides basic load balancing and endpoint selection capabilities for inference workloads. ![](https://theagentrouter.ai/img/inference-httproute.svg) ## Step 4: Configure Gateway and HTTPRoute Create a Gateway and HTTPRoute that uses the InferencePool: ```yaml cat </... # endpointConfig: # rootPrefix: "/" ``` Or with helm CLI: ```bash helm upgrade --install ai-gateway envoyproxy/ai-gateway-helm \ -n envoy-ai-gateway-system --create-namespace \ --set 'endpointConfig.openai=/' \ --set 'endpointConfig.cohere=/cohere' \ --set 'endpointConfig.anthropic=/anthropic' \ --set 'endpointConfig.typesafe=/typesafe' ``` Notes: - `endpointConfig.rootPrefix` (default `/`) is prepended to all provider prefixes. - Only these keys are accepted: `openai`, `cohere`, `anthropic`, `typesafe`. - If any key is omitted or empty, defaults are applied as listed above. ## What's Next To learn more about configuring and using the Agent Router with these endpoints: - **[Supported Providers](./supported-providers.md)** - Complete list of supported AI providers and their configurations - **[Usage-Based Rate Limiting](../traffic/usage-based-ratelimiting.md)** - Configure token-based rate limiting and cost controls - **[Provider Fallback](../traffic/provider-fallback.md)** - Set up automatic failover between providers for high availability - **[Metrics and Monitoring](../observability/metrics.md)** - Monitor usage, costs, and performance metrics [issue#609]: https://github.com/theagentrouter/agent-router/issues/609 --- # Connecting to AI Providers Source: https://theagentrouter.ai/docs/capabilities/llm-integrations/connect-providers Agent Router provides a unified interface for connecting to multiple AI providers through a standardized configuration approach. This page explains the fundamental concepts, resources, and relationships required to establish connectivity with any supported AI provider. ## Overview Establishing connectivity with an AI provider involves configuring three key Kubernetes resources that work together to enable secure, scalable access to AI services: 1. **AIServiceBackend** - Defines the backend service and its API schema 2. **BackendSecurityPolicy** - Configures authentication credentials 3. **AIGatewayRoute** - Routes client requests to the appropriate backends These resources provide a consistent configuration model regardless of which AI provider you're connecting to, whether it's OpenAI, AWS Bedrock, Azure OpenAI, or any other [supported provider](./supported-providers). ## Core Resources for Provider Connectivity ### AIServiceBackend The `AIServiceBackend` resource represents an individual AI service backend and serves as the bridge between your gateway and the AI provider's API. #### Purpose and Configuration - **API Schema Definition**: Specifies which API format the backend expects (OpenAI v1, AWS Bedrock, Azure OpenAI, etc.) - **Backend Reference**: Points to the Envoy Gateway Backend resource - **Security Integration**: Links to authentication policies for upstream services #### Key Fields ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIServiceBackend metadata: name: my-provider-backend spec: # API schema the backend expects schema: name: OpenAI # Provider API format version: "v1" # API version (optional for OpenAI, defaults to "v1") # Reference to the Envoy Gateway Backend resource backendRef: name: my-provider-backend kind: Backend group: gateway.envoyproxy.io ``` #### Schema Configuration Examples Different providers require different schema configurations: | Provider | Schema Configuration | | -------------------------- | --------------------------------------------------------- | | OpenAI | `{"name":"OpenAI","version":"v1"}` | | AWS Bedrock | `{"name":"AWSBedrock"}` | | Azure OpenAI | `{"name":"AzureOpenAI","version":"2025-01-01-preview"}` | | GCP Vertex AI | `{"name":"GCPVertexAI"}` | | GCP Anthropic on Vertex AI | `{"name":"GCPAnthropic", "version": "vertex-2023-10-16"}` | **Tip:** Many providers offer OpenAI-compatible APIs, which allows them to use the OpenAI schema configuration with provider-specific version paths. ### BackendSecurityPolicy The `BackendSecurityPolicy` resource configures authentication credentials needed to access upstream AI services securely. #### Purpose and Configuration - **Credential Management**: Stores API keys, cloud credentials, or other authentication mechanisms - **Security Isolation**: Keeps sensitive credentials separate from routing configuration - **Provider Flexibility**: Supports multiple authentication types for different providers #### Authentication Types ##### API Key Authentication Commonly used across many providers ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: openai-auth spec: type: APIKey apiKey: secretRef: name: openai-secret ``` **Note:** The secret must contain the API key with the key name `"apiKey"`. ##### AWS Credentials Used when connecting to AWS Bedrock. Supports three authentication methods: **Option 1: EKS Pod Identity or IRSA (Recommended for production)** When running on EKS, the AWS SDK automatically uses the default credential chain, which includes EKS Pod Identity and IRSA. Simply configure the region: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: bedrock-auth spec: type: AWSCredentials awsCredentials: region: us-east-1 # No credentialsFile needed - automatically uses: # - EKS Pod Identity (if Pod Identity association exists) # - IRSA (if ServiceAccount has eks.amazonaws.com/role-arn annotation) ``` See the [Connect AWS Bedrock guide](../../getting-started/connect-providers/aws-bedrock.md) for detailed setup instructions. **Option 2: Static Credentials (Development/Testing)** ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: bedrock-auth spec: type: AWSCredentials awsCredentials: region: us-east-1 credentialsFile: secretRef: name: aws-secret profile: default # Optional, defaults to "default" ``` **Note:** When using static credentials, the secret must contain the AWS credentials file with the key name `"credentials"`. ##### Azure Credentials Used for connecting to Azure OpenAI ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: azure-auth spec: type: AzureCredentials azureCredentials: clientID: "your-azure-client-id" tenantID: "your-azure-tenant-id" clientSecretRef: name: azure-secret ``` **Note:** The secret must contain the Azure client secret with the key name `"client-secret"`. ##### GCP Credentials Used for connecting to GCP Vertex AI and Anthropic on GCP. Supports three authentication methods: 1. Application Default Credentials (Recommended for GKE): When running on GKE with Workload Identity configured, you can use Application Default Credentials (ADC) without managing service account keys. Simply specify the project and region: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: gcp-auth-adc namespace: default spec: type: GCPCredentials gcpCredentials: projectName: "your-gcp-project" region: "us-central1" ``` ADC automatically supports GKE Workload Identity, the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, and default service account credentials when running on GCP. 2. Service Account Key Files: A service account key file is a JSON file containing a private key that authenticates as a service account. You create a service account in GCP, generate a key file, download it, and then store it in the k8s secret referenced by BackendSecurityPolicy. Agent Router uses this key file to generate an access token and authenticate with GCP Vertex AI. ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: gcp-auth-service-account-key namespace: default spec: type: GCPCredentials gcpCredentials: projectName: GCP_PROJECT_NAME # Replace with your GCP project name region: GCP_REGION # Replace with your GCP region credentialsFile: secretRef: name: envoy-ai-gateway-basic-gcp-service-account-key-file ``` 3. Workload Identity Federation: Workload Identity Federation is a modern, keyless authentication method that allows workloads running outside of GCP to impersonate a service account using their own native identity. It leverages a trust relationship between GCP and an external identity provider such as OIDC. ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: gcp-auth spec: type: GCPCredentials gcpCredentials: projectName: "your-gcp-project" region: "us-central1" workloadIdentityFederationConfig: projectID: "your-gcp-project-id" workloadIdentityPoolName: "your-workload-identity-pool" workloadIdentityProviderName: "your-identity-provider" serviceAccountImpersonation: serviceAccountName: "your-service-account" oidcExchangeToken: oidc: provider: issuer: "https://your-oidc-provider.com" clientID: "your-oidc-client-id" clientSecret: name: "gcp-client-secret" namespace: default ``` ##### Secrets in another namespace A `secretRef` or `clientSecretRef` without `namespace` refers to a Secret in the BackendSecurityPolicy's own namespace. A Secret in another namespace also needs a [ReferenceGrant](https://gateway-api.sigs.k8s.io/api-types/referencegrant/) in the Secret's namespace that allows the reference. Without it, the BackendSecurityPolicy is marked `NotAccepted`. ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: allow-backend-security-policy-secrets namespace: credentials # the Secret's namespace spec: from: - group: aigateway.envoyproxy.io kind: BackendSecurityPolicy namespace: ai-backends # the BackendSecurityPolicy's namespace to: - group: "" kind: Secret ``` With `aigw run`, put the ReferenceGrant in the configuration file. #### Security Best Practices - **Store credentials in Kubernetes Secrets**: Never expose sensitive data in plain text - **Use principle of least privilege**: Grant only necessary permissions - **Rotate credentials regularly**: Implement credential rotation policies - **Separate environments**: Use different credentials for development, staging, and production ### AIGatewayRoute The `AIGatewayRoute` resource defines how client requests are routed to appropriate AI backends and manages the unified API interface. #### Purpose and Configuration - **Request Routing**: Directs traffic to specific backends based on model names or other criteria - **API Unification**: Provides a consistent interface regardless of backend provider - **Request Transformation**: Automatically converts between different API schemas - **Load Balancing**: Distributes traffic across multiple backends #### Basic Configuration ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: multi-provider-route spec: # Gateway to attach to parentRefs: - name: my-gateway kind: Gateway group: gateway.networking.k8s.io # Routing rules rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: gpt-4o-mini backendRefs: - name: openai-backend - matches: - headers: - type: Exact name: x-ai-eg-model value: claude-3-sonnet backendRefs: - name: bedrock-backend ``` ## Resource Relationships and Data Flow Understanding how these resources work together is crucial for successful provider connectivity: ```mermaid graph TD A[Client Request] --> B[AIGatewayRoute] B --> C{Model Header} C -->|gpt-4o-mini| D[OpenAI AIServiceBackend] C -->|claude-3-sonnet| E[Bedrock AIServiceBackend] D --> F[OpenAI BackendSecurityPolicy] E --> G[AWS BackendSecurityPolicy] F --> H[OpenAI API] G --> I[AWS Bedrock API] style A fill:#e1f5fe style B fill:#f3e5f5 style D fill:#e8f5e8 style E fill:#e8f5e8 style F fill:#fff3e0 style G fill:#fff3e0 ``` ### Data Flow Process 1. **Request Reception**: Client sends a request to the AI Gateway with the OpenAI-compatible format 2. **Route Matching**: AIGatewayRoute examines request headers (like `x-ai-eg-model`) to determine the target backend 3. **Backend Resolution**: The matching rule identifies the appropriate AIServiceBackend 4. **Authentication**: The AIServiceBackend's security policy provides credentials for upstream authentication 5. **Schema Transformation**: If needed, the request is transformed from the input schema to the backend's expected schema 6. **Provider Communication**: The request is forwarded to the actual AI provider with proper authentication 7. **Response Processing**: The provider's response is transformed back to the unified schema format 8. **Client Response**: The standardized response is returned to the client ## Common Configuration Patterns ### Single Provider Setup For a simple single-provider setup: ```yaml # Backend configuration apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIServiceBackend metadata: name: openai-backend spec: schema: name: OpenAI version: v1 backendRef: name: openai-backend kind: Backend group: gateway.envoyproxy.io --- # Security configuration apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: openai-auth spec: targetRefs: - group: aigateway.envoyproxy.io kind: AIServiceBackend name: openai-backend type: APIKey apiKey: secretRef: name: openai-secret --- # Routing configuration apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: openai-route spec: parentRefs: - name: my-gateway kind: Gateway group: gateway.networking.k8s.io rules: - backendRefs: - name: openai-backend ``` ### Multi-Provider Setup with Fallback For high availability with multiple providers: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: multi-provider-fallback spec: parentRefs: - name: my-gateway kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: gpt-4o-mini backendRefs: - name: openai-backend - name: groq-backend # Fallback provider ``` ### Model-Specific Routing For routing different models to specialized providers: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: model-specific-routing spec: parentRefs: - name: my-gateway kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: gpt-4o-mini backendRefs: - name: openai-backend - matches: - headers: - type: Exact name: x-ai-eg-model value: claude-3-sonnet backendRefs: - name: bedrock-backend - matches: - headers: - type: Exact name: x-ai-eg-model value: text-embedding-ada-002 backendRefs: - name: openai-embeddings-backend ``` Configure model ownership and creation information for the `/models` endpoint: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: model-metadata spec: parentRefs: - name: my-gateway kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: gpt-4o-mini modelsOwnedBy: "OpenAI" modelsCreatedAt: "2024-01-01T00:00:00Z" backendRefs: - name: openai-backend ``` ## Provider-Specific Considerations ### OpenAI-Compatible Providers Many providers offer OpenAI-compatible APIs: - Use OpenAI schema configuration - Adjust the version field if the provider uses custom paths - Standard API key authentication typically applies ### Cloud Provider Integration Cloud providers like AWS Bedrock and Azure OpenAI require: - Cloud-specific credential types (AWS IAM, Azure Service Principal, GCP Workload Identity) - Region specification for multi-region services - Custom schema configurations for native APIs ### Self-Hosted Models Self-hosted models using frameworks like vLLM: - Often compatible with OpenAI schema - May not require authentication (internal networks) - Custom endpoints through Envoy Gateway Backend resources ## Validation and Troubleshooting ### Configuration Validation The AI Gateway validates configurations at deployment time: - **Schema Compatibility**: Ensures input and output schemas are compatible - **Resource References**: Validates that referenced resources exist - **Credential Access**: Verifies that secrets are accessible ### Status Conditions All resources provide status conditions to monitor their health: - **Accepted**: Resource is valid and has been accepted by the controller - **NotAccepted**: Resource has validation errors or configuration issues ### Common Issues and Solutions **Authentication Failures (401/403)** - Verify API keys and credentials are correct - Check secret references and key names - Ensure credentials have necessary permissions **Schema Mismatch Errors** - Confirm the backend schema matches the provider's API - Check version specifications for provider-specific paths - Review API documentation for schema requirements **Routing Issues** - Verify header matching rules in AIGatewayRoute - Check that model names match expected values - Ensure backend references point to existing AIServiceBackends **Backend Reference Errors** - Ensure backendRef points to Envoy Gateway Backend resources - Verify the Backend resource exists and is properly configured - Check that the group field is set to `gateway.envoyproxy.io` ## Next Steps Now that you understand the connectivity fundamentals: - **[Supported Providers](https://theagentrouter.ai/docs/capabilities/llm-integrations/supported-providers)** - View the complete list of supported providers and their configurations - **[Supported Endpoints](https://theagentrouter.ai/docs/capabilities/llm-integrations/supported-endpoints)** - Learn about available API endpoints and their capabilities - **[Getting Started Guide](https://theagentrouter.ai/docs/getting-started/connect-providers)** - Follow hands-on tutorials for specific providers - **[Traffic Management](https://theagentrouter.ai/docs/capabilities/traffic)** - Configure advanced routing, rate limiting, and fallback strategies - **[Security](https://theagentrouter.ai/docs/capabilities/security)** - Implement comprehensive security policies for your AI traffic ## API Reference For detailed information about resource fields and configuration options: - [AIServiceBackend API Reference](../../api/api.mdx#aiservicebackend) - [BackendSecurityPolicy API Reference](../../api/api.mdx#backendsecuritypolicy) - [AIGatewayRoute API Reference](../../api/api.mdx#aigatewayroute) --- # Extension Fields Source: https://theagentrouter.ai/docs/capabilities/llm-integrations/vendor-specific-fields The AI Gateway supports extension fields that allow you to specify unified or backend-specific parameters directly as inline fields in your OpenAI-compatible requests. These fields are applied during the translation process to the target backend's native API format. ## Overview Extension fields enable you to: - Use advanced backend-specific features not available in the OpenAI API - Use unified configuration fields that work across multiple providers not available in the OpenAI API ### Vendor Extension Fields Vendor specific fields are specified as inline fields in your OpenAI request and are applied after the standard OpenAI-to-backend translation. ### Unified Extension Fields For thinking/reasoning capabilities, you can use a unified `thinking` field that automatically translates to the correct backend-specific format: - **GCP Vertex AI (Gemini)**: Translates to `generationConfig.thinkingConfig` - **GCP Anthropic**: Uses `thinking` field directly - **AWS Bedrock**: Uses `thinking` field directly This unified approach allows you to write provider-agnostic requests while still leveraging thinking capabilities. ## Supported Backends The following backends support extension fields: ### GCP Vertex AI (Gemini) - **API Schema Name**: `GCPVertexAI` - **Supported Fields**: - `safetySettings`: Configure the safety settings for gemini models that translates to `SafetySetting`. [Gemini Docs](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/multimodal/configure-safety-filters) - `thinking`: Configure thinking process for reasoning models that automatically translates to `generationConfig.thinkingConfig`. [Gemini Docs](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/thinking) - **Supported Tools**: - `google_search`: Enable Google Search grounding for Gemini models. Configuration options vary by platform: `exclude_domains` and `blocking_confidence` are Vertex AI only, while `time_range_filter` is Gemini API only. [Google Search Grounding Docs](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/grounding-with-google-search) ### GCP Anthropic - **API Schema Name**: `GCPAnthropic` - **Supported Fields**: - `thinking`: Configuration for enabling Claude's extended thinking. [Anthropic Docs](https://docs.anthropic.com/en/api/messages#body-thinking) - **Supported Tool Fields** (set on an entry of `tools[].function`): - `eager_input_streaming`: Stream a tool's input as the model generates it, instead of buffering and validating each parameter value first. [Anthropic Docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/fine-grained-tool-streaming) ### AWS Anthropic - **API Schema Name**: `AWSAnthropic` - **Supported Tool Fields** (set on an entry of `tools[].function`): - `eager_input_streaming`: Stream a tool's input as the model generates it, instead of buffering and validating each parameter value first. [Anthropic Docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/fine-grained-tool-streaming) ### AWS Bedrock - **API Schema Name**: `AWSBedrock` - **Supported Fields**: - `thinking`: Configuration for enabling Anthropic Claude's extended thinking. [AWS Docs](https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-extended-thinking.html) ## Usage Add extension fields directly as inline fields in your OpenAI request: ### Using Unified Thinking Configuration The simplest way to enable thinking capabilities across all providers is to use the unified `thinking` field: ```json { "model": "gemini-2.5-pro", "messages": [ { "role": "user", "content": "Explain quantum computing and show me a simple code example." } ], "temperature": 0.7, "max_tokens": 2000, "thinking": { "type": "enabled", "budget_tokens": 1000, "includeThoughts": true } } ``` This configuration will work with any provider that supports thinking, automatically translating to the correct backend format. ### Using Provider-Specific Fields For more fine-grained control or provider-specific features, you can use the vendor-specific fields like `safetySettings` for gemini models: ```json { "model": "gemini-1.5-pro", "messages": [ { "role": "user", "content": "Explain quantum computing and show me a simple code example." } ], "temperature": 0.7, "max_tokens": 2000, "safetySettings": [ { "category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_ONLY_HIGH" } ] } ``` ### Using Google Search Grounding To enable Google Search grounding for Gemini models, add `google_search` to the tools array. For basic usage without configuration options: ```json { "model": "gemini-2.0-flash", "messages": [ { "role": "user", "content": "What are the latest developments in quantum computing?" } ], "tools": [ { "type": "google_search" } ] } ``` For Vertex AI, you can add filtering options: ```json { "model": "gemini-2.0-flash", "messages": [ { "role": "user", "content": "What are the latest developments in quantum computing?" } ], "tools": [ { "type": "google_search", "google_search": { "exclude_domains": ["example.com"], "blocking_confidence": "BLOCK_LOW_AND_ABOVE" } } ] } ``` ### Using Eager Tool Input Streaming By default, Anthropic buffers each tool parameter value and validates it before emitting it to the stream. Setting `eager_input_streaming` on a tool makes the model stream that tool's input as it is generated, so `tool_calls[].function.arguments` fragments start arriving sooner. This matches the per-token streaming behavior of OpenAI and most other backends. ```json { "model": "gcp.claude-3-5-haiku", "messages": [ { "role": "user", "content": "What's the weather in New York?" } ], "stream": true, "tools": [ { "type": "function", "function": { "name": "get_weather", "eager_input_streaming": true, "parameters": { "type": "object", "properties": { "location": { "type": "string" } } } } } ] } ``` The field is a tri-state. Omitting it defaults to buffered streaming, `true` enables eager streaming for that tool, and an explicit `false` also buffers but holds that even when the deprecated `fine-grained-tool-streaming-2025-05-14` beta header is present, which otherwise turns unset tools on. How arguments are accumulated does not change: clients concatenate the `arguments` fragments exactly as before. What changes is the guarantee. With eager streaming the server no longer validates the fragments, so the accumulated string is not guaranteed to be valid JSON — a response ending with `finish_reason: "length"` can cut a parameter off midway. Guard the parse and handle failure rather than assuming it succeeds. See [Accumulating tool input deltas](https://platform.claude.com/docs/en/agents-and-tools/tool-use/fine-grained-tool-streaming#accumulating-tool-input-deltas). ### Field Conflicts Vendor fields override translated fields when conflicts occur. When using unified thinking configuration, the `thinking` field takes precedence over any provider-specific thinking configurations. ### Unsupported Fields/Backends Fields and Backends other than specified in [Supported Backends](#supported-backends) will be ignored. --- # Observability Source: https://theagentrouter.ai/docs/capabilities/observability/ Agent Router extends the capabilities of Envoy Gateway, and as you run Envoy Gateway you have access to the foundational observability in the Envoy Gateway system. We recommend you familiarize yourself with the [Envoy Gateway Observability Documentation](https://gateway.envoyproxy.io/docs/tasks/observability/). ## AI/LLM Observability Features The Agent Router provides specialized observability capabilities for AI and LLM workloads: - **[GenAI Metrics](./metrics.md)** - Prometheus metrics following OpenTelemetry Gen AI semantic conventions for monitoring token usage, latency, and model performance. - **[GenAI Tracing](./tracing.md)** - OpenTelemetry integration with OpenInference semantic conventions for LLM request tracing and evaluation. - **[Access Logs with AI/LLM metadata](./accesslogs.md)** - AI metadata produced by the AI gateway (model name, token usage, etc.) can be included in the Envoy Access Logs. - **[Gateway Configuration](../gateway-config.md)** - Per-gateway configuration of the external processor container, including environment variables for tracing and resource requirements. --- # AI/LLM Metrics Source: https://theagentrouter.ai/docs/capabilities/observability/metrics When using the Agent Router, it will collect AI specific metrics and expose them to Prometheus for monitoring by default. This guide provides an overview of the metrics collected by the AI Gateway and how to monitor them using Prometheus. ## Overview Agent Router is designed to intercept and process AI/LLM requests, that enables it to collect metrics for monitoring and observability. Currently, it collects metrics and exports them to Prometheus for monitoring in the OpenTelemetry format as specified by the [OpenTelemetry Gen AI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/). Not all metrics are supported yet, but the Agent Router will continue to add more metrics in the future. ### Supported Endpoints Metrics are collected for the following LLM endpoints: - **`/v1/chat/completions`** - Chat completions (streaming and non-streaming) - **`/v1/completions`** - Legacy text completions (streaming and non-streaming) - **`/v1/embeddings`** - Text embeddings - **`/cohere/v2/rerank`** - Rerank - **`/typesafe/v1/systemone`** - TypeSafe System One (Jev) - **`/anthropic/v1/messages`** - Anthropic messages (streaming and non-streaming) For example, the Agent Router collects metrics such as: - [**`gen_ai.client.token.usage`**](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiclienttokenusage): Number of tokens processed. The attribute `gen_ai.token.type` can be used to differentiate between input, output, and total tokens. - [**`gen_ai.server.request.duration`**](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiserverrequestduration): Measured from the start of the received request headers in the Agent Router filter to the end of the processed response body processing. - [**`gen_ai.server.time_to_first_token`**](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiservertime_to_first_token): Measured from the start of the received request headers in the Agent Router filter to the receiving of the first token in the response body handling. - [**`gen_ai.server.time_per_output_token`**](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiservertime_per_output_token): The latency between consecutive tokens, if supported, or by chunks/tokens otherwise. Each metric comes with some default attributes such as: - `gen_ai.operation.name` - `chat`: For `/v1/chat/completions` endpoint. - `completion`: For `/v1/completions` endpoint. - `embedding`: For `/v1/embeddings` endpoint. - `rerank`: For `/cohere/v2/rerank` endpoint. - `systemone`: For `/typesafe/v1/systemone` endpoint. - `image_generation`: For `/v1/images/generations` endpoint. - `messages`: For `/anthropic/v1/messages` endpoint. - `gen_ai.original.model` - The original model name from the request body - `gen_ai.request.model` - The model name requested (may be overridden) - `gen_ai.response.model` - The model name returned in the response - `gen_ai.provider.name` - The provider name (e.g., `openai`, `anthropic`) - `gen_ai.backend` - The `AIServiceBackend` that served the request, as `namespace/name` **Tip:** You can enrich the metrics with custom labels extracted from HTTP request headers. Use `controller.requestHeaderAttributes` for a base mapping shared with spans and access logs, and `controller.metricsRequestHeaderAttributes` for metrics-only mappings. Metrics never default to `session.id` because it is high-cardinality. See [values.yaml](https://github.com/theagentrouter/agent-router/blob/main/manifests/charts/ai-gateway-helm/values.yaml) for more details including other configurations. ## Trying it out Before you begin, you'll need to complete the basic setup from the [Basic Usage](https://theagentrouter.ai/docs/getting-started/basic-usage) guide. Then, you can install the prometheus using the following commands: ```shell kubectl apply -f https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/monitoring/monitoring.yaml ``` Let's wait for a while until the Prometheus is up and running. ```shell kubectl wait --for=condition=ready pod -l app=prometheus -n monitoring ``` To access the Prometheus dashboard, you need to port-forward the Prometheus service to your local machine like this: ```shell kubectl port-forward -n monitoring svc/prometheus 9090:9090 ``` Now open your browser and navigate to `http://localhost:9090` to access the Prometheus dashboard to explore the metrics. Alternatively, you can make the following requests to see the raw metrics: ```shell curl http://localhost:9090/api/v1/query --data-urlencode \ 'query=sum(gen_ai_client_token_usage_sum{gateway_envoyproxy_io_owning_gateway_name = "envoy-ai-gateway-basic"}) by (gen_ai_request_model, gen_ai_token_type)' \ | jq '.data.result[]' ``` and then you would get the response like this, assuming you have made some requests with the model `gpt-4o-mini`: ```json lines { "metric": { "gen_ai_request_model": "gpt-4o-mini", "gen_ai_token_type": "input" }, "value": [ 1743105857.684, "12" ] } { "metric": { "gen_ai_request_model": "gpt-4o-mini", "gen_ai_token_type": "output" }, "value": [ 1743105857.684, "13" ] } { "metric": { "gen_ai_request_model": "gpt-4o-mini", "gen_ai_token_type": "total" }, "value": [ 1743105857.684, "25" ] } ``` --- # GenAI Distributed Tracing Source: https://theagentrouter.ai/docs/capabilities/observability/tracing Agent Router's router joins and records distributed traces when supplied with an [OpenTelemetry](https://opentelemetry.io/) collector endpoint. This guide provides an overview of the spans recorded by the AI Gateway and how export them to your choice of OpenTelemetry collector. ## Overview Agent Router's router joins and records distributed traces when supplied with an [OpenTelemetry](https://opentelemetry.io/) collector endpoint. Requests to the OpenAI Chat Completions, Completions (legacy), and Embeddings endpoints are recorded as Spans which include typical timing and request details. In addition, there are GenAI attributes representing the LLM or Embeddings call including full request and response details, defined by [OpenInference semantic conventions][openinference]. OpenInference attributes default to include full request and response data for both chat completions and embeddings. This can be toggled with configuration, but when enabled allows systems like [Arize Phoenix][phoenix] to perform evaluations of production requests captured in OpenTelemetry spans. For chat completions, this includes traditional LLM metrics such as correctness and hallucination detection. For embeddings, it enables agentic RAG evaluations focused on retrieval and semantic analysis. ## Trying it out Before you begin, you'll need to complete the basic setup from the [Basic Usage](https://theagentrouter.ai/docs/getting-started/basic-usage) guide, which includes installing Envoy Gateway and AI Gateway. ### Install Phoenix for LLM observability ```shell # Install Phoenix using PostgreSQL storage. helm install phoenix oci://registry-1.docker.io/arizephoenix/phoenix-helm \ --namespace envoy-ai-gateway-system \ --set auth.enableAuth=false \ --set server.port=6006 # Wait for Phoenix to be ready (first run may take a few minutes to pull images) kubectl wait --timeout=5m -n envoy-ai-gateway-system \ pods -l app=phoenix --for=condition=Ready ``` ### Configure AI Gateway with OpenTelemetry Upgrade your AI Gateway installation with [OpenTelemetry configuration][otel-config]: ```shell helm upgrade aieg oci://docker.io/envoyproxy/ai-gateway-helm \ --version v1.2.0 \ --namespace envoy-ai-gateway-system \ --set "extProc.extraEnvVars[0].name=OTEL_EXPORTER_OTLP_ENDPOINT" \ --set "extProc.extraEnvVars[0].value=http://phoenix-svc.envoy-ai-gateway-system:6006" \ --set "extProc.extraEnvVars[1].name=OTEL_METRICS_EXPORTER" \ --set "extProc.extraEnvVars[1].value=none" # OTEL_SERVICE_NAME defaults to "ai-gateway" if not set # OTEL_METRICS_EXPORTER=none because Phoenix only supports traces, not metrics # Note: Use fully-qualified service name because ext-proc runs in envoy-gateway-system namespace ``` Wait for the gateway pod to be ready: ```shell kubectl wait --for=condition=Ready -n envoy-gateway-system \ pods -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic ``` ### Generate traces Make requests to your AI Gateway to generate traces. Follow the instructions from [Testing the Gateway](https://theagentrouter.ai/docs/getting-started/basic-usage#testing-the-gateway) in the Basic Usage guide to: 1. Set up port forwarding (if needed) 2. Test the chat completions endpoint Each request will generate traces that are sent to Phoenix. ### Check Phoenix is receiving traces ```bash kubectl logs -n envoy-ai-gateway-system deployment/phoenix | grep "POST /v1/traces" ``` ### Access Phoenix UI Port-forward to access the Phoenix dashboard: ```shell kubectl port-forward -n envoy-ai-gateway-system svc/phoenix-svc 6006:6006 ``` Then open http://localhost:6006 in your browser to explore the traces. ## Semantic Conventions Spans are recorded using [OpenInference semantic conventions][openinference] by default. You can instead emit the [OpenTelemetry GenAI semantic conventions][otel-genai], which use `gen_ai.*` attributes and are consumed by OpenTelemetry-native backends: ```yaml extProc: extraEnvVars: - name: AI_GATEWAY_TRACING_SEMCONV value: "gen_ai" ``` | Value | Behavior | | ------------------------ | ---------------------------------- | | unset or `openinference` | OpenInference attributes (default) | | `gen_ai` | OpenTelemetry GenAI attributes | Any other value fails startup rather than silently falling back, so a typo is reported immediately instead of producing traces nobody is watching. Only one convention is emitted at a time. Choosing `gen_ai` changes the shape of your spans, so update dashboards and alerts before switching: | | OpenInference | OpenTelemetry GenAI | | --------- | ------------------------ | ----------------------------------------------- | | Span name | `ChatCompletion` | `chat {model}` | | Span kind | `INTERNAL` | `CLIENT` | | Model | `llm.model_name` | `gen_ai.request.model`, `gen_ai.response.model` | | Provider | `llm.system` | `gen_ai.provider.name` | | Tokens | `llm.token_count.prompt` | `gen_ai.usage.input_tokens` | | Messages | `llm.input_messages.N.*` | `gen_ai.input.messages` (single JSON value) | | Errors | `exception` event | `error.type` attribute | Note that the GenAI conventions are still marked Development upstream, so attribute names may change in future releases. Since span names include the model, they are higher cardinality than OpenInference's fixed names. This matters for backends that index on span name. ### Capturing message content with GenAI Unlike OpenInference, which records request and response content by default, the GenAI conventions treat message content as opt-in because it routinely contains sensitive data. Enable it explicitly: ```yaml extProc: extraEnvVars: - name: AI_GATEWAY_TRACING_SEMCONV value: "gen_ai" - name: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT value: "true" ``` Token counts and sampling parameters are recorded either way, since the conventions treat those as metadata rather than content. Message content is currently mapped for chat completions and Anthropic messages. Other endpoints record operation, model, usage and sampling parameters. Several — image generation, speech, transcription, translation and rerank and TypeSafe System One — have no content attributes defined by the conventions at all. The `OPENINFERENCE_HIDE_*` variables described below apply only to the OpenInference convention. They have no effect when `gen_ai` is selected. ### MCP spans `AI_GATEWAY_TRACING_SEMCONV` selects the vocabulary for MCP spans as well as for the LLM endpoints. OpenInference defines no MCP conventions, so the default keeps the gateway-specific attributes MCP spans have always used; `gen_ai` opts into the [OpenTelemetry MCP semantic conventions][otel-mcp]. | | default (`openinference`) | `gen_ai` | | ------------- | ------------------------- | ---------------------------------------------- | | Span name | `CallTool`, `ListTools` | `tools/call {tool}`, `tools/list` | | Tool name | `mcp.tool.name` | `gen_ai.tool.name` + `gen_ai.operation.name` | | Prompt name | `mcp.prompt.name` | `gen_ai.prompt.name` | | Request ID | `mcp.request.id` | `jsonrpc.request.id` | | Transport | `mcp.transport` | `network.transport`, `network.protocol.*` | | Errors | `exception` event | `error.type`, `rpc.response.status_code` | | Session | on the per-backend event | also `mcp.session.id` on the span | | List sizes | not recorded | `mcp.tools.count`, `mcp.resources.count`, ... | | Tool call I/O | not recorded | `gen_ai.tool.call.arguments`/`.result`, opt-in | Tool call arguments and results are message content, so under `gen_ai` they follow the same `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` opt-in as the LLM endpoints. The default convention never records them. **Deprecation:** The gateway-specific MCP attributes are deprecated in favor of the OpenTelemetry MCP conventions. They remain the default for now; a future release will announce a version in which the default flips. Migrate by setting `AI_GATEWAY_TRACING_SEMCONV=gen_ai` once your dashboards query the new names. ## Privacy Configuration **Note:** This section applies to the default OpenInference convention. See [Capturing message content with GenAI](#capturing-message-content-with-genai) for the `gen_ai` equivalent. Control sensitive data in traces by adding [OpenInference configuration][openinference-config] to Helm values when you reconfigure the AI Gateway. There is [similar config][openinference-embeddings] for embeddings: For example, if you are using a `values.yaml` file instead of command line arguments, you can add the following to control redaction: ```yaml extProc: extraEnvVars: # Base OTEL configuration... # Hide sensitive data (all default to false) - name: OPENINFERENCE_HIDE_INPUTS value: "true" # Hide input messages to the LLM - name: OPENINFERENCE_HIDE_OUTPUTS value: "true" # Hide output messages from the LLM # Reduce volume for embeddings (all default to false) - name: OPENINFERENCE_HIDE_EMBEDDINGS_TEXT value: "true" # Hide embeddings input - name: OPENINFERENCE_HIDE_EMBEDDINGS_VECTORS value: "true" # Hide embeddings output ``` Note: Hiding inputs/outputs prevents human or LLM-as-a-Judge evaluation of your LLM requests, such as done with the [Phoenix Evals library][phoenix-evals]. ## Session Tracking Sessions help track and organize related traces across multi-turn conversations with your AI app. Maintaining context between interactions is key for observability. With sessions, you can: - Track a conversation's full history in one thread. - View inputs/outputs for a given agent. - Monitor token usage and latency per conversation. By tagging spans with a consistent session ID, you get a connected view of performance across a user's journey. The challenge is that requests to the gateway may not send traces, making grouping difficult. Many GenAI frameworks allow you to set custom HTTP headers when sending traffic to an LLM. Propagating sessions this way is simpler than instrumenting applications with tracing code and can still achieve grouping. There's no standard name for session ID headers, but there is a common attribute in OpenTelemetry, [session.id][otel-session], which has special handling in some OpenTelemetry platforms such as [Phoenix][phoenix-session]. To bridge this gap, Agent Router lets you map HTTP request headers to OpenTelemetry attributes. You can define a base mapping shared by metrics, spans, and access logs, plus optional per-signal mappings for metrics, spans, and access logs. - `controller.requestHeaderAttributes` - `controller.spanRequestHeaderAttributes` - `controller.metricsRequestHeaderAttributes` - `controller.logRequestHeaderAttributes` `controller.spanRequestHeaderAttributes` and `controller.logRequestHeaderAttributes` default to `agent-session-id:session.id` when unset (set them to an empty string to disable the default). Metrics never default to `session.id`. Both of these use the same value format: a comma-separated list of `:` pairs. For example, if your session ID header is `agent-session-id`, you can map it to the standard OpenTelemetry attribute `session.id` like this: `agent-session-id:session.id`. Some metrics systems will be able to do fine-grained aggregation, but not all. Here's an example of keeping the default session mapping for spans/logs while only adding a low-cardinality attribute to metrics: ```shell helm upgrade aieg oci://docker.io/envoyproxy/ai-gateway-helm \ --version v1.2.0 \ --namespace envoy-ai-gateway-system \ --reuse-values \ --set "controller.metricsRequestHeaderAttributes=x-tenant-id:tenant.id" ``` ## Cleanup To remove Phoenix and disable tracing: ```shell # Uninstall Phoenix helm uninstall phoenix -n envoy-ai-gateway-system # Disable tracing in AI Gateway helm upgrade aieg oci://docker.io/envoyproxy/ai-gateway-helm \ --version v1.2.0 \ --namespace envoy-ai-gateway-system \ --reuse-values \ --unset extProc.extraEnvVars ``` ## Per-Gateway Configuration For deployments with multiple Gateways that need different tracing configurations, use the `GatewayConfig` CRD instead of global Helm values. This allows you to: - Configure different OTEL endpoints for different Gateways - Set per-gateway service names for better trace organization - Override global tracing settings for specific Gateways ### Example ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: GatewayConfig metadata: name: production-tracing namespace: default spec: extProc: kubernetes: env: - name: OTEL_EXPORTER_OTLP_ENDPOINT value: "http://production-collector:4317" - name: OTEL_SERVICE_NAME value: "ai-gateway-production" --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: production-gateway annotations: aigateway.envoyproxy.io/gateway-config: production-tracing spec: # ... ``` See the [Gateway Configuration](../gateway-config.md) guide for detailed information on `GatewayConfig` usage, including environment variable precedence and shared configurations. ## See Also - [Gateway Configuration](../gateway-config.md) - Per-gateway configuration using GatewayConfig - [OpenInference Specification][openinference] - GenAI Semantic conventions for traces - [OpenTelemetry Configuration][otel-config] - Environment variable reference - [Arize Phoenix Documentation][phoenix] - LLM observability platform --- [openinference]: https://github.com/Arize-ai/openinference/tree/main/spec [otel-genai]: https://github.com/open-telemetry/semantic-conventions-genai [otel-mcp]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md [openinference-config]: https://github.com/Arize-ai/openinference/blob/main/spec/configuration.md [openinference-embeddings]: https://github.com/Arize-ai/openinference/blob/main/spec/embedding_spans.md [otel-config]: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/ [phoenix]: https://docs.arize.com/phoenix [phoenix-evals]: https://arize.com/docs/phoenix/evaluation/llm-evals [otel-session]: https://opentelemetry.io/docs/specs/semconv/registry/attributes/session/ [phoenix-session]: https://arize.com/docs/phoenix/tracing/llm-traces/sessions --- # Access Logs With AI/LLM/MCP Metadata Source: https://theagentrouter.ai/docs/capabilities/observability/accesslogs It is possible to configure Envoy Access Logs to include metadata generated by the AI Gateway such as the selected model and token consumption. This guide walks through the steps to configure the AI Gateway Routes and MCP Routes to generate LLM token consumption and MCP metadata, and to configure the Envoy Access Logs to include the AI/LLM/MCP details. ## LLM Metadata in Access Logs Agent Router populates information in the filter dynamic metadata under the `io.envoy.ai_gateway` namespace. This metadata includes information about the selected model, prompt and completion token usage, and other details about the LLM request, and can be extracted and included in the Envoy Access Logs. ### AIGatewayRoute configuration The contents of the dynamic metadata are configured in the `AIGatewayRoute` resource under the `llmRequestCosts` field. The `llmRequestCosts` field configures a list of metadata keys to populate in the Envoy filter metadata, to be later accessed in the access logs. For example: ```yaml apiVersion: ai-gateway.envoyproxy.io/v1alpha1 kind: AIGatewayRoute metadata: name: ai-gateway-route spec: (...) llmRequestCosts: - metadataKey: llm_input_token type: InputToken - metadataKey: llm_output_token type: OutputToken - metadataKey: llm_total_token type: TotalToken rules: (...) ``` ### EnvoyProxy Configuration Once the `AIGatewayRoute` is configured to populate the dynamic metadata, the Envoy Access Logs can be configured to include the metadata in the log entries. This can be done by configuring the access logs in the `EnvoyProxy` resource: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: EnvoyProxy metadata: name: ai-access-logs namespace: default spec: telemetry: accessLog: settings: - sinks: - type: File file: path: /dev/stdout format: type: JSON json: # AI specific fields. The properties in the dynamic metadata expressions must match the ones # defined in the AIGatewayRoute llmRequestCosts field. gen_ai.request.model: "%REQ(X-AI-EG-MODEL)%" gen_ai.response.model: "%DYNAMIC_METADATA(io.envoy.ai_gateway:response_model)%" gen_ai.request.model_override: "%DYNAMIC_METADATA(io.envoy.ai_gateway:model_name_override)%" gen_ai.provider.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:backend_name)%" gen_ai.usage.total_tokens: "%DYNAMIC_METADATA(io.envoy.ai_gateway:llm_total_token)%" gen_ai.usage.input_tokens: "%DYNAMIC_METADATA(io.envoy.ai_gateway:llm_input_token)%" gen_ai.usage.output_tokens: "%DYNAMIC_METADATA(io.envoy.ai_gateway:llm_output_token)%" # Common fields start_time: "%START_TIME%" method: "%REQ(:METHOD)%" x-envoy-origin-path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" protocol: "%PROTOCOL%" response_code: "%RESPONSE_CODE%" response_flags: "%RESPONSE_FLAGS%" response_code_details: "%RESPONSE_CODE_DETAILS%" connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%" upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%" bytes_received: "%BYTES_RECEIVED%" bytes_sent: "%BYTES_SENT%" duration: "%DURATION%" x-envoy-upstream-service-time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%" x-forwarded-for: "%REQ(X-FORWARDED-FOR)%" user-agent: "%REQ(USER-AGENT)%" x-request-id: "%REQ(X-REQUEST-ID)%" ":authority": "%REQ(:AUTHORITY)%" upstream_host: "%UPSTREAM_HOST%" upstream_cluster: "%UPSTREAM_CLUSTER%" upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%" downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%" downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%" requested_server_name: "%REQUESTED_SERVER_NAME%" route_name: "%ROUTE_NAME%" ``` In this example we're adding the `gen_ai.*` properties with the values extracted from the filter metadata populated by the AI Gateway. The `EnvoyProxy` resource must be attached to the `Gateway` object or to the `GatewayClass`. For example: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: envoy-ai-gateway-basic namespace: default spec: gatewayClassName: envoy-ai-gateway-basic listeners: - name: http protocol: HTTP port: 80 infrastructure: parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: ai-access-logs ``` With this configuration, the access log entries will include the AI Gateway metadata and look like this: ```json { ":authority": "api.router.tetrate.ai", "bytes_received": 105, "bytes_sent": 432, "connection_termination_details": null, "downstream_local_address": "127.0.0.1:1975", "downstream_remote_address": "127.0.0.1:64484", "duration": 1526, "gen_ai.provider.name": "default/tars/route/aigw-run/rule/0/ref/0", "gen_ai.request.model": "gpt-4o-mini", "gen_ai.request.model_override": null, "gen_ai.usage.input_tokens": 15, "gen_ai.usage.output_tokens": 7, "gen_ai.usage.total_tokens": 22, "method": "POST", "protocol": "HTTP/1.1", "requested_server_name": null, "response_code": 200, "response_code_details": "via_upstream", "response_flags": "-", "route_name": "httproute/default/aigw-run/rule/0/match/0/*", "start_time": "2025-08-30T16:51:37.894Z", "upstream_cluster": "httproute/default/aigw-run/rule/0", "upstream_host": "34.128.142.77:443", "upstream_local_address": "192.168.1.38:64485", "upstream_transport_failure_reason": null, "user-agent": "curl/8.7.1", "x-envoy-origin-path": "/v1/chat/completions", "x-envoy-upstream-service-time": "451", "x-forwarded-for": "192.168.1.38", "x-request-id": "83b63da3-45a9-4196-98b8-1a7f697d6d8d" } ``` ### Trying it out You can deploy the example to quickly try the access log configuration against a local backend: ```shell kubectl apply -f https://raw.githubusercontent.com/theagentrouter/agent-router/main/examples/access-log/basic.yaml ``` Once everything is applied you can send requests to the gateway and see the access logs in the gateway pod logs. ## MCP Metadata in Access Logs Agent Router automatically populates MCP information in the filter dynamic metadata under the `io.envoy.ai_gateway` namespace. This metadata includes information about the request, MCP session, MCP method call, etc, and can be extracted and included in the Envoy Access Logs. ### EnvoyProxy Configuration The Envoy Access Logs can be configured to include the MCP metadata in the log entries as shown in the following `EnvoyProxy` resource: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: EnvoyProxy metadata: name: ai-access-logs namespace: default spec: telemetry: accessLog: settings: - sinks: - type: File file: path: /dev/stdout format: type: JSON json: # MCP specific fields jsonrpc.request.id: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_request_id)%" mcp.session.id: "%REQ(MCP-SESSION-ID)%" mcp.method.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_method)%" mcp.tool.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_tool_name)%" mcp.provider.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_backend)%" # Common fields start_time: "%START_TIME%" method: "%REQ(:METHOD)%" x-envoy-origin-path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" protocol: "%PROTOCOL%" response_code: "%RESPONSE_CODE%" response_flags: "%RESPONSE_FLAGS%" response_code_details: "%RESPONSE_CODE_DETAILS%" connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%" upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%" bytes_received: "%BYTES_RECEIVED%" bytes_sent: "%BYTES_SENT%" duration: "%DURATION%" x-envoy-upstream-service-time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%" x-forwarded-for: "%REQ(X-FORWARDED-FOR)%" user-agent: "%REQ(USER-AGENT)%" x-request-id: "%REQ(X-REQUEST-ID)%" ":authority": "%REQ(:AUTHORITY)%" upstream_host: "%UPSTREAM_HOST%" upstream_cluster: "%UPSTREAM_CLUSTER%" upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%" downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%" downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%" requested_server_name: "%REQUESTED_SERVER_NAME%" route_name: "%ROUTE_NAME%" ``` In this example we're adding the `mcp_*` properties with the values extracted from the filter metadata populated by the AI Gateway. The `EnvoyProxy` resource must be attached to the `Gateway` object or to the `GatewayClass` as explained in the previous section. With this configuration, the access log entries will include the AI Gateway metadata and look like this: ```json { "bytes_received": 165, "bytes_sent": 10255, "connection_termination_details": null, "downstream_local_address": "127.0.0.1:10088", "downstream_remote_address": "127.0.0.1:58124", "duration": 653, "gen_ai.provider.name": null, "gen_ai.request.model": null, "gen_ai.request.model_override": null, "gen_ai.usage.input_tokens": null, "gen_ai.usage.output_tokens": null, "mcp.provider.name": "context7", "mcp.method.name": "tools/call", "jsonrpc.request.id": "3", "mcp.session.id": null, "method": "POST", "response_code": 200, "start_time": "2025-10-08T10:42:11.430Z", "upstream_cluster": "httproute/default/ai-eg-mcp-br-mcp-route-context7/rule/0", "upstream_host": "13.217.64.66:443", "upstream_local_address": "192.168.40.127:58172", "upstream_transport_failure_reason": null, "user-agent": "Go-http-client/1.1", "x-envoy-origin-path": "/mcp", "x-request-id": "cf2c0454-6290-498e-af6c-e32a611b5da1" } ``` ### Trying it out You can deploy the example to quickly try the access log configuration by following the [MCP examples](https://github.com/theagentrouter/agent-router/tree/main/examples/mcp) and use any of the provided configurations. --- # Security Source: https://theagentrouter.ai/docs/capabilities/security/ As Agent Router is built on Envoy Gateway, you can leverage the Envoy Gateway Security Policy by attaching them to the Gateway and/or generated HTTPRoutes. **Tip:** View all **[Envoy Gateway Security Docs](https://gateway.envoyproxy.io/docs/tasks/security/)** to learn more what security configurations are available to you. ## Common Security Docs Below are a list of common security configurations that can be useful when securing your gateway leveraging Envoy Gateway configurations. ### Access Control - [JWT Validation](https://gateway.envoyproxy.io/docs/tasks/security/jwt-authentication/) - _Validate signed JWT tokens_ - [JWT Claim Based Authorization](https://gateway.envoyproxy.io/docs/tasks/security/jwt-claim-authorization/) - _Assert values of claims in JWT tokens_ - [Mutual TLS](https://gateway.envoyproxy.io/docs/tasks/security/mutual-tls/) - _Certificate based access control_ - [Integrate with External Authorization Service](https://gateway.envoyproxy.io/docs/tasks/security/ext-auth/) - _Useful for custom logic to your business_ - [OIDC Provider Integration](https://gateway.envoyproxy.io/docs/tasks/security/oidc/) - _When you want to have integration with a login flow for an end-user_ - [Require Basic Auth](https://gateway.envoyproxy.io/docs/tasks/security/basic-auth/) - [Require API Key](https://gateway.envoyproxy.io/docs/tasks/security/apikey-auth/) - [IP Allowlist/Denylist](https://gateway.envoyproxy.io/docs/tasks/security/restrict-ip-access/) ### TLS - [Setup TLS Certificate](https://gateway.envoyproxy.io/docs/tasks/security/secure-gateways/) - [Using TLS cert-manager](https://gateway.envoyproxy.io/docs/tasks/security/tls-cert-manager/) --- # Upstream Authentication Source: https://theagentrouter.ai/docs/capabilities/security/upstream-auth ## Overview Upstream Authentication secures the connectivity between the Gateway and the LLM Providers. This layer ensures secure communication and proper access control when the Gateway interacts with various AI service providers like AWS Bedrock, Azure OpenAI, Open AI, Mistral, Gemini, and other platforms. ## Why Upstream Authentication Matters Upstream Authentication is essential for several reasons: 1. **Security Layer**: It provides a secure authentication mechanism between the Gateway and upstream providers, preventing credentials sprawl across teams and organizations. 1. **Credential Management**: It allows AI platform teams to handle the secure storage and management of credentials and use them in one place. This makes the revocation and rotation of credentials easier. 1. **Compliance**: It helps organizations meet security and compliance requirements by maintaining proper authentication protocols and enforcing controlled usage of AI resources. ## Enterprise Security Architecture In enterprise environments, security is implemented in multiple layers: 1. **Client to Gateway Authentication** - Clients must authenticate to the Gateway using appropriate methods (API keys, OAuth, etc.) aligned with the organization's security standards. - This ensures only authorized clients can access the Gateway's services, and the Gateway enforces the access to the upstream providers and models. :::tip Check out [Envoy Gateway Security Documentation](https://gateway.envoyproxy.io/docs/tasks/security/) for Client to Gateway security configuration options. ::: 2. **Gateway to Upstream Authentication** - The Gateway must authenticate to upstream providers. - This layer is managed by Upstream Authentication. - Ensures secure communication between the Gateway and AI service providers. ![Upstream Authentication](https://theagentrouter.ai/diagrams/upstream-auth.png) ## Credential Management **Where the providers support short lived access credentials**, the Agent Router control plane supports automated credential management with the providers' identity system. This ensures a short-lived proof of authorization, such as an access token, is used when the request is sent to the upstream service. **For providers that support long lived access credentials**, the Agent Router control plane supports a manual credential management process. In these cases the credentials, like API keys, are stored in Kubernetes secrets and managed by the administrator. ### Automated Credential Management The Gateway integrates with each provider's identity system to ensure secure, short-lived authentication: - **AWS Bedrock**: Uses OIDC integration with AWS STS to generate temporary credentials for each request - **Azure OpenAI**: Leverages Entra ID (formerly Azure AD) to provide short-lived access tokens - **GCP VertexAI**: Uses GCP workload federation with Google STS to generate temporary credentials for each request In both cases, the Gateway automatically manages these credentials, ensuring that each request to upstream providers is sent with short-lived credentials. This approach significantly reduces the risk of credential exposure and aligns with enterprise security best practices. **Info:** Learn more about connecting to [AWS Bedrock](https://theagentrouter.ai/docs/getting-started/connect-providers/aws-bedrock) and [Azure OpenAI](https://theagentrouter.ai/docs/getting-started/connect-providers/azure-openai) in the provider specific documentation. ### Manual Credential Management For providers that support long lived access credentials, the Agent Router control plane supports a manual credential management process. In these cases the credentials, like API keys, are stored in Kubernetes secrets and managed by the AI Gateway administrator. Agent Router will use the credentials from the secret to authenticate with the upstream service, attaching them to each request by securely retrieving them from the secret and attaching them to the request. **Info:** Learn more about connecting to [OpenAI](https://theagentrouter.ai/docs/getting-started/connect-providers/openai) and adding your API key to the secret. You can use the same approach for other providers that support long lived credentials. ## Per-Request Credential Override By default, a `BackendSecurityPolicy` attaches one credential to every request. The optional `credentialOverride` field instead lets a trusted filter that runs ahead of the AI Gateway processor — typically an external authorization service wired in via `ext_authz` — supply the upstream credential for each request individually. This is how a multi-tenant gateway can bill every tenant against their own provider account while still sharing routes and backends. The per-request credential can come from one of two sources. ### From Dynamic Metadata (recommended) The trusted filter writes the credential into Envoy dynamic metadata, and the policy names the metadata namespace to read it from: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: BackendSecurityPolicy metadata: name: openai-per-tenant spec: targetRefs: - group: aigateway.envoyproxy.io kind: AIServiceBackend name: openai type: APIKey apiKey: secretRef: name: openai-fallback-key credentialOverride: fromDynamicMetadata: namespace: envoy.filters.http.ext_authz key: x-aigw-api-key ``` For `AWSCredentials` the metadata value must be a struct rather than a string, because SigV4 signing needs the whole credential (`sessionToken` is optional and omitted for long-lived credentials): ```yaml credentialOverride: fromDynamicMetadata: namespace: envoy.filters.http.ext_authz key: x-aigw-aws-credentials # { accessKeyId, secretAccessKey, sessionToken } ``` Envoy only forwards dynamic metadata namespaces the filter chain is configured to forward. The namespace the policy reads from must be declared in the [GatewayConfig](../gateway-config.md) referenced by the Gateway, under `extProc.metadataForwardingNamespaces`: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: GatewayConfig metadata: name: my-gateway-config spec: extProc: metadataForwardingNamespaces: - envoy.filters.http.ext_authz ``` The declared namespaces apply to all AI traffic of the Gateways referencing the config. The list is deliberately not derived from the policies that read it: a `BackendSecurityPolicy` is namespaced and can be written by a backend owner, while what Envoy hands to the external processor is the Gateway owner's decision. Envoy forwards exactly what that owner declared, so a policy naming an undeclared namespace behaves as if the metadata were absent, and the controller logs an error naming it. The declaration governs the Gateway that makes it, so attaching a route to a Gateway cannot widen what a different Gateway forwards. An external authorization service produces the metadata through the `dynamic_metadata` field of its `CheckResponse`. **Note:** Only **untyped** dynamic metadata is read. A filter that writes the credential as typed metadata (a `google.protobuf.Any` under the same namespace) produces no value, and the request falls back to the static credential or is rejected according to `fallbackToConfigured`. A policy reads a namespace Envoy forwards into the processor. `io.envoy.ai_gateway` travels the other way: the processor writes it and Envoy receives it, for token-based rate limiting and access logs. Dynamic metadata is the recommended source. Clients cannot forge it, since only filters in the gateway's own configuration can write it. And because the credential never appears as a request header, there is nothing to strip, nothing for access logs to capture, and nothing that can reach another backend on retries or fallback. ### From Request Headers Alternatively, the trusted filter can inject the credential as a request header, which the gateway reads and strips before forwarding the request to the backend it authenticated: ```yaml credentialOverride: fromRequestHeaders: {} # reads x-aigw-api-key for type: APIKey ``` Because any client can set request headers, this source is only safe when the trusted boundary overwrites or rejects client-supplied values of the configured header. The header also travels with the request through the filter chain, and stripping is configured per backend — so on routes with multiple backends (for example provider fallback), prefer `fromDynamicMetadata`, which has neither exposure. ### Fallback Behaviour `fallbackToConfigured` (default `true`) controls what happens when the source carries no credential: fall back to the static credential configured in the policy, or, when set to `false`, reject the request with a 401 without contacting the provider. A *partial* AWS credential — an access key ID without a secret access key, or the reverse — always fails the request rather than falling back, since it indicates a misconfigured credential producer. Falling back requires something to fall back *to*: if the policy has no static credential — for example `GCPCredentials` relying on Application Default Credentials with no credentials file or workload identity federation — the controller rejects `credentialOverride` unless `fallbackToConfigured: false` is set explicitly. When enabling `fromDynamicMetadata` on a backend already serving traffic, prefer starting with `fallbackToConfigured: true` and tightening to `false` afterwards: the processor configuration and Envoy's forwarding configuration are delivered on separate channels, and with `false` a request that lands in between is rejected with a 401. ## Conclusion Upstream Authentication is a key component of the Agent Router's security architecture. It ensures secure communication between the Gateway and upstream AI service providers while supporting modern authentication methods and enterprise security requirements. Leverage Agent Router's Upstream Authentication to maintain a secure and compliant AI infrastructure in your enterprise environments. --- # Traffic Handling Source: https://theagentrouter.ai/docs/capabilities/traffic/ This section provides information about traffic routing related capabilities in Agent Router, including [Quota Policy](./quota-policy.md) for controlling total token consumption budgets across time windows. --- # Usage-based Rate Limiting Source: https://theagentrouter.ai/docs/capabilities/traffic/usage-based-ratelimiting This guide focuses on AI Gateway's specific capabilities for token-based rate limiting in LLM requests. For general rate limiting concepts and configurations, refer to [Envoy Gateway's Rate Limiting documentation](https://gateway.envoyproxy.io/docs/tasks/traffic/global-rate-limit/). **Quota Policy vs. usage-based rate limiting:** `QuotaPolicy` manages the platform's upstream consumption budget for a provider backend and model across all routes and consumers. Usage-based rate limiting (this page) manages each client's allowance on a route. Both can count tokens and return `429 Too Many Requests`. See [Choosing a policy](./quota-policy.md#choosing-a-policy) for a comparison of enforcement behavior, scope, and time windows. ## Overview AI Gateway leverages Envoy Gateway's Global Rate Limit API to provide token-based rate limiting for LLM requests. Key features include: - Token usage tracking based on model and user identifiers - Configuration for tracking input, output, and total token metadata from LLM responses - Model-specific rate limiting using AI Gateway headers (`x-ai-eg-model`) which is inserted by the AI Gateway filter with the model name extracted from the request body. - Support for custom token cost calculations using CEL expressions ## Token Usage Behavior AI Gateway has specific behavior for token tracking and rate limiting: 1. **Token Extraction**: AI Gateway automatically extracts token usage from LLM responses that follow the OpenAI schema format. The token counts are stored in the metadata specified in your `llmRequestCosts` configuration. 2. **Rate Limit Timing**: Token usage is charged after the response completes. When a request is received: - Envoy checks the token budget using usage that has already been charged - If the existing charged usage is over the limit, the request is rejected with a 429 status code - If the request is admitted, AI Gateway extracts token usage from the completed response and charges it towards the total 3. **Token Types**: - `InputToken`: Counts tokens in the request prompt - `CachedInputToken`: Counts _cached_ input tokens in the request prompt - `OutputToken`: Counts tokens in the model's response - `TotalToken`: Combines both input and output tokens - `CEL`: Allows custom token calculations using CEL expressions 4. **Multiple Rate Limits**: You can configure multiple rate limit rules for the same user-model combination. For example: - Limit total tokens per hour - Separate limits for input and output tokens - Custom limits using CEL expressions 5. **Per Route Custom Token Calculation**: The `llmRequestCosts` defined in your `AIGatewayRoute` spec are scoped exclusively to that specific route. This means multiple `AIGatewayRoute` resources can define the exact same metadata keys, but the gateway will independently apply the correct calculations based on the route's name and namespace. To map the request to the correct calculation, the AI Gateway parses the route from the Envoy Gateway Metadata. If that metadata is not present, it falls back to parsing the route name from the route configuration, assuming Envoy Gateway has generated the name in the following format: `httproute///rule/`. **Note:** For model providers with OpenAI schema transformations (like AWS Bedrock), AI Gateway automatically captures token usage through its request/response transformer. This enables consistent token tracking and rate limiting across different AI services using a unified OpenAI-compatible format. ### Streaming responses For streaming responses, token usage is only known after the stream completes and the final usage data has been observed. AI Gateway does not interrupt an already admitted stream if that stream pushes the token count over the configured limit. Instead, the stream completes, its token usage is charged at stream end, and later matching requests are rejected with `429 Too Many Requests` until enough usage expires from the rate-limit window. For example, with a 1,000-token hourly limit, a request that is admitted while the bucket is empty can stream 1,200 tokens successfully. The bucket is then 200 tokens over the limit, and subsequent matching requests are limited until the window has enough capacity again. ## Configuration **Prerequisites:** Rate limiting requires two components to be configured: 1. **Redis Deployment**: A Redis instance must be running to store rate limit data. See the [redis.yaml example](https://github.com/theagentrouter/agent-router/blob/main/examples/token_ratelimit/redis.yaml) for a simple deployment. 2. **Envoy Gateway Configuration**: Envoy Gateway must be configured at installation time to enable rate limiting and point to your Redis instance. See [Envoy Gateway Installation Guide](../../getting-started/prerequisites.md#additional-features-rate-limiting-inferencepool-etc) ### 1. Configure Token Tracking AI Gateway automatically tracks token usage for each request. Configure which token counts you want to track in your `AIGatewayRoute`: ```yaml spec: llmRequestCosts: - metadataKey: llm_input_token type: InputToken # Counts tokens in the request - metadataKey: llm_cached_input_token type: CachedInputToken # Counts cached input tokens in the request prompt - metadataKey: llm_output_token type: OutputToken # Counts tokens in the response - metadataKey: llm_total_token type: TotalToken # Tracks combined usage ``` For advanced token calculations specific to your use case: ```yaml spec: llmRequestCosts: - metadataKey: custom_cost type: CEL cel: "(input_tokens - cached_input_tokens) + (cached_input_tokens * 0.1) + output_tokens * 1.5" # Example: Weight cached tokens less and weight output tokens more heavily ``` LLMRequestCosts can be defined on a per-route level. ### 2. Configure Rate Limits AI Gateway uses Envoy Gateway's Global Rate Limit API to configure rate limits. Rate limits should be defined using a combination of user and model identifiers to properly control costs at the model level. Configure this using a `BackendTrafficPolicy`: #### Example: Cost-Based Model Rate Limiting The following example demonstrates a common use case where different models have different token limits based on their costs. This is useful when: - You want to limit expensive models (like GPT-4) more strictly than cheaper ones - You need to implement different quotas for different tiers of service - You want to prevent cost overruns while still allowing flexibility with cheaper models ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: model-specific-token-limit-policy namespace: default spec: targetRefs: - name: envoy-ai-gateway-token-ratelimit kind: Gateway group: gateway.networking.k8s.io rateLimit: type: Global global: rules: # Rate limit rule for GPT-4: 1000 total tokens per hour per user # Stricter limit due to higher cost per token - clientSelectors: - headers: - name: x-tenant-id type: Distinct - name: x-ai-eg-model type: Exact value: gpt-4 limit: requests: 1000 # 1000 total tokens per hour unit: Hour cost: request: from: Number number: 0 # Set to 0 so only token usage counts response: from: Metadata metadata: namespace: io.envoy.ai_gateway key: llm_total_token # Uses total tokens from the responses # Rate limit rule for GPT-3.5: 5000 total tokens per hour per user # Higher limit since the model is more cost-effective - clientSelectors: - headers: - name: x-tenant-id type: Distinct - name: x-ai-eg-model type: Exact value: gpt-3.5-turbo limit: requests: 5000 # 5000 total tokens per hour (higher limit for less expensive model) unit: Hour cost: request: from: Number number: 0 # Set to 0 so only token usage counts response: from: Metadata metadata: namespace: io.envoy.ai_gateway key: llm_total_token # Uses total tokens from the response ``` **Warning:** When configuring rate limits: 1. Always set the request cost number to 0 to ensure only token usage counts towards the limit 2. Set appropriate limits for different models based on their costs and capabilities 3. Ensure both user and model identifiers are used in rate limiting rules ::: ## Per-Tenant Limits Without Per-Tenant Policies The rate limits above use a static limit value defined in the `BackendTrafficPolicy` (for example `requests: 1000`). If each tenant needs a different number, that forces a policy per tenant — and, because the policy attaches to routes, often a set of routes per tenant too. At a few thousand tenants that is the dominant cost of the configuration. Envoy can instead read the limit value from **dynamic metadata** on each request, so one shared policy serves every tenant and the tenant's own number arrives with the request. Envoy Gateway exposes this as `limit.fromMetadata` on the rate limit rule. Nothing about this requires AI Gateway. Any Envoy filter that writes dynamic metadata can supply the number, and the policy points directly at whatever namespace that filter writes to. AI Gateway's part is the other half — turning the LLM's token usage into the **cost** charged against the budget, which is covered above and is the piece nothing else can do. **Requires Envoy Gateway with `limit.fromMetadata`:** `limit.fromMetadata` was added in [envoyproxy/gateway#9216](https://github.com/envoyproxy/gateway/pull/9216). Older releases silently drop the field, which leaves the static `requests` in force. Check your Envoy Gateway version before relying on it. ### 1. Emit the limit from a preceding filter The limit must be a struct with `requests_per_unit` and `unit`, which is what Envoy's rate limit override reads. `unit` is one of `SECOND`, `MINUTE`, `HOUR`, `DAY`, `MONTH`, `YEAR`. With `ext_authz`, the auth server already resolves the tenant, so it can return the budget in the same Check response. Envoy files everything a Check response returns under the `envoy.filters.http.ext_authz` namespace: ```json { "dynamic_metadata": { "fields": { "total_limit": { "struct_value": { "fields": { "requests_per_unit": { "number_value": 100000 }, "unit": { "string_value": "HOUR" } } } } } } } ``` Dynamic metadata is deliberate here rather than a request header: it can only be written by filters in the chain, never by the downstream client, so a tenant cannot raise its own limit. No header stripping is needed to make it safe. ### 2. Read it in a `BackendTrafficPolicy` Point `limit.fromMetadata` at the namespace and key the filter wrote. The static `requests`/`unit` stay as the default that applies when the metadata is absent: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: tenant-token-budget namespace: default spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: envoy-ai-gateway rateLimit: type: Global global: rules: - clientSelectors: - headers: - name: x-tenant-id type: Distinct limit: requests: 1000 # default when ext_authz says nothing unit: Hour fromMetadata: namespace: envoy.filters.http.ext_authz # where ext_authz writes key: total_limit cost: request: { from: Number, number: 0 } # charge tokens, not requests response: from: Metadata metadata: namespace: io.envoy.ai_gateway key: llm_total_token # emitted by globalLLMRequestCosts ``` `Distinct` on `x-tenant-id` gives each tenant its own bucket, and `fromMetadata` gives each bucket its own size. One policy, arbitrary per-tenant numbers, no per-tenant objects. ### 3. Separate input, output and total budgets A tenant plan is usually stated as a token triplet rather than one number. Declare one cost key per token kind on the `GatewayConfig`: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: GatewayConfig metadata: name: envoy-ai-gateway namespace: default spec: globalLLMRequestCosts: - metadataKey: llm_input_token type: InputToken - metadataKey: llm_output_token type: OutputToken - metadataKey: llm_total_token type: TotalToken ``` Have the auth server return `input_limit`, `output_limit` and `total_limit`, then add one rule per kind, each pairing its limit key with the matching cost key. A tenant stops at whichever budget it exhausts first. ### Behavior to plan for | Metadata value | Effect | | -------------------------------------------- | ------------------------------------------------ | | a valid `{ requests_per_unit, unit }` struct | becomes the limit for that request | | absent | the static `requests`/`unit` on the rule applies | | `requests_per_unit: 0` | a real limit of `0` — see below | **Missing metadata fails open to the static default:** When the metadata is absent — the auth server didn't set it, or was bypassed — the request falls back to the static `requests`/`unit`. Because that direction is permissive, keep the static value a **conservative ceiling** rather than a placeholder. **A `requests_per_unit` of `0` suspends the tenant:** `0` is a real limit, and it is the deliberate way to suspend a tenant. Exactly when it starts denying depends on how you charge cost: - **Token cost** (`cost.request.from: Number, number: 0`, charged from the response) — the budget is checked before anything is charged, so the request already in flight is admitted and everything after it is denied. - **Request count** (no `cost` block, one unit per request) — denied immediately, including the first request. So a source meaning _"I have no opinion about this request"_ must **omit** the field entirely. Emitting `0` from an uninitialized field, a failed lookup, or a tenant record with no quota set suspends that tenant instead of falling back. **When the source can't produce the struct:** Some sources emit their own shape and can't be changed — `jwt_authn` writes claims verbatim, and a third-party auth sidecar may only return a string like `"100000/HOUR"`. Envoy's override reads only the `{ requests_per_unit, unit }` struct, so in that case you need a filter in between to transcode it, such as a small Lua filter writing the struct into its own metadata namespace and pointing `fromMetadata` there. ## Making Requests For proper cost control and rate limiting, requests must include: - `x-tenant-id`: Identifies the user making the request Example request: ```shell curl -H "Content-Type: application/json" \ -H "x-tenant-id: user123" \ -d '{ "model": "gpt-4", "messages": [ { "role": "user", "content": "Hello!" } ] }' \ $GATEWAY_URL/v1/chat/completions ``` --- # Provider Fallback Source: https://theagentrouter.ai/docs/capabilities/traffic/provider-fallback Agent Router supports provider fallback to ensure high availability and reliability for AI/LLM workloads. With fallback, you can configure multiple upstream providers for a single route, so that if the primary provider fails (due to network errors, 5xx responses, or other health check failures), traffic is automatically routed to a healthy fallback provider. ## When to Use Fallback - To ensure uninterrupted service when a primary AI/LLM provider is unavailable. - To provide redundancy across multiple cloud or on-premise model providers. - To implement active-active or active-passive failover strategies for critical AI workloads. ## How Fallback Works - **Primary and Fallback Backends:** You can specify a prioritized list of backends in your `AIGatewayRoute` using `backendRefs`. The first backend is treated as primary, and subsequent backends are considered fallbacks. - **Retry Policy:** Fallback is triggered based on retry policies, which can be configured using the [`BackendTrafficPolicy`](https://gateway.envoyproxy.io/contributions/design/backend-traffic-policy/) API. - **Automatic Failover:** When the primary backend becomes unhealthy, Agent Router automatically shifts traffic to the next healthy fallback backend. ## Example Below is an example configuration that demonstrates provider fallback from a failing upstream to AWS Bedrock: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: provider-fallback namespace: default spec: parentRefs: - name: provider-fallback kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: us.meta.llama3-2-1b-instruct-v1:0 backendRefs: - name: provider-fallback-always-failing-upstream # Primary backend (expected to fail) priority: 0 - name: provider-fallback-aws # Fallback backend priority: 1 ``` The corresponding `Backend` resources: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: Backend metadata: name: provider-fallback-always-failing-upstream namespace: default spec: endpoints: - fqdn: hostname: provider-fallback-always-failing-upstream.default.svc.cluster.local port: 443 --- apiVersion: gateway.envoyproxy.io/v1alpha1 kind: Backend metadata: name: provider-fallback-aws namespace: default spec: endpoints: - fqdn: hostname: bedrock-runtime.us-east-1.amazonaws.com port: 443 ``` ## Configuring Fallback Behavior Attach a `BackendTrafficPolicy` to the generated `HTTPRoute` to control retry behavior: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: provider-fallback spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: provider-fallback # HTTPRoute is created with the same name as AIGatewayRoute retry: # This ensures that only one attempt is made per priority. # For example, if the primary backend fails, it will not retry on the same backend. numAttemptsPerPriority: 1 numRetries: 5 perRetry: backOff: baseInterval: 100ms maxInterval: 10s timeout: 30s retryOn: httpStatusCodes: - 500 triggers: - connect-failure - retriable-status-codes ``` ## References - [Provider Fallback Example](https://github.com/theagentrouter/agent-router/tree/main/examples/provider_fallback) - [`BackendTrafficPolicy` API Design](https://gateway.envoyproxy.io/contributions/design/backend-traffic-policy/) --- # Quota Policy Source: https://theagentrouter.ai/docs/capabilities/traffic/quota-policy `QuotaPolicy` enables token-based quota management for AI inference services in Agent Router. When all related backend's quota are exceeded, requests are rejected with a `429 Too Many Requests` status code. **QuotaPolicy vs. usage-based rate limiting:** `QuotaPolicy` manages the platform's upstream consumption budget: how many tokens the gateway may spend against a provider backend and model across all routes and consumers. [Usage-based rate limiting](./usage-based-ratelimiting.md) manages consumer usage: how much a client may consume on a route. Both can count tokens and return `429 Too Many Requests`, but they answer different questions: whether the platform has exhausted its budget for a provider/model, or whether a client has exceeded its allowance. See [Choosing a policy](#choosing-a-policy) for details. ## Overview Key features of QuotaPolicy: - **Per-model token quotas** — assign token budgets to individual models served by an `AIServiceBackend`. - **CEL cost expressions** — weight input, output, cached, and reasoning tokens differently when computing how much a request burns down a quota. - **Client-selector bucket rules** — carve out per-tenant or per-header quotas using request attributes. - **Shadow mode** — evaluate quota rules without enforcing them, for safe rollout. ## Choosing a Policy Both policies use Envoy's rate limit protocol and Redis-backed counters, but `QuotaPolicy` is enforced by a dedicated AI Gateway rate limit service. Usage-based rate limiting uses Envoy Gateway's global rate limit service. Choose between them based on the budget's intent and scope: | | `QuotaPolicy` | Usage-based rate limiting | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | Configuration | One AI Gateway policy containing the token cost and quota buckets | `llmRequestCosts` on an `AIGatewayRoute`, plus an Envoy Gateway `BackendTrafficPolicy` | | Scope | Backend- and model-scoped - the budget follows an `AIServiceBackend` across every route that sends traffic to it | Route/Gateway-scoped - budgets are keyed by client descriptors on a route | | Token accounting | One default or custom CEL cost per model | Multiple metadata keys can track and limit input, output, total, or custom token costs separately | | Client budgets | Default and header-selected buckets, with optional shadow mode | Envoy Gateway client selectors and rate limit rules | | Enforcement | In `Shared` mode, a request is denied only when all applicable quota buckets are exhausted; the quota is evaluated for the selected backend | A request is denied when any matched rate limit is exceeded; limits are evaluated from the route's client descriptors | | Time windows | Exactly one second, minute, hour, or day | One second, minute, hour, day, month, or year | Use `QuotaPolicy` when the platform needs to protect its provider budget for a backend and model. For example, a platform team can give every tenant a separate daily budget for an expensive model, set a shared backend budget alongside those tenant budgets, and evaluate a new tenant rule in shadow mode. The policy stays with the `AIServiceBackend` even when multiple routes send traffic to it. Use usage-based rate limiting when you need to define each client's allowance on a route. It is also the natural choice when you need lower-level Envoy Gateway controls, route-specific token cost metadata, separate limits for input and output tokens, or a monthly or yearly window. ## How It Works 1. A `QuotaPolicy` is attached to one or more `AIServiceBackend` resources via `targetRefs`. 2. For each completed request, the token cost is computed using the configured cost expression (defaults to `total_tokens`). 3. The cost is charged against the matching quota bucket (the per-model default bucket, or a matching bucket rule). 4. When all related quota buckets for that model are exceeded, subsequent matching requests receive `429 Too Many Requests`. **Prerequisites:** Quota enforcement requires two components that are not deployed by the AI Gateway Helm chart today: 1. **Redis** stores quota counters. See the [redis.yaml example](https://github.com/theagentrouter/agent-router/blob/main/examples/token_ratelimit/redis.yaml) for a simple deployment. 2. **A dedicated rate limit service** evaluates the `ai-gateway-quota` domain. It must use the AI Gateway controller's xDS server for configuration, with node ID `envoy-ai-gateway-ratelimit`, and listen at the controller's `quotaRateLimitServiceAddr`. The [quota E2E manifest](https://github.com/theagentrouter/agent-router/blob/main/tests/e2e/testdata/backend_quota_ratelimit.yaml) provides a deployment example. Envoy Gateway's rate-limit addon is a separate service used by usage-based rate limiting. It is not required for a QuotaPolicy-only deployment, although both services can use the same Redis instance. By default, `controller.quotaRateLimitFailureModeDeny` is `false`. If the dedicated service is absent or unreachable, quota checks fail open and requests continue without enforcement. Set it to `true` if unavailable quota enforcement should reject requests instead. ## Configuration ### Per-Model Quotas Use `perModelQuotas` to apply a token budget to a specific model served by the targeted backend(s). **The model name must match the route:** A `perModelQuotas` entry only applies when its `modelName` matches the `modelNameOverride` set on the `AIGatewayRoute` rule's `backendRefs` for the targeted backend. If they do not match, the quota is silently **not** applied. Given an `AIGatewayRoute` that routes a model to the backend: ```yaml apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: AIGatewayRoute metadata: name: my-route spec: rules: - backendRefs: - name: my-backend modelNameOverride: my-model # <-- the QuotaPolicy modelName must match this ``` attach a `QuotaPolicy` to the backend: ```yaml apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: QuotaPolicy metadata: name: my-quota-policy spec: targetRefs: - group: aigateway.envoyproxy.io kind: AIServiceBackend name: my-backend perModelQuotas: - modelName: "my-model" quota: mode: Shared defaultBucket: limit: 10000 # Maximum tokens allowed in the window. duration: "1h" # Sliding window. ``` You can attach quotas for multiple models, each with its own budget: ```yaml perModelQuotas: - modelName: gpt-4 quota: defaultBucket: limit: 10000 # Strict limit for the expensive model. duration: "1h" - modelName: gpt-3.5-turbo quota: defaultBucket: limit: 100000 # Higher limit for the cost-effective model. duration: "1h" ``` **Note:** When multiple `QuotaPolicy` resources define the same model for the same `AIServiceBackend`, the policy whose namespace/name sorts alphabetically first takes precedence. ### Custom Cost Expression By default, a request's cost is its `total_tokens`. You can override this with a [CEL](https://github.com/google/cel-spec) expression that weights token types differently. The following variables are available in a `costExpression`: | Variable | Type | Description | | ----------------------------- | ------ | ------------------------------------------------ | | `input_tokens` | uint | Prompt / input tokens. | | `output_tokens` | uint | Completion / output tokens. | | `total_tokens` | uint | Total tokens (the default cost). | | `cached_input_tokens` | uint | Input tokens served from the provider's cache. | | `cache_creation_input_tokens` | uint | Input tokens charged for writing to the cache. | | `reasoning_tokens` | uint | Reasoning tokens (for reasoning-capable models). | | `model` | string | The resolved model name. | | `backend` | string | The serving backend name. | | `route_name` | string | The route name. | ```yaml perModelQuotas: - modelName: gpt-4 quota: # Cached input tokens count as 1/10 of a regular input token; # output tokens count 6x. costExpression: "input_tokens + cached_input_tokens / 10u + output_tokens * 6u" defaultBucket: limit: 50000 duration: "1h" ``` **Tip:** Use a custom cost expression when token types have significantly different costs with your provider — for example, output tokens are typically more expensive than input tokens. The token variables are **unsigned integers**, so numeric literals must carry a `u` suffix (for example `output_tokens * 6u`) and the expression must evaluate to a non-negative integer. Integer division truncates (`cached_input_tokens / 10u`); for an exact fractional weight, cast through floating point — for example `uint(double(cached_input_tokens) * 0.1)`. ### Bucket Mode The `mode` field on a per-model `quota` controls how the `defaultBucket` and matching `bucketRules` interact when a request matches one or more rules. Currently only **`Shared`** mode is supported (it is also the default, so the field can be omitted): - The request is charged to **all** matching `bucketRules` **and** the `defaultBucket`. - The request is allowed only if quota is available in **at least one** matching bucket. ```yaml perModelQuotas: - modelName: gpt-4 quota: mode: Shared # Default — may be omitted. defaultBucket: limit: 10000 duration: "1h" ``` **Note:** An exclusive bucket mode (charging matching rules **or** the default bucket, but not both) is planned but not yet available. The only accepted value today is `Shared`. ### Client Selectors with Bucket Rules Bucket rules let you carve out dedicated quotas for specific clients identified by request attributes such as headers. This is useful for multi-tenant deployments where each tenant needs its own budget. Because the mode is `Shared`, a request that matches a bucket rule is charged against **both** that rule's bucket **and** the `defaultBucket`. ```yaml perModelQuotas: - modelName: gpt-4 quota: defaultBucket: limit: 10000 # Shared budget across all tenants. duration: "1h" bucketRules: # Premium tenant gets a dedicated, higher per-tenant budget. - clientSelectors: - headers: - name: x-tenant-id type: Exact value: premium-tenant quota: limit: 50000 duration: "1h" ``` Use `type: Distinct` to create a separate bucket for every unique value of a header — for example, a per-tenant budget keyed on the tenant ID: ```yaml bucketRules: - clientSelectors: - headers: - name: x-tenant-id type: Distinct # One bucket per unique tenant ID. quota: limit: 5000 duration: "1h" ``` Supported header match types are `Exact`, `Distinct`, and `RegularExpression`. `clientSelectors` reuse the Envoy Gateway [`RateLimitSelectCondition`](https://gateway.envoyproxy.io/docs/api/extension_types/#ratelimitselectcondition) type, but QuotaPolicy currently applies only the `headers` matcher. The other fields on that type (`sourceCIDR`, `methods`, `path`, `queryParams`) are accepted by the schema but **not yet honored** for quota buckets. ### Shadow Mode Shadow mode lets you test a bucket rule without rejecting traffic. When `shadowMode` is enabled on a bucket rule, all quota checks are performed (cache lookups, counter updates, telemetry), but the outcome is never enforced — the request always succeeds even if the quota is exceeded. ```yaml bucketRules: - clientSelectors: - headers: - name: x-tenant-id type: Distinct quota: limit: 5000 duration: "1h" shadowMode: true # Evaluate but do not enforce. ``` **Tip:** Use shadow mode when rolling out a new quota rule. Monitor the telemetry to confirm the limit is set correctly, then enable enforcement by removing `shadowMode` (or setting it to `false`). Shadow mode is configured per bucket rule. It cannot be set on the `defaultBucket`. ## Duration Format The `duration` field selects the sliding-window size. It must be exactly one of the following values: | Value | Window | | ------ | ---------- | | `"1s"` | One second | | `"1m"` | One minute | | `"1h"` | One hour | | `"1d"` | One day | The window is fixed-size — arbitrary multiples such as `"30s"` or `"15m"` are **not** valid and will be rejected by the CRD schema. Choose the `limit` to express your budget within one of these windows. The one-day maximum is a current `QuotaPolicy` API restriction, not an inherent limitation of token accounting. Token-based rate limiting configured through an Envoy Gateway `BackendTrafficPolicy` also supports `Month` and `Year` units. Use [usage-based rate limiting](./usage-based-ratelimiting.md) when a token budget needs either of those longer windows. ## Service-Wide Quota **Not yet available:** The API exposes a backend-wide `serviceQuota` field — intended to apply a single budget across all models on a backend — but it is **not yet enforced** (it is currently a known TODO in the API). Configuring it has no effect on traffic, so use [Per-Model Quotas](#per-model-quotas) to enforce token quotas today. This section will document `serviceQuota` once enforcement lands. ## Full Example The following `QuotaPolicy` combines per-model quotas, custom cost expressions, bucket rules with client selectors, and shadow mode. ```yaml apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: QuotaPolicy metadata: name: full-quota-policy spec: targetRefs: - group: aigateway.envoyproxy.io kind: AIServiceBackend name: my-backend perModelQuotas: - modelName: gpt-4 quota: costExpression: "input_tokens + cached_input_tokens / 10u + output_tokens * 6u" mode: Shared defaultBucket: limit: 10000 duration: "1h" bucketRules: # Premium tenant gets a dedicated, higher quota. - clientSelectors: - headers: - name: x-tenant-id type: Exact value: premium-tenant quota: limit: 50000 duration: "1h" # Track per-tenant usage in shadow mode (no enforcement). - clientSelectors: - headers: - name: x-tenant-id type: Distinct quota: limit: 5000 duration: "1h" shadowMode: true - modelName: gpt-3.5-turbo quota: defaultBucket: limit: 100000 duration: "1h" ``` ## References - [API Reference](../../api/api.mdx) - [Usage-Based Rate Limiting](./usage-based-ratelimiting.md) - [Provider Fallback](./provider-fallback.md) --- # Header and Body Mutations Source: https://theagentrouter.ai/docs/capabilities/traffic/header-body-mutations Agent Router allows you to mutate HTTP headers and JSON request body fields before requests are sent to backends. This is useful for adding provider-specific headers, setting service tiers, or removing internal fields before forwarding to upstream providers. ## Use Cases - Add custom headers required by specific backends (e.g., organization IDs, API keys). - Set JSON body fields like `service_tier`, `max_tokens`, or `temperature` per backend. - Remove internal or sensitive fields before forwarding requests to upstream providers. - Override default values on a per-backend or per-route basis. ## Where Mutations Can Be Configured Mutations can be configured at two levels: - **AIServiceBackend** — Mutations defined here apply to **all requests** routed to this backend, regardless of which route matched. - **AIGatewayRoute backendRef** — Mutations defined here apply **only when traffic matches a specific route rule** and is sent to the referenced backend. **Precedence:** When both route-level and backend-level mutations are defined, **route-level takes precedence** over backend-level for conflicting operations. Non-conflicting operations from both levels are applied together. ## Header Mutations Use `headerMutation` to add, overwrite, or remove HTTP headers on the outgoing request. ### Setting Headers The `set` field overwrites an existing header or adds it if not present. A maximum of 16 entries is allowed. ```yaml headerMutation: set: - name: "x-custom-header" value: "my-value" ``` For example, given an incoming request with `my-header: foo`, the following configuration changes it to `my-header: bar`: ```yaml headerMutation: set: - name: "my-header" value: "bar" ``` ### Removing Headers The `remove` field removes the specified headers from the request. Header names are **case-insensitive** (per [RFC 2616](https://datatracker.ietf.org/doc/html/rfc2616#section-4.2)). A maximum of 16 entries is allowed. ```yaml headerMutation: remove: - "x-internal-header" - "x-debug-header" ``` ## Body Mutations Use `bodyMutation` to add, overwrite, or remove top-level JSON fields in the HTTP request body. **Warning:** Only **top-level** fields are currently supported. Nested field paths are not available. ### Setting Body Fields The `set` field overwrites an existing JSON field or adds it if not present. A maximum of 16 entries is allowed. ```yaml bodyMutation: set: - path: "service_tier" value: '"scale"' - path: "max_tokens" value: "4096" - path: "temperature" value: "0.7" ``` The `value` field is parsed as raw JSON. This means different value types require different formatting: | Type | Example `value` | Result in JSON body | | ------- | -------------------- | ------------------- | | String | `'"scale"'` | `"scale"` | | Number | `"42"` | `42` | | Boolean | `"true"` | `true` | | Object | `'{"key": "value"}'` | `{"key": "value"}` | | Array | `"[1, 2, 3]"` | `[1, 2, 3]` | | Null | `"null"` | `null` | **Tip:** String values require inner quotes. For example, to set a field to the string `"scale"`, use `'"scale"'` in YAML. Numeric and boolean values do not need inner quotes. ### Removing Body Fields The `remove` field removes the specified top-level fields from the request body. A maximum of 16 entries is allowed. ```yaml bodyMutation: remove: - "internal_flag" - "debug_mode" ``` ## Complete Examples ### Example 1: AIServiceBackend with Mutations This example configures an `AIServiceBackend` that adds a custom organization header and sets `service_tier` while removing an internal tracking field for all requests routed to this backend: ```yaml apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: AIServiceBackend metadata: name: my-openai-backend spec: schema: name: OpenAI backendRef: name: my-openai-backend kind: Backend group: gateway.envoyproxy.io headerMutation: set: - name: "x-custom-org" value: "my-org-id" bodyMutation: set: - path: "service_tier" value: '"scale"' remove: - "internal_tracking_id" ``` ### Example 2: AIGatewayRoute backendRef with Mutations This example configures route-level mutations that apply only when requests match a specific route rule. Here, requests for `gpt-4` get a premium header and an increased `max_tokens` value: ```yaml apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: AIGatewayRoute metadata: name: my-route spec: parentRefs: - name: my-gateway kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: gpt-4 backendRefs: - name: my-openai-backend headerMutation: set: - name: "x-route-specific" value: "premium" bodyMutation: set: - path: "max_tokens" value: "8192" ``` ## Precedence Rules **Note:** When both route-level and backend-level mutations are defined for the same backend: - **Route-level mutations take precedence** over backend-level mutations for conflicting operations (e.g., both setting the same header name or body field path). - **Non-conflicting operations from both levels are applied together.** For example, if the backend-level sets header `x-org` and the route-level sets header `x-tier`, both headers are added to the request. ::: ## References - [AIServiceBackend](../../api/api.mdx#aiservicebackend) - [AIGatewayRoute](../../api/api.mdx#aigatewayroute) --- # Model Name Virtualization Source: https://theagentrouter.ai/docs/capabilities/traffic/model-name-virtualization Agent Router provides an advanced model name virtualization capability that allows you to manage and route requests to different AI models seamlessly. This guide covers the key feature and configuration for model virtualization. ## Motivation It is not uncommon for multiple AI providers to offer a similar or identical model, such as Llama-3-70b, etc. However, each provider tends to have its own unique naming convention for the same model. For example, `Claude 4 Sonnet` is hosted both on GCP and AWS Bedrock, but they have different model names: - GCP: `claude-sonnet-4@20250514`, etc. - AWS Bedrock: `anthropic.claude-sonnet-4-20250514-v1:0` Even within the same provider, model names may vary based on deployment configurations or versions. For example, an OpenAI Platform request to `gpt-5-nano` might result in a response from `gpt-5-nano-2025-08-07`. Azure OpenAI uses deployment names in the URL path rather than model names in the request body, so you won't know the model in use until you get the response. From downstream GenAI applications' perspective, it is beneficial to have a unified model name that abstracts away these differences. We do this while still availing the authoritative model name that served a response. ## Provider-specific Virtualization AI providers handle model naming and execution differently to balance flexibility, determinism, and optimization. This section categorizes these behaviors into virtualization types, enabling Agent Router to abstract differences for unified downstream access while retaining executed model details where possible. - **Automatic Routing & Resolution**: Providers select and optimize models at runtime behind generic identifiers, returning the actual executed version for reproducibility and debugging. - **Static Model Execution**: Uses direct mapping to immutable identifiers or fixed versions for deterministic execution, ensuring requested models run exactly as specified, ideal for consistent embeddings. - **Deterministic Snapshot Mapping**: Resolves models via endpoint/URL with dated snapshots (no response model field), providing consistent cross-platform behavior through timestamped versions. - **URI-Based Resolution**: Ignores request body model field, using URI path for deployment-based routing while returning the actual versioned model name in responses. - **Third-Party Delegation**: Delegates embeddings to specialized third-party services with separate APIs, integrating ecosystem expertise while focusing on core capabilities. | Upstream | API Type | Virtualization Type | Request Model Example | Response Model Example | | ---------------- | ---------------- | ------------------------------ | ----------------------------------------- | -------------------------- | | **awsbedrock** | Chat Completions | Static Model Execution | `anthropic.claude-sonnet-4-20250514-v1:0` | N/A (no model field) | | **awsbedrock** | Embeddings | Static Model Execution | `amazon.titan-embed-text-v2:0` | N/A (no model field) | | **azureopenai** | Chat Completions | URI-Based Resolution | `{any-value}` (ignored) | `gpt-5-nano-2025-08-07` | | **azureopenai** | Embeddings | URI-Based Resolution | `{any-value}` (ignored) | `text-embedding-ada-002-2` | | **gcpanthropic** | Chat Completions | Deterministic Snapshot Mapping | `claude-sonnet-4@20250514` | N/A (no model field) | | **gcpanthropic** | Embeddings | Third-Party Delegation | `voyage-3.5` | N/A (no model field) | | **gcpvertexai** | Chat Completions | Deterministic Snapshot Mapping | `gemini-1.5-pro-002` | N/A (no model field) | | **gcpvertexai** | Embeddings | Static Model Execution | `text-embedding-004` | N/A (no model field) | | **openai** | Chat Completions | Automatic Routing & Resolution | `gpt-5-nano` | `gpt-5-nano-2025-08-07` | | **openai** | Embeddings | Static Model Execution | `text-embedding-3-small` | `text-embedding-3-small` | ### Azure OpenAI URI-Based Behavior Azure OpenAI has a unique approach where [the model field in the request JSON is completely ignored][azure-model-ignored]. The deployment name in the URI path determines which model is actually used, and the response returns the actual versioned model name: - **URI Format**: `https://{resource}.openai.azure.com/openai/deployments/{deployment-name}/chat/completions` - **Request Body**: `{"model": "anything"}` ← This value is ignored - **Response**: `{"model": "gpt-5-nano-2025-08-07"}` ← Returns the actual versioned model name This means [Azure OpenAI doesn't read the model field from the request JSON at all][azure-deployment-resolution] - it uses the deployment name from the URI path exclusively, but returns the real underlying versioned model identifier in the response. ### AWS Bedrock Static Execution AWS Bedrock uses static model execution with immutable versioned identifiers. The Converse API does not return a model field in responses. Each model ID like `anthropic.claude-sonnet-4-20250514-v1:0` represents a specific, frozen model version with no automatic updates or routing. ### GCP Anthropic Embeddings Delegation GCP Anthropic delegates embeddings to [Voyage AI's specialized embedding models][anthropic-embeddings-voyage] rather than providing Claude-native embeddings. Available models include: - `voyage-3.5` - Balanced performance embedding model - `voyage-3-large` - Best quality general-purpose embeddings - `voyage-code-3` - Optimized for code retrieval - `voyage-law-2` - Legal domain-specific embeddings This delegation approach allows Anthropic to focus on language model capabilities while leveraging domain expertise for embeddings. ### OpenAI Virtualization OpenAI performs automatic model routing where [generic identifiers like `gpt-5-nano` may route to different model versions][openai-model-routing] like `gpt-5-nano-2025-08-07` based on availability and optimization. ### GCP Provider Behavior Both GCP Anthropic and GCP Vertex AI use endpoint-based model resolution where [the model is specified in the URI path rather than returned in the response body][gcp-endpoint-resolution]. This provides deterministic behavior with timestamped snapshots. ## Virtualization with modelNameOverride API In our top level AIGatewayRoute configuration, you can specify a `modelNameOverride` inside [AIGatewayRouteBackendRef](https://theagentrouter.ai/api/api.mdx#aigatewayrouterulebackendref) on each route rule to override the model name that is sent to the upstream AI provider. This feature is primarily designed for scenarios where you want to dynamically change the model name based on the actual AI provider the request is being sent to. The example configuration looks like this: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: test-route spec: targetRefs: [...] rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: claude-4-sonnet backendRefs: - name: aws-backend modelNameOverride: anthropic.claude-sonnet-4-20250514-v1:0 weight: 50 - name: gcp-backend modelNameOverride: claude-sonnet-4@20250514 weight: 50 ``` This configuration allows downstream applications to use a unified model name `claude-4-sonnet` while splitting traffic between the AWS Bedrock and GCP AI providers based on the specified `modelNameOverride`. This is what the word "Virtualization" means in this context: abstracting away the differences in model names across different AI providers and providing a unified interface for downstream applications. It also can be thought of as "one-to-many" aliasing of model names, where one unified model name can map to multiple different model names on different providers depending on the routing path. ## Virtualization for fallback scenarios As we see in the [Provider Fallback](./provider-fallback) page, Agent Router allows you to fallback to a different AI provider if the primary one fails. However, sometimes we want to fallback to a different model on the same provider. For example, it is natural to set up the Agent Router in a way that if the primary expensive model fails (rate limit, etc), Envoy retries the request to a less expensive model on the same provider. More concretely, if the request to `gpt-5-nano` fails, we want to retry it with `gpt-5-nano-mini` on the same OpenAI provider. `modelNameOverride` can also be used in this scenario to achieve the desired behavior. The configuration would look like this: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute metadata: name: test-route spec: targetRefs: [...] rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: gpt-5-nano backendRefs: - name: openai-backend # This doesn't specify modelNameOverride, so it will use the default model name `gpt-5-nano` in the request. priority: 0 - name: openai-backend modelNameOverride: gpt-5-nano-mini priority: 1 ``` With this configuration, assuming the retry is properly configured as per the [Provider Fallback](./provider-fallback) page, if the request to `gpt-5-nano` fails, Agent Router will automatically retry the request to `gpt-5-nano-mini` on the same OpenAI provider without requiring any changes to the downstream application. --- [azure-model-ignored]: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/chatgpt [azure-deployment-resolution]: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/faq [anthropic-embeddings-voyage]: https://docs.anthropic.com/en/docs/build-with-claude/embeddings [openai-model-routing]: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/chatgpt [gcp-endpoint-resolution]: https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude --- # Compatibility Matrix Source: https://theagentrouter.ai/docs/compatibility This document provides compatibility information for Agent Router releases with their dependencies. | Agent Router | Envoy Gateway | Kubernetes | Gateway API | Inference Extension | Support Status | | ------------ | ----------------------------- | ---------- | ----------- | ------------------- | -------------- | | main | v1.9.2+ (Envoy Proxy v1.39.x) | v1.32+ | v1.6.1 | v1.6.2 | Development | | v1.2.x | v1.9.2+ (Envoy Proxy v1.39.x) | v1.32+ | v1.6.1 | v1.6.2 | Supported | | v1.1.x | v1.8.1+ (Envoy Proxy v1.38.x) | v1.32+ | v1.5.x | v1.6.0 | Supported | | v1.0.x | v1.8.1+ (Envoy Proxy v1.38.x) | v1.32+ | v1.5.x | v1.0.2 | Supported | | v0.7.x | v1.8.x+ (Envoy Proxy v1.38.x) | v1.32+ | v1.5.x | v1.0.2 | Supported | | v0.6.x | v1.7.x+ (Envoy Proxy v1.37.x) | v1.32+ | v1.4.x | v1.0.2 | Supported | | v0.5.x | v1.6.x+ (Envoy Proxy v1.35.x) | v1.32+ | v1.4.x | v1.0.2 | Supported | | others | N/A | N/A | N/A | N/A | End of Life | Note that "compatibility" means that these specific combinations have been tested and verified to work together. Other versions may work but are not officially supported. Please refer to our [Support Policy](https://github.com/theagentrouter/agent-router/blob/main/RELEASES.md#support-policy) for more details on how we manage releases and support for different versions. To upgrade to a new Agent Router version, make sure upgrade your dependencies accordingly to maintain compatibility, especially make sure that Envoy Gateway and Gateway API versions are up-to-date as per the compatibility matrix above. Then, upgrade the AI Gateway using the standard helm upgrade process. ## Upgrading and Migration ### Helm Upgrade Commands ```bash # 1. Upgrade CRDs helm upgrade aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \ --version \ envoy-ai-gateway-system < NEW_VERSION > -n # 2. Upgrade application helm upgrade aieg oci://docker.io/envoyproxy/ai-gateway-helm \ --version \ envoy-ai-gateway-system < NEW_VERSION > -n ``` ### Migrating from v1alpha1 to v1beta1 AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy, GatewayConfig, and MCPRoute support both v1alpha1 (deprecated) and v1beta1. When you upgrade: - **Existing v1alpha1 resources** continue to work. The API server can serve them via both v1alpha1 and v1beta1 endpoints. - **Storage version migration** is not automatic. To migrate existing resources to v1beta1 storage, you must manually re-apply them or use the storage migration API. - **New resources** should use `apiVersion: aigateway.envoyproxy.io/v1beta1`. - **QuotaPolicy** remains v1alpha1-only (no v1beta1 available). #### Migrating Storage Version Storage version migration requires manual action. After upgrading the CRDs, existing resources remain stored in v1alpha1 format until you explicitly trigger migration. The Kubernetes API server can serve these resources via both v1alpha1 and v1beta1 endpoints, but the underlying storage version in etcd won't change automatically. To migrate existing resources from v1alpha1 to v1beta1 storage version: **Re-apply resources** (recommended for small deployments) Update your manifests to use v1beta1 and re-apply them with `kubectl apply`. For more information about storage version migration in Kubernetes, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/storage-version/#migrating-to-a-different-storage-version). To adopt v1beta1 in your manifests, change the `apiVersion` field and re-apply: ```yaml # Before (v1alpha1, deprecated) apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: AIGatewayRoute # ... # After (v1beta1, preferred) apiVersion: aigateway.envoyproxy.io/v1beta1 kind: AIGatewayRoute # ... ``` --- # Terminology Source: https://theagentrouter.ai/docs/terminology This glossary provides definitions for key terms and concepts used in AI Gateway and GenAI traffic handling. ## Quick Reference | Term | Quick Definition | | ---------------- | -------------------------------------------------- | | GenAI Gateway | Gateway for managing AI model traffic | | Foundation Model | Base pre-trained AI model | | Token | Basic unit of text in LLM processing | | Token Usage | Monitoring and limiting model resource consumption | | Model Routing | Directing requests to appropriate models | | Prompt | Input text guiding AI model response | | Temperature | Control for model output randomness | ## Categories - [AI/ML Fundamentals](#ai-ml-fundamentals): Token, Prompt, Context Window, Temperature - [Inference Infrastructure](#inference-infrastructure): Inference Instance, Service, Model Provider - [Gateway Components](#gateway-components): GenAI Gateway, Gateway API Inference Extension - [Usage & Analytics](#usage--analytics): Usage Monitoring, Token Usage Limiting - [Model Types & Management](#model-types--management): Foundation Model, Fine-Tuned Model - [Content & Safety](#content--safety): Content Filtering ## AI/ML Fundamentals {#ai-ml-fundamentals} ### Context Window The maximum amount of text (in tokens) that a model can process in a single request. **Related**: [Token](#token) ### Prompt The input text that guides the AI model's response, including instructions, context, and specific queries. ### Temperature A parameter that controls the randomness/creativity of model outputs, typically ranging from 0 (deterministic) to 1 (more creative). ### Token The basic unit of text processing in LLMs, representing parts of words or characters. **Related**: [Context Window](#context-window) · [Token Cost](#token-cost) ### Token Cost The financial or resource cost associated with token usage in model requests. **Related**: [Token](#token) · [Rate of LLM Token Consumption](#rate-of-llm-token-consumption) ## Content & Safety {#content--safety} ### Content Filtering A mechanism to screen and moderate AI-generated content to ensure compliance with ethical standards, company policies, or regulatory requirements. ## Gateway Components {#gateway-components} ### Gateway API Inference Extension A Kubernetes SIG Network extension for Gateway API that provides specialized routing and load balancing capabilities for AI/ML workloads, handling traffic management at the level of inference instances. **Related**: [Inference Instance](#inference-instance) ### GenAI Gateway A specialized gateway solution designed to manage, monitor, and route traffic to Generative AI models. It provides capabilities such as load balancing, authorization, token usage monitoring, and integration with multiple model providers. **Related**: [Token](#token) · [Model Provider](#model-provider) ### Hybrid GenAI Gateway A GenAI Gateway configuration that supports both local inference instances and external cloud-based AI models, providing flexibility in deployment and cost management. **Related**: [GenAI Gateway](#genai-gateway) · [Inference Instance](#inference-instance) · [Model Provider](#model-provider) ## Inference Infrastructure {#inference-infrastructure} ### Inference Instance An individual compute resource or container used to run a machine learning model for generating AI outputs (inference). ### Inference Service A service that provides model inference capabilities, including model loading, input processing, inference execution, and output formatting. **Related**: [Inference Instance](#inference-instance) ### Model Endpoint The API endpoint provided by a specific AI model, whether hosted by a cloud provider, open-source solution, or private deployment. ### Model Provider Services providing AI model capabilities through APIs, which can be either first-party providers who develop their own models (like OpenAI, Anthropic) or third-party providers who host other companies' models (like AWS Bedrock, Azure OpenAI Service). ## Model Types & Management {#model-types--management} ### Fine-Tuned Model A version of a base Generative AI model that has been customized for specific tasks or domains using additional training data. **Related**: [Foundation Model](#foundation-model) ### Foundation Model Foundation models are large-scale, pre-trained AI models designed to handle a broad range of tasks. They are trained on extensive datasets and can be fine-tuned or adapted to specific use cases. **Related**: [Fine-Tuned Model](#fine-tuned-model) ### Model Routing A feature in GenAI Gateways that dynamically routes requests to specific models or model versions based on client configuration, use case requirements, or service level agreements. **Related**: [GenAI Gateway](#genai-gateway) ## Usage & Analytics {#usage--analytics} ### GenAI Usage Analytics The collection and analysis of data regarding how users interact with AI models via the GenAI Gateway, including token usage, request patterns, and latency metrics. **Related**: [GenAI Gateway](#genai-gateway) · [Token](#token) ### GenAI Usage Monitoring The tracking of resource consumption across different types of models, including token-based monitoring for LLMs, image resolution and compute resources for LVMs, and combined metrics for multimodal models. **Related**: [Token](#token) ### LLM Token Usage Limiting A mechanism to monitor and control the number of tokens processed by an LLM GenAI model, including input, output, and total token limits. **Related**: [Token](#token) · [GenAI Gateway](#genai-gateway) ### Rate of LLM Token Consumption The speed at which tokens are consumed by an AI model during processing. This metric is crucial for cost estimation and performance optimization. **Related**: [Token](#token) **Note:** This glossary is continuously evolving as the field of GenAI traffic handling develops. If you'd like to contribute or suggest changes, please visit our [GitHub repository](https://github.com/theagentrouter/agent-router). **See Also:** - Check our [Getting Started](./getting-started/index.md) guide for practical examples - Join our [Community Discord](https://discord.gg/xuxtPq43gZ) for discussions ::: --- # API Reference Source: https://theagentrouter.ai/docs/api/ ## Versions * [aigateway.envoyproxy.io/v1alpha1](#aigatewayenvoyproxyiov1alpha1) * [aigateway.envoyproxy.io/v1beta1](#aigatewayenvoyproxyiov1beta1) ## aigateway.envoyproxy.io/v1alpha1 Package v1alpha1 contains API schema definitions for the aigateway.envoyproxy.io API group. ## Resource Kinds ### Available Kinds - [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroute) - [AIGatewayRouteList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutelist) - [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackend) - [AIServiceBackendList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendlist) - [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicy) - [BackendSecurityPolicyList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicylist) - [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfig) - [GatewayConfigList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfiglist) - [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproute) - [MCPRouteList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutelist) - [QuotaPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicy) - [QuotaPolicyList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicylist) ### Kind Definitions #### AIGatewayRoute **Appears in:** - [AIGatewayRouteList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutelist) AIGatewayRoute combines multiple AIServiceBackends and attaching them to Gateway(s) resources. This serves as a way to define a "unified" AI API for a Gateway which allows downstream clients to use a single schema API to interact with multiple AI backends. Envoy AI Gateway will generate the following k8s resources corresponding to the AIGatewayRoute: - HTTPRoute of the Gateway API as a top-level resource to bind all backends. The name of the HTTPRoute is the same as the AIGatewayRoute. - HTTPRouteFilter of the Envoy Gateway API per namespace for automatic hostname rewrite. The name of the HTTPRouteFilter is `ai-eg-host-rewrite-${AIGatewayRoute.Name}`. All of these resources are created in the same namespace as the AIGatewayRoute. Note that this is the implementation detail subject to change. If you want to customize the default behavior of the Envoy AI Gateway, you can use these resources as a reference and create your own resources. Alternatively, you can use EnvoyPatchPolicy API of the Envoy Gateway to patch the generated resources. For example, you can configure the retry fallback behavior by attaching BackendTrafficPolicy API of Envoy Gateway to the generated HTTPRoute. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `AIGatewayRoute` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutespec), required): Spec defines the details of the AIGatewayRoute. - **status** ([AIGatewayRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutestatus), required): Status defines the status details of the AIGatewayRoute. #### AIGatewayRouteList AIGatewayRouteList contains a list of AIGatewayRoute. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `AIGatewayRouteList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroute) array, required): #### AIServiceBackend **Appears in:** - [AIServiceBackendList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendlist) AIServiceBackend is a resource that represents a single backend for AIGatewayRoute. A backend is a service that handles traffic with a concrete API specification. A AIServiceBackend is "attached" to a Backend which is either a k8s Service or a Backend resource of the Envoy Gateway. When a backend with an attached AIServiceBackend is used as a routing target in the AIGatewayRoute (more precisely, the HTTPRouteSpec defined in the AIGatewayRoute), the ai-gateway will generate the necessary configuration to do the backend specific logic in the final HTTPRoute. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `AIServiceBackend` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendspec), required): Spec defines the details of AIServiceBackend. - **status** ([AIServiceBackendStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendstatus), required): Status defines the status details of the AIServiceBackend. #### AIServiceBackendList AIServiceBackendList contains a list of AIServiceBackends. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `AIServiceBackendList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackend) array, required): #### BackendSecurityPolicy **Appears in:** - [BackendSecurityPolicyList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicylist) BackendSecurityPolicy specifies configuration for authentication and authorization rules on the traffic exiting the gateway to the backend. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `BackendSecurityPolicy` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec), required): - **status** ([BackendSecurityPolicyStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicystatus), required): Status defines the status details of the BackendSecurityPolicy. #### BackendSecurityPolicyList BackendSecurityPolicyList contains a list of BackendSecurityPolicy ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `BackendSecurityPolicyList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicy) array, required): #### GatewayConfig **Appears in:** - [GatewayConfigList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfiglist) GatewayConfig provides configuration for the AI Gateway external processor container that is deployed alongside the Gateway. A GatewayConfig is referenced by a Gateway via the annotation "aigateway.envoyproxy.io/gateway-config". The GatewayConfig must be in the same namespace as the Gateway that references it. This allows gateway-level configuration of the external processor, including environment variables (e.g., for tracing configuration) and resource requirements. Multiple Gateways can reference the same GatewayConfig to share configuration. Environment Variable Precedence: When merging environment variables, the following precedence applies (highest to lowest): 1. GatewayConfig.Spec.ExtProc.Kubernetes.Env (this resource) 2. Global controller flags (extProcExtraEnvVars) If the same environment variable name exists in both sources, the GatewayConfig value takes precedence. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `GatewayConfig` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigspec), required): Spec defines the configuration for the external processor. - **status** ([GatewayConfigStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigstatus), required): Status defines the status of the GatewayConfig. #### GatewayConfigList GatewayConfigList contains a list of GatewayConfig. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `GatewayConfigList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfig) array, required): #### MCPRoute **Appears in:** - [MCPRouteList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutelist) MCPRoute defines how to route MCP requests to the backend MCP servers. This serves as a way to define a "unified" AI API for a Gateway which allows downstream clients to use a single schema API to interact with multiple MCP backends. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `MCPRoute` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec), required): Spec defines the details of the MCPRoute. - **status** ([MCPRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutestatus), required): Status defines the status details of the MCPRoute. #### MCPRouteList MCPRouteList contains a list of MCPRoute. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `MCPRouteList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproute) array, required): #### QuotaPolicy **Appears in:** - [QuotaPolicyList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicylist) QuotaPolicy specifies token quota configuration for inference services. Generates rate limit configuration and tracks quota usage. Reject requests with 429 once all related quota to that request has been exceeded. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `QuotaPolicy` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([QuotaPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicyspec), required): - **status** ([QuotaPolicyStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicystatus), required): Status defines the status details of the QuotaPolicy. #### QuotaPolicyList QuotaPolicyList contains a list of QuotaPolicy ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1alpha1` of the API. - **kind** (String, required): This is a `QuotaPolicyList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([QuotaPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicy) array, required): ## Supporting Types ### Available Types - [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterule) - [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulebackendref) - [AIGatewayRouteRuleMatch](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulematch) - [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutespec) - [AIGatewayRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutestatus) - [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendspec) - [AIServiceBackendStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendstatus) - [APISchema](#github-com-envoyproxy-ai-gateway-api-v1alpha1-apischema) - [AWSCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1alpha1-awscredentialsfile) - [AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-awsoidcexchangetoken) - [AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-azureoidcexchangetoken) - [BackendSecurityPolicyAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyapikey) - [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyawscredentials) - [BackendSecurityPolicyAnthropicAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyanthropicapikey) - [BackendSecurityPolicyAzureAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyazureapikey) - [BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyazurecredentials) - [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicygcpcredentials) - [BackendSecurityPolicyOIDC](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyoidc) - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) - [BackendSecurityPolicyStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicystatus) - [BackendSecurityPolicyType](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicytype) - [GCPCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpcredentialsfile) - [GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpoidcexchangetoken) - [GCPServiceAccountImpersonationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpserviceaccountimpersonationconfig) - [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityfederationconfig) - [GCPWorkloadIdentityProvider](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityprovider) - [GatewayConfigExtProc](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigextproc) - [GatewayConfigForwardProxy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigforwardproxy) - [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigspec) - [GatewayConfigStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigstatus) - [HTTPBodyField](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpbodyfield) - [HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpbodymutation) - [HTTPHeaderMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpheadermutation) - [JWKS](#github-com-envoyproxy-ai-gateway-api-v1alpha1-jwks) - [JWTSource](#github-com-envoyproxy-ai-gateway-api-v1alpha1-jwtsource) - [LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1alpha1-llmrequestcost) - [LLMRequestCostType](#github-com-envoyproxy-ai-gateway-api-v1alpha1-llmrequestcosttype) - [MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationsource) - [MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationtarget) - [MCPBackendAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendapikey) - [MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendsecuritypolicy) - [MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendselector) - [MCPBackendSelectorRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendselectorrule) - [MCPHeaderForward](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpheaderforward) - [MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorization) - [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorizationrule) - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendref) - [MCPRouteBackendTrafficPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendtrafficpolicy) - [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteoauth) - [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutesecuritypolicy) - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) - [MCPRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutestatus) - [MCPToolFilter](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcptoolfilter) - [PerModelQuota](#github-com-envoyproxy-ai-gateway-api-v1alpha1-permodelquota) - [ProtectedResourceMetadata](#github-com-envoyproxy-ai-gateway-api-v1alpha1-protectedresourcemetadata) - [QuotaBucketMode](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotabucketmode) - [QuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotadefinition) - [QuotaPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicyspec) - [QuotaPolicyStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicystatus) - [QuotaRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotarule) - [QuotaValue](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotavalue) - [ServiceQuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-servicequotadefinition) - [ToolCall](#github-com-envoyproxy-ai-gateway-api-v1alpha1-toolcall) - [VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1alpha1-versionedapischema) ### Type Definitions #### AIGatewayRouteRule **Appears in:** - [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutespec) AIGatewayRouteRule is a rule that defines the routing behavior of the AIGatewayRoute. ##### Fields - **name** ([SectionName](#sigs-k8s-io-gateway-api-apis-v1-sectionname)): Name is the name of the route rule. This name must be unique within the route. When specified, it is copied to the generated HTTPRoute rule name. - **backendRefs** ([AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulebackendref) array): BackendRefs is the list of backends that this rule will route the traffic to. Each backend can have a weight that determines the traffic distribution. The namespace of each backend defaults to the same namespace as the AIGatewayRoute when not specified. Cross-namespace references are supported by specifying the namespace field. When a namespace different than the AIGatewayRoute's namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. BackendRefs can reference either AIServiceBackend resources (default) or InferencePool resources from the Gateway API Inference Extension. When referencing InferencePool resources: - Only one InferencePool backend is allowed per rule - Cannot mix InferencePool with AIServiceBackend references in the same rule - Fallback behavior is handled by the InferencePool's endpoint picker For AIServiceBackend references, you can achieve fallback behavior by configuring multiple backends combined with the BackendTrafficPolicy of Envoy Gateway. Please refer to https://gateway.envoyproxy.io/docs/tasks/traffic/failover/ as well as https://gateway.envoyproxy.io/docs/tasks/traffic/retry/. - **matches** ([AIGatewayRouteRuleMatch](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulematch) array): Matches is the list of AIGatewayRouteMatch that this rule will match the traffic to. This is a subset of the HTTPRouteMatch in the Gateway API. See for the details: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRouteMatch - **timeouts** ([HTTPRouteTimeouts](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#httproutetimeouts)): Timeouts defines the timeouts that can be configured for an HTTP request. If this field is not set, or the timeout.requestTimeout is nil, Envoy AI Gateway defaults to set 60s for the request timeout as opposed to 15s of the Envoy Gateway's default value. For streaming responses (like chat completions with stream=true), consider setting longer timeouts as the response may take time until the completion. Timeouts.Request acts as the maximum total time the gateway will wait for the entire response, including all streamed chunks. - **streamIdleTimeout** ([Duration](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.Duration)): StreamIdleTimeout is the maximum time Envoy will wait without receiving any bytes from the upstream. If the timer fires before the first response byte arrives, Envoy resets the upstream stream and a retry policy can fall over to the next backend. If it fires mid-stream after bytes have already arrived, the stream is cut and the client receives a 504. The AI Gateway extension server sets route.retry_policy.per_try_idle_timeout to this value on every xDS route generated from this rule before it is sent to the data plane. Pair this field with Timeouts.Request, which acts as the overall deadline. If this field is not set, no per-try idle timeout is applied. - **modelsOwnedBy** (string): ModelsOwnedBy represents the owner of the running models serving by the backends, which will be exported as the field of `OwnedBy` in openai-compatible API `/models`. This is used only when this rule contains `x-ai-eg-model` in its header matching where the header value will be recognized as a `model` in `/models` endpoint. All the matched models will share the same owner. Default to `Envoy AI Gateway` if not set. - **modelsCreatedAt** ([Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#time-v1-meta)): ModelsCreatedAt represents the creation timestamp of the running models serving by the backends, which will be exported as the field of `Created` in openai-compatible API `/models`. It follows the format of RFC 3339, for example `2024-05-21T10:00:00Z`. This is used only when this rule contains `x-ai-eg-model` in its header matching where the header value will be recognized as a `model` in `/models` endpoint. All the matched models will share the same creation time. Default to the creation timestamp of the AIGatewayRoute if not set. - **excludeFromModelsEndpoint** (boolean): ExcludeFromModelsEndpoint prevents models matched by this rule from being returned by the OpenAI-compatible `/models` endpoint. Requests still match this rule and apply its configured transformations, including model name overrides. This is useful for internal or backwards-compatible aliases, as well as duplicate model definitions with different schemas. #### AIGatewayRouteRuleBackendRef **Appears in:** - [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterule) AIGatewayRouteRuleBackendRef is a reference to a backend with a weight. It can reference either an AIServiceBackend or an InferencePool resource. ##### Fields - **name** (string, required): Name is the name of the backend resource. When Group and Kind are not specified, this refers to an AIServiceBackend. When Group and Kind are specified, this refers to the resource of the specified type. - **namespace** ([Namespace](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#namespace)): Namespace is the namespace of the backend resource. When unspecified (or empty string), this refers to the local namespace of the AIGatewayRoute. Note that when a namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. - **group** (string): Group is the group of the backend resource. When not specified, defaults to aigateway.envoyproxy.io (AIServiceBackend). Currently, only `inference.networking.k8s.io` is supported for InferencePool resources. - **kind** (string): Kind is the kind of the backend resource. When not specified, defaults to AIServiceBackend. Currently, only `InferencePool` is supported when Group is specified. - **modelNameOverride** (string): Name of the model in the backend. If provided this will override the name provided in the request. This field is ignored when referencing InferencePool resources. - **headerMutation** ([HTTPHeaderMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpheadermutation)): HeaderMutation defines the request header mutation to be applied to this backend. When both route-level and backend-level HeaderMutation are defined, route-level takes precedence over backend-level for conflicting operations. This field is ignored when referencing InferencePool resources. - **bodyMutation** ([HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpbodymutation)): BodyMutation defines the request body mutation to be applied to this backend. This allows modification of JSON fields in the request body before sending to the backend. When both route-level and backend-level BodyMutation are defined, route-level takes precedence over backend-level for conflicting operations. This field is ignored when referencing InferencePool resources. - **weight** (integer): Weight is the weight of the backend. This is exactly the same as the weight in the BackendRef in the Gateway API. See for the details: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.BackendRef Default is 1. - **priority** (integer): Priority is the priority of the backend. This sets the priority on the underlying endpoints. See: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/priority Note: This will override the `faillback` property of the underlying Envoy Gateway Backend This field is ignored when referencing InferencePool resources. Default is 0. #### AIGatewayRouteRuleMatch **Appears in:** - [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterule) ##### Fields - **headers** ([HTTPHeaderMatch](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#httpheadermatch) array): Headers specifies HTTP request header matchers. See HeaderMatch in the Gateway API for the details: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPHeaderMatch #### AIGatewayRouteSpec **Appears in:** - [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroute) AIGatewayRouteSpec details the AIGatewayRoute configuration. ##### Fields - **parentRefs** ([ParentReference](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#parentreference) array): ParentRefs are the names of the Gateway resources this AIGatewayRoute is being attached to. Currently, each reference's Kind must be Gateway. - **hostnames** (Hostname array): Hostnames is a list of hostnames matched against the HTTP Host header to select an AIGatewayRoute used to process the request. This is equivalent to the Hostnames field in the Gateway API HTTPRouteSpec. When specified, the generated HTTPRoute will include these hostnames for hostname-based filtering. See https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRouteSpec for the details of the Hostnames field in the Gateway API. - **rules** ([AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterule) array, required): Rules is the list of AIGatewayRouteRule that this AIGatewayRoute will match the traffic to. Each rule is a subset of the HTTPRoute in the Gateway API (https://gateway-api.sigs.k8s.io/api-types/httproute/). AI Gateway controller will generate a HTTPRoute based on the configuration given here with the additional modifications to achieve the necessary jobs, notably inserting the AI Gateway filter responsible for the transformation of the request and response, etc. In the matching conditions in the AIGatewayRouteRule, `x-ai-eg-model` header is available if we want to describe the routing behavior based on the model name. The model name is extracted from the request content before the routing decision. How multiple rules are matched is the same as the Gateway API. See for the details: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRoute At most 15 rules are allowed per AIGatewayRoute, corresponding to the Gateway API's limit on HTTPRoute.spec.rules (one slot is reserved for a controller-injected catch-all rule). To configure more rules on the same Gateway, split them across multiple AIGatewayRoute resources. - **llmRequestCosts** ([LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1alpha1-llmrequestcost) array): LLMRequestCosts specifies how to capture the cost of the LLM-related request, notably the token usage. The AI Gateway filter will capture each specified number and store it in the Envoy's dynamic metadata per HTTP request. The namespaced key is `io.envoy.ai_gateway`. These route-level costs override any global defaults defined in GatewayConfig.Spec.GlobalLLMRequestCosts for the same metadataKey. If a metadataKey is not defined in either place, no cost is calculated for it. This allows you to define common cost formulas once at the gateway level (e.g., via GatewayConfig) and only override them in specific routes when needed (e.g., premium routes with different pricing). For example, let's say we have the following LLMRequestCosts configuration: ```yaml llmRequestCosts: - metadataKey: llm_input_token type: InputToken - metadataKey: llm_output_token type: OutputToken - metadataKey: llm_total_token type: TotalToken - metadataKey: llm_cached_input_token type: CachedInputToken - metadataKey: llm_cache_creation_input_token type: CacheCreationInputToken ``` Then, with the following BackendTrafficPolicy of Envoy Gateway, you can have three rate limit buckets for each unique x-tenant-id header value. One bucket is for the input token, the other is for the output token, and the last one is for the total token. Each bucket will be reduced by the corresponding token usage captured by the AI Gateway filter. ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: some-example-token-rate-limit namespace: default spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: usage-rate-limit rateLimit: type: Global global: rules: - clientSelectors: # Do the rate limiting based on the x-tenant-id header. - headers: - name: x-tenant-id type: Distinct limit: # Configures the number of `tokens` allowed per hour. requests: 10000 unit: Hour cost: request: from: Number # Setting the request cost to zero allows to only check the rate limit budget, # and not consume the budget on the request path. number: 0 # This specifies the cost of the response retrieved from the dynamic metadata set by the AI Gateway filter. # The extracted value will be used to consume the rate limit budget, and subsequent requests will be rate limited # if the budget is exhausted. response: from: Metadata metadata: namespace: io.envoy.ai_gateway key: llm_input_token - clientSelectors: - headers: - name: x-tenant-id type: Distinct limit: requests: 10000 unit: Hour cost: request: from: Number number: 0 response: from: Metadata metadata: namespace: io.envoy.ai_gateway key: llm_output_token - clientSelectors: - headers: - name: x-tenant-id type: Distinct limit: requests: 10000 unit: Hour cost: request: from: Number number: 0 response: from: Metadata metadata: namespace: io.envoy.ai_gateway key: llm_total_token ``` Note that when multiple AIGatewayRoute resources are attached to the same Gateway, and different costs are configured for the same metadata key, each route's rule is carried in the filter configuration with the route identity; the data plane selects the matching rule per request (by route), so each route can define its own cost for the same metadata key. #### AIGatewayRouteStatus **Appears in:** - [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroute) AIGatewayRouteStatus contains the conditions by the reconciliation result. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array, required): Conditions is the list of conditions by the reconciliation result. Currently, at most one condition is set. Known .status.conditions.type are: `Accepted`, `NotAccepted`. #### AIServiceBackendSpec **Appears in:** - [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackend) AIServiceBackendSpec details the AIServiceBackend configuration. ##### Fields - **schema** ([VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1alpha1-versionedapischema), required): APISchema specifies the API schema of the output format of requests from Envoy that this AIServiceBackend can accept as incoming requests. Based on this schema, the ai-gateway will perform the necessary transformation for the pair of AIGatewayRouteSpec.APISchema and AIServiceBackendSpec.APISchema. This is required to be set. - **backendRef** ([BackendObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.BackendObjectReference), required): BackendRef is the reference to the Backend resource that this AIServiceBackend corresponds to. A backend must be a Backend resource of Envoy Gateway. Note that k8s Service will be supported as a backend in the future. See https://github.com/envoyproxy/ai-gateway/issues/902 for more details. This is required to be set. - **headerMutation** ([HTTPHeaderMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpheadermutation)): HeaderMutation defines the mutation of HTTP headers that will be applied to the request before sending it to the backend. - **bodyMutation** ([HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpbodymutation)): BodyMutation defines the mutation of HTTP request body JSON fields that will be applied to the request before sending it to the backend. #### AIServiceBackendStatus **Appears in:** - [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackend) AIServiceBackendStatus contains the conditions by the reconciliation result. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array, required): Conditions is the list of conditions by the reconciliation result. Currently, at most one condition is set. Known .status.conditions.type are: `Accepted`, `NotAccepted`. #### APISchema **Underlying type:** string **Appears in:** - [VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1alpha1-versionedapischema) APISchema defines the API schema. ##### Possible Values - **OpenAI** (enum): APISchemaOpenAI is the OpenAI schema. https://github.com/openai/openai-openapi - **Cohere** (enum): APISchemaCohere is the Cohere schema. https://docs.cohere.com/v2 - **AWSBedrock** (enum): APISchemaAWSBedrock is the AWS Bedrock schema. https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations_Amazon_Bedrock_Runtime.html - **AzureOpenAI** (enum): APISchemaAzureOpenAI APISchemaAzure is the Azure OpenAI schema. https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#api-specs - **GCPVertexAI** (enum): APISchemaGCPVertexAI is the schema followed by Gemini models hosted on GCP's Vertex AI platform. Note: Using this schema requires a BackendSecurityPolicy to be configured and attached, as the transformation will use the gcp-region and project-name from the BackendSecurityPolicy. https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.endpoints/generateContent?hl=en - **GCPAnthropic** (enum): APISchemaGCPAnthropic is the schema for Anthropic models hosted on GCP's Vertex AI platform. Returns native Anthropic format responses for seamless integration. https://docs.anthropic.com/en/api/claude-on-vertex-ai - **Anthropic** (enum): APISchemaAnthropic is the native Anthropic API schema. https://docs.claude.com/en/home - **AWSAnthropic** (enum): APISchemaAWSAnthropic is the schema for Anthropic models hosted on AWS Bedrock. Uses the native Anthropic Messages API format for requests and responses. When used with /v1/chat/completions endpoint, translates OpenAI format to Anthropic. When used with /v1/messages endpoint, passes through native Anthropic format. https://aws.amazon.com/bedrock/anthropic/ https://docs.claude.com/en/api/claude-on-amazon-bedrock - **AWSOpenAI** (enum): APISchemaAWSOpenAI is the OpenAI-compatible API schema provided by AWS. - **TypeSafe** (enum): APISchemaTypeSafe is the native TypeSafe AI schema used by the Jev System One model. Requests to /v1/systemone are passed through unchanged, and the version defaults to v1. https://docs.typesafe.ai/api.md #### AWSCredentialsFile **Appears in:** - [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyawscredentials) AWSCredentialsFile specifies the credentials file to use for the AWS provider. Envoy reads the secret file, and the profile to use is specified by the Profile field. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the credential file. The secret should contain the AWS credentials file keyed on `credentials`. - **profile** (string, required): Profile is the profile to use in the credentials file. #### AWSOIDCExchangeToken **Appears in:** - [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyawscredentials) AWSOIDCExchangeToken specifies credentials to obtain oidc token from a sso server. For AWS, the controller will query STS to obtain AWS AccessKeyId, SecretAccessKey, and SessionToken, and store them in a temporary credentials file. ##### Fields - **oidc** ([OIDC](https://gateway.envoyproxy.io/docs/api/extension_types/#oidc), required): OIDC is used to obtain oidc tokens via an SSO server which will be used to exchange for provider credentials. - **grantType** (string): GrantType is the method application gets access token. - **aud** (string): Aud defines the audience that this ID Token is intended for. - **awsRoleArn** (string, required): AwsRoleArn is the AWS IAM Role with the permission to use specific resources in AWS account which maps to the temporary AWS security credentials exchanged using the authentication token issued by OIDC provider. #### AzureOIDCExchangeToken **Appears in:** - [BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyazurecredentials) AzureOIDCExchangeToken specifies credentials to obtain oidc token from a sso server. For Azure, the controller will query Azure Entra ID to get an Azure Access Token, and store them in a secret. ##### Fields - **oidc** ([OIDC](https://gateway.envoyproxy.io/docs/api/extension_types/#oidc), required): OIDC is used to obtain oidc tokens via an SSO server which will be used to exchange for provider credentials. - **grantType** (string): GrantType is the method application gets access token. - **aud** (string): Aud defines the audience that this ID Token is intended for. #### BackendSecurityPolicyAPIKey **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) BackendSecurityPolicyAPIKey specifies the API key. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the secret containing the API key. ai-gateway must be given the permission to read this secret. The key of the secret should be `apiKey`. #### BackendSecurityPolicyAWSCredentials **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) BackendSecurityPolicyAWSCredentials contains the supported authentication mechanisms to access AWS. When neither CredentialsFile nor OIDCExchangeToken is specified, the AWS SDK's default credential chain will be used. This automatically supports various authentication methods in the following order: 1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) 2. EKS Pod Identity - automatically rotates credentials for pods in EKS clusters 3. IAM Roles for Service Accounts (IRSA) - injects credentials via mounted service account tokens 4. EC2 instance metadata (IAM instance roles) 5. ECS task roles The default credential chain is recommended for Kubernetes deployments as it supports automatic credential rotation without manual configuration. Credentials are refreshed automatically when they approach expiration (typically hourly for IRSA and Pod Identity). ##### Fields - **region** (string, required): Region specifies the AWS region associated with the policy. - **credentialsFile** ([AWSCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1alpha1-awscredentialsfile)): CredentialsFile specifies the credentials file to use for the AWS provider. When specified, this takes precedence over the default credential chain. - **oidcExchangeToken** ([AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-awsoidcexchangetoken)): OIDCExchangeToken specifies the oidc configurations used to obtain an oidc token. The oidc token will be used to obtain temporary credentials to access AWS. When specified, this takes precedence over the default credential chain. #### BackendSecurityPolicyAnthropicAPIKey **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) BackendSecurityPolicyAnthropicAPIKey specifies the Anthropic API key. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the secret containing the Anthropic API key. ai-gateway must be given the permission to read this secret. The key of the secret should be `apiKey`. #### BackendSecurityPolicyAzureAPIKey **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) BackendSecurityPolicyAzureAPIKey specifies the Azure OpenAI API key. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the secret containing the Azure API key. ai-gateway must be given the permission to read this secret. The key of the secret should be `apiKey`. #### BackendSecurityPolicyAzureCredentials **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) BackendSecurityPolicyAzureCredentials contains the supported authentication mechanisms to access Azure. Only one of ClientSecretRef or OIDCExchangeToken must be specified. Credentials will not be generated if neither are set. ##### Fields - **clientID** (string, required): ClientID is a unique identifier for an application in Azure. - **tenantID** (string, required): TenantId is a unique identifier for an Azure Active Directory instance. - **clientSecretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference)): ClientSecretRef is the reference to the secret containing the Azure client secret. ai-gateway must be given the permission to read this secret. The key of secret should be `client-secret`. - **oidcExchangeToken** ([AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-azureoidcexchangetoken)): OIDCExchangeToken specifies the oidc configurations used to obtain an oidc token. The oidc token will be used to obtain temporary credentials to access Azure. #### BackendSecurityPolicyGCPCredentials **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) BackendSecurityPolicyGCPCredentials contains the supported authentication mechanisms to access GCP. ##### Fields - **projectName** (string, required): ProjectName is the GCP project name. - **region** (string, required): Region is the GCP region associated with the policy. - **credentialsFile** ([GCPCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpcredentialsfile)): CredentialsFile specifies the service account credentials file to use for the GCP provider. - **workloadIdentityFederationConfig** ([GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityfederationconfig)): WorkloadIdentityFederationConfig is the configuration for the GCP Workload Identity Federation. #### BackendSecurityPolicyOIDC **Appears in:** - [AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-awsoidcexchangetoken) - [AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-azureoidcexchangetoken) - [GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpoidcexchangetoken) - [GCPWorkloadIdentityProvider](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityprovider) BackendSecurityPolicyOIDC specifies OIDC related fields. ##### Fields - **oidc** ([OIDC](https://gateway.envoyproxy.io/docs/api/extension_types/#oidc), required): OIDC is used to obtain oidc tokens via an SSO server which will be used to exchange for provider credentials. - **grantType** (string): GrantType is the method application gets access token. - **aud** (string): Aud defines the audience that this ID Token is intended for. #### BackendSecurityPolicySpec **Appears in:** - [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicy) BackendSecurityPolicySpec specifies authentication rules on access the provider from the Gateway. Only one mechanism to access a backend(s) can be specified. Only one type of BackendSecurityPolicy can be defined. ##### Fields - **targetRefs** ([LocalPolicyTargetReference](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1alpha2.LocalPolicyTargetReference) array): TargetRefs are the names of the AIServiceBackend or InferencePool resources this BackendSecurityPolicy is being attached to. Attaching multiple BackendSecurityPolicies to the same resource is invalid and will result in an error during the reconciliation of the resource. - **type** ([BackendSecurityPolicyType](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicytype), required): Type specifies the type of the backend security policy. - **apiKey** ([BackendSecurityPolicyAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyapikey)): APIKey is a mechanism to access a backend(s). The API key will be injected into the Authorization header. - **awsCredentials** ([BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyawscredentials)): AWSCredentials is a mechanism to access a backend(s). AWS specific logic will be applied. - **azureAPIKey** ([BackendSecurityPolicyAzureAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyazureapikey)): AzureAPIKey is a mechanism to access Azure OpenAI backend(s). The API key will be injected into the api-key header. - **azureCredentials** ([BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyazurecredentials)): AzureCredentials is a mechanism to access a backend(s). Azure OpenAI specific logic will be applied. - **gcpCredentials** ([BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicygcpcredentials)): GCPCredentials is a mechanism to access a backend(s). GCP specific logic will be applied. - **anthropicAPIKey** ([BackendSecurityPolicyAnthropicAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyanthropicapikey)): AnthropicAPIKey is a mechanism to access Anthropic backend(s). The API key will be injected into the `x-api-key` header. https://docs.claude.com/en/api/overview#authentication #### BackendSecurityPolicyStatus **Appears in:** - [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicy) BackendSecurityPolicyStatus contains the conditions by the reconciliation result. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array, required): Conditions is the list of conditions by the reconciliation result. Currently, at most one condition is set. Known .status.conditions.type are: `Accepted`, `NotAccepted`. #### BackendSecurityPolicyType **Underlying type:** string **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) BackendSecurityPolicyType specifies the type of auth mechanism used to access a backend. ##### Possible Values - **APIKey** (enum): - **AWSCredentials** (enum): - **AzureAPIKey** (enum): - **AnthropicAPIKey** (enum): - **AzureCredentials** (enum): - **GCPCredentials** (enum): #### GCPCredentialsFile **Appears in:** - [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicygcpcredentials) GCPCredentialsFile specifies the service account key json file to authenticate with GCP provider. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the credential file. The secret should contain the GCP service account credentials file keyed on `service_account.json`. #### GCPOIDCExchangeToken **Appears in:** - [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityfederationconfig) ##### Fields - **oidc** ([OIDC](https://gateway.envoyproxy.io/docs/api/extension_types/#oidc), required): OIDC is used to obtain oidc tokens via an SSO server which will be used to exchange for provider credentials. - **grantType** (string): GrantType is the method application gets access token. - **aud** (string): Aud defines the audience that this ID Token is intended for. #### GCPServiceAccountImpersonationConfig **Appears in:** - [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityfederationconfig) ##### Fields - **serviceAccountName** (string, required): ServiceAccountName is the name of the service account to impersonate. #### GCPWorkloadIdentityFederationConfig **Appears in:** - [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicygcpcredentials) ##### Fields - **projectID** (string, required): ProjectID is the GCP project ID. - **workloadIdentityProviderName** (string, required): WorkloadIdentityProviderName is the name of the external identity provider as registered on Google Cloud Platform. - **oidcExchangeToken** ([GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpoidcexchangetoken), required): OIDCExchangeToken specifies the oidc configurations used to obtain an oidc token. The oidc token will be used to obtain temporary credentials to access GCP. - **workloadIdentityPoolName** (string, required): WorkloadIdentityPoolName is the name of the workload identity pool defined in GCP. https://cloud.google.com/iam/docs/workload-identity-federation?hl=en - **serviceAccountImpersonation** ([GCPServiceAccountImpersonationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpserviceaccountimpersonationconfig)): ServiceAccountImpersonation is the service account impersonation configuration. This is used to impersonate a service account when getting access token. #### GatewayConfigExtProc **Appears in:** - [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigspec) GatewayConfigExtProc holds runtime-specific configuration for the external processor. ##### Fields - **kubernetes** ([KubernetesContainerSpec](https://gateway.envoyproxy.io/docs/api/extension_types/#kubernetescontainerspec)): Kubernetes defines the configuration for running the external processor as a Kubernetes container. #### GatewayConfigForwardProxy **Appears in:** - [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigspec) GatewayConfigForwardProxy configures an HTTP CONNECT forward proxy for upstream egress. ##### Fields - **address** (string, required): Address is the `host:port` of the HTTP CONNECT proxy. The host may be a hostname or an IP address; the port is required. Upstream connections are tunnelled through this proxy via Envoy's http_11_proxy transport socket, preserving the upstream TLS session. #### GatewayConfigSpec **Appears in:** - [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfig) GatewayConfigSpec defines the configuration for the AI Gateway. ##### Fields - **extProc** ([GatewayConfigExtProc](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigextproc)): ExtProc defines the configuration for the external processor container. - **globalLLMRequestCosts** ([LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1alpha1-llmrequestcost) array): GlobalLLMRequestCosts defines default LLM request costs that apply to all routes referencing this GatewayConfig. These costs can be overridden on a per-route basis via AIGatewayRoute.Spec.LLMRequestCosts. When a request matches a route, the cost calculation proceeds as follows: 1. If the route defines LLMRequestCosts with a matching metadataKey, use that. 2. Otherwise, fall back to the global cost with that metadataKey (if defined here). 3. If neither exists, the cost is not calculated for that metadataKey. This allows you to define common cost formulas once at the gateway level (e.g., billing_charges = input_tokens + output_tokens) and only override them in specific routes when needed (e.g., premium routes with different pricing). - **forwardProxy** ([GatewayConfigForwardProxy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigforwardproxy)): ForwardProxy routes all upstream AI/LLM traffic from Gateways referencing this GatewayConfig through an HTTP CONNECT forward proxy. This is intended for data planes in locked-down networks where direct egress to providers is not permitted and all outbound traffic must traverse a proxy. #### GatewayConfigStatus **Appears in:** - [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfig) GatewayConfigStatus defines the observed state of GatewayConfig. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array): Conditions describe the current conditions of the GatewayConfig. #### HTTPBodyField **Appears in:** - [HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpbodymutation) HTTPBodyField represents a JSON field name and value for body mutation ##### Fields - **path** (string, required): Path is the top-level field name to set in the request body. Examples: `service_tier`, `max_tokens`, `temperature` - **value** (string, required): Value is the JSON value to set at the specified field. This can be any valid JSON value: string, number, boolean, object, array, or null. The value will be parsed as JSON and inserted at the specified field. Examples: - `\`scale\`` (string) - `42` (number) - `true` (boolean) - `\{\`key\`: \`value\`\}` (object) - `[1, 2, 3]` (array) - `null` (null) #### HTTPBodyMutation **Appears in:** - [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulebackendref) - [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendspec) HTTPBodyMutation defines the mutation of HTTP request body JSON fields that will be applied to the request ##### Fields - **set** ([HTTPBodyField](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpbodyfield) array): Set overwrites/adds the request body with the given JSON field (name, value) before sending to the backend. Only top-level fields are currently supported. Input: \{ `model`: `gpt-4`, `service_tier`: `default` \} Config: set: - path: `service_tier` value: `scale` Output: \{ `model`: `gpt-4`, `service_tier`: `scale` \} - **remove** (string array): Remove the given JSON field(s) from the HTTP request body before sending to the backend. The value of Remove is a list of top-level field names to remove. Input: \{ `model`: `gpt-4`, `service_tier`: `default`, `internal_flag`: true \} Config: remove: [`service_tier`, `internal_flag`] Output: \{ `model`: `gpt-4` \} #### HTTPHeaderMutation **Appears in:** - [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulebackendref) - [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendspec) HTTPHeaderMutation defines the mutation of HTTP headers that will be applied to the request ##### Fields - **set** ([HTTPHeader](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#httpheader) array): Set overwrites/adds the request with the given header (name, value) before the action. Input: GET /foo HTTP/1.1 my-header: foo Config: set: - name: `my-header` value: `bar` Output: GET /foo HTTP/1.1 my-header: bar - **remove** (string array): Remove the given header(s) from the HTTP request before the action. The value of Remove is a list of HTTP header names. Note that the header names are case-insensitive (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). Input: GET /foo HTTP/1.1 my-header1: foo my-header2: bar my-header3: baz Config: remove: [`my-header1`, `my-header3`] Output: GET /foo HTTP/1.1 my-header2: bar #### JWKS **Appears in:** - [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteoauth) JWKS defines how to obtain JSON Web Key Sets (JWKS) either from a remote HTTP/HTTPS endpoint or from a local source. ##### Fields - **remoteJWKS** ([RemoteJWKS](https://gateway.envoyproxy.io/docs/api/extension_types/#remotejwks)): RemoteJWKS defines how to fetch and cache JSON Web Key Sets (JWKS) from a remote HTTP/HTTPS endpoint. - **localJWKS** ([LocalJWKS](https://gateway.envoyproxy.io/docs/api/extension_types/#localjwks)): LocalJWKS defines how to get the JSON Web Key Sets (JWKS) from a local source. #### JWTSource **Appears in:** - [MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationsource) JWTSource defines the MCP authorization source for JWT tokens. At least one of scopes or claims must be provided. Scopes and claims are AND-ed: when both are specified, both sets must match. ##### Fields - **scopes** (JWTScope array): Scopes defines the list of JWT scopes required for the rule. If multiple scopes are specified, all scopes must be present in the JWT for the rule to match. - **claims** (JWTClaim array): Claims defines the list of JWT claims required for the rule. Each claim must exist on the token and have at least one of the expected values. Use to enforce tenant or subject-based access. #### LLMRequestCost **Appears in:** - [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutespec) - [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigspec) LLMRequestCost configures each request cost. ##### Fields - **metadataKey** (string, required): MetadataKey is the key of the metadata to store this cost of the request. - **type** ([LLMRequestCostType](#github-com-envoyproxy-ai-gateway-api-v1alpha1-llmrequestcosttype), required): Type specifies the type of the request cost. The default is `OutputToken`, and it uses `output token` as the cost. The other types are `InputToken`, `TotalToken`, `CachedInputToken`, `CacheCreationInputToken`, `ReasoningToken`, and `CEL`. - **cel** (string): CEL is the CEL expression to calculate the cost of the request. The CEL expression must return a signed or unsigned integer. If the return value is negative, it will be error. The expression can use the following variables: * model: the model name extracted from the request content. Type: string. * backend: the backend name in the form of `name.namespace`. Type: string. * input_tokens: the number of input tokens. Type: unsigned integer. * cached_input_tokens: the number of cached read input tokens. Type: unsigned integer. * cache_creation_input_tokens: the number of cache creation input tokens. Type: unsigned integer. * output_tokens: the number of output tokens. Type: unsigned integer. * total_tokens: the total number of tokens. Type: unsigned integer. * reasoning_tokens: the number of reasoning tokens. Type: unsigned integer. For example, the following expressions are valid: * `model == 'llama' ? input_tokens + output_token * 0.5 : total_tokens` * `backend == 'foo.default' ? input_tokens + output_tokens : total_tokens` * `backend == 'bar.default' ? (input_tokens - cached_input_tokens) + cached_input_tokens * 0.1 + cache_creation_input_tokens * 1.25 + output_tokens : total_tokens` * `input_tokens + output_tokens + total_tokens` * `input_tokens * output_tokens` #### LLMRequestCostType **Underlying type:** string **Appears in:** - [LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1alpha1-llmrequestcost) LLMRequestCostType specifies the type of the LLMRequestCost. ##### Possible Values - **InputToken** (enum): LLMRequestCostTypeInputToken is the cost type of the input token. - **CachedInputToken** (enum): LLMRequestCostTypeCachedInputToken is the cost type of the cached input token. - **CacheCreationInputToken** (enum): LLMRequestCostTypeCacheCreationInputToken is the cost type of the cached input token. - **OutputToken** (enum): LLMRequestCostTypeOutputToken is the cost type of the output token. - **TotalToken** (enum): LLMRequestCostTypeTotalToken is the cost type of the total token. - **ReasoningToken** (enum): LLMRequestCostTypeReasoningToken is the cost type of the reasoning token. - **CEL** (enum): LLMRequestCostTypeCEL is for calculating the cost using the CEL expression. #### MCPAuthorizationSource **Appears in:** - [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorizationrule) MCPAuthorizationSource defines the source of an authorization rule. ##### Fields - **jwt** ([JWTSource](#github-com-envoyproxy-ai-gateway-api-v1alpha1-jwtsource), required): JWT defines the JWT scopes required for this rule to match. #### MCPAuthorizationTarget **Appears in:** - [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorizationrule) MCPAuthorizationTarget defines the target of an authorization rule. ##### Fields - **tools** ([ToolCall](#github-com-envoyproxy-ai-gateway-api-v1alpha1-toolcall) array, required): Tools defines the list of tools this rule applies to. #### MCPBackendAPIKey **Appears in:** - [MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendsecuritypolicy) MCPBackendAPIKey defines the configuration for the API Key Authentication to a backend. When both `header` and `queryParam` are unspecified, the API key will be injected into the "Authorization" header by default. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference)): secretRef is the Kubernetes secret which contains the API keys. The key of the secret should be `apiKey`. - **inline** (string): Inline contains the API key as an inline string. - **header** (string): Header is the HTTP header to inject the API key into. If not specified, defaults to `Authorization`. When the header is `Authorization`, the injected header value will be prefixed with `Bearer `. Either one of Header or QueryParam can be specified to inject the API key. - **queryParam** (string): QueryParam is the HTTP query parameter to inject the API key into. For example, if QueryParam is set to `api_key`, and the API key is `mysecretkey`, the request URL will be modified to include `?api_key=mysecretkey`. Either one of Header or QueryParam can be specified to inject the API key. #### MCPBackendSecurityPolicy **Appears in:** - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendref) MCPBackendSecurityPolicy defines the security policy for a backend MCP server. ##### Fields - **apiKey** ([MCPBackendAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendapikey)): APIKey is a mechanism to access a backend. The API key will be injected into the request headers. #### MCPBackendSelector **Appears in:** - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) MCPBackendSelector defines which of a MCPRoute's backends a request may fan out to. ##### Fields - **defaultAction** ([AuthorizationAction](#github-com-envoyproxy-gateway-api-v1alpha1-authorizationaction)): DefaultAction is the action to take when no rules match. If unspecified, defaults to Deny. - **rules** ([MCPBackendSelectorRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendselectorrule) array): Rules defines a list of backend selection rules. These rules are evaluated in order, the first matching rule will be applied, and the rest will be skipped. If no rules are defined, DefaultAction is applied to every candidate backend (defaults to Deny). #### MCPBackendSelectorRule **Appears in:** - [MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendselector) MCPBackendSelectorRule defines a single backend selection rule. ##### Fields - **cel** (string): CEL specifies a Common Expression Language (CEL) expression, evaluated once for each backend already listed in backendRefs. It does not parse a list of backends out of the request. Each evaluation binds request.mcp.backend to the name of the one candidate backend under test, so the expression should answer `is this backend allowed`, not `which backends should be used`. The expression must return a boolean; evaluation errors or non-boolean results are treated as `no match`. A match means the rule applies, and Action below decides whether that's an Allow or a Deny for that backend. Example CEL expressions: * `request.mcp.backend in request.auth.jwt.claims.mcp_backends` * `(`,` + request.headers[`x-ai-eg-mcp-backend-subset`] + `,`).contains(`,` + request.mcp.backend + `,`)` Available attributes are the same as documented on MCPRouteAuthorizationRule.CEL, except request.mcp.method, request.mcp.tool, and request.mcp.params are not populated (no MCP method has been selected yet at backend-selection time). Note: The CEL expression support is experimental, and the attributes available to the expression may change in future releases. - **action** ([AuthorizationAction](#github-com-envoyproxy-gateway-api-v1alpha1-authorizationaction)): Action is the decision for matching backends. If unspecified, defaults to Allow. #### MCPHeaderForward **Appears in:** - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendref) MCPHeaderForward specifies a header to extract from the incoming request and forward to a backend. ##### Fields - **name** (string, required): Name is the header name to extract from the incoming client request. - **backendHeader** (string): BackendHeader is the header name to use when forwarding to the backend. If not specified, the original header name is used. #### MCPRouteAuthorization **Appears in:** - [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutesecuritypolicy) MCPRouteAuthorization defines the authorization configuration for a MCPRoute. ##### Fields - **defaultAction** ([AuthorizationAction](#github-com-envoyproxy-gateway-api-v1alpha1-authorizationaction)): DefaultAction is the action to take when no rules match. If unspecified, defaults to Deny. - **rules** ([MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorizationrule) array): Rules defines a list of authorization rules. These rules are evaluated in order, the first matching rule will be applied, and the rest will be skipped. If no rules are defined, the default action will be applied to all requests. #### MCPRouteAuthorizationRule **Appears in:** - [MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorization) MCPRouteAuthorizationRule defines an authorization rule for MCPRoute based on the MCP authorization spec. Reference: https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-challenge-handling ##### Fields - **source** ([MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationsource), required): Source defines the authorization source for this rule. If not specified, the rule will match all sources. - **target** ([MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationtarget), required): Target defines the authorization target for this rule. If not specified, the rule will match all targets. - **cel** (string): CEL specifies a Common Expression Language (CEL) expression evaluated for this rule. The expression must return a boolean; evaluation errors or non-boolean results are treated as `no match`. Example CEL expressions: * `request.method == `POST`` * `request.headers[`x-custom-header`] == `AllowedValue`` * `request.mcp.tool in [`toolA`, `toolB`]` Available attributes in the CEL expression: * request.method: HTTP method such as GET or POST. Type: string. * request.headers: map of headers with lowercased keys, first value only. Type: map[string]string. * request.headers_all: map of headers with lowercased keys, all values. Type: map[string][]string. * request.path: request path such as /mcp. Type: string. * request.auth.jwt.claims: JWT claims when a bearer JWT is present. Type: map[string]any. * request.auth.jwt.scopes: JWT scopes when a bearer JWT is present. Type: []string. * request.mcp.method: MCP method such as tools/list or tools/call. Type: string. * request.mcp.backend: upstream backend name (for example, `kiwi` or `github`). Type: string. * request.mcp.tool: tool name without backend prefix (for example, `list_issues`). Type: string. * request.mcp.params: parameters of the MCP method, including keys like `_meta` and `arguments`. Type: object. Note: The CEL expression support is experimental, and the attributes available to the expression may change in future releases. - **action** ([AuthorizationAction](#github-com-envoyproxy-gateway-api-v1alpha1-authorizationaction)): Action is the authorization decision for matching requests. If unspecified, defaults to Allow. #### MCPRouteBackendRef **Appears in:** - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) MCPRouteBackendRef wraps a EG's BackendObjectReference to reference an MCP server. ##### Fields - **group** ([Group](#sigs-k8s-io-gateway-api-apis-v1-group)): Group is the group of the referent. For example, `gateway.networking.k8s.io`. When unspecified or empty string, core API group is inferred. - **kind** ([Kind](#sigs-k8s-io-gateway-api-apis-v1-kind)): Kind is the Kubernetes resource kind of the referent. For example `Service`. Defaults to `Service` when not specified. ExternalName services can refer to CNAME DNS records that may live outside of the cluster and as such are difficult to reason about in terms of conformance. They also may not be safe to forward to (see CVE-2021-25740 for more information). Implementations SHOULD NOT support ExternalName Services. Support: Core (Services with a type other than ExternalName) Support: Implementation-specific (Services with type ExternalName) - **name** ([ObjectName](#sigs-k8s-io-gateway-api-apis-v1-objectname), required): Name is the name of the referent. - **namespace** ([Namespace](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#namespace)): Namespace is the namespace of the backend. When unspecified, the local namespace is inferred. Note that when a namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. Support: Core - **port** ([PortNumber](#sigs-k8s-io-gateway-api-apis-v1-portnumber)): Port specifies the destination port number to use for this resource. Port is required when the referent is a Kubernetes Service. In this case, the port number is the service port number, not the target port. For other resources, destination port might be derived from the referent resource or this field. - **path** (string): Path is the HTTP endpoint path of the backend MCP server. If not specified, the default is `/mcp`. - **toolSelector** ([MCPToolFilter](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcptoolfilter)): ToolSelector filters the tools exposed by this MCP server. Supports exact matches and RE2-compatible regular expressions for both include and exclude patterns. If not specified, all tools from the MCP server are exposed. - **securityPolicy** ([MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendsecuritypolicy)): SecurityPolicy is the security policy to apply to this MCP server. - **forwardHeaders** ([MCPHeaderForward](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpheaderforward) array): ForwardHeaders specifies HTTP headers to extract from the incoming client request and forward to this backend MCP server. This enables per-user authentication passthrough (e.g., personal access tokens) without requiring OAuth configuration. Each entry specifies a header name to extract and an optional rename for the backend. #### MCPRouteBackendTrafficPolicy **Appears in:** - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) MCPRouteBackendTrafficPolicy configures how the Envoy Gateway BackendTrafficPolicy resources auto-generated for a MCPRoute interact with other BackendTrafficPolicy resources (e.g. an operator-defined rate limiting policy) targeting the same HTTPRoute. ##### Fields - **mergeType** ([MergeType](#github-com-envoyproxy-gateway-api-v1alpha1-mergetype)): MergeType determines how the auto-generated BackendTrafficPolicy (for example, the OAuth protected resource metadata policy) is merged with other BackendTrafficPolicy configurations targeting the same route or a parent Gateway/Listener. When set, the auto-generated BackendTrafficPolicy will merge with the closest parent BackendTrafficPolicy in the route's attachment hierarchy instead of overriding it entirely, per Envoy Gateway's merge semantics. If unset, no merging occurs, and the auto-generated BackendTrafficPolicy fully overrides any other BackendTrafficPolicy targeting the same route. See: https://gateway.envoyproxy.io/docs/tasks/traffic/backend-traffic-policy/ #### MCPRouteOAuth **Appears in:** - [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutesecuritypolicy) MCPRouteOAuth defines a MCP spec compatible OAuth authentication configuration for a MCPRoute. Reference: https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization ##### Fields - **issuer** (string, required): Issuer is the authorization server's issuer identity. - **audiences** (string array): Audiences is a list of JWT audiences allowed access. It is recommended to set this field for token audience validation, as it is a security best practice to prevent token misuse. Reference: https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#token-audience-binding-and-validation - **jwks** ([JWKS](#github-com-envoyproxy-ai-gateway-api-v1alpha1-jwks)): JWKS defines how a JSON Web Key Sets (JWKS) can be obtained to verify the access tokens presented by the clients. If not specified, the JWKS URI will be discovered from the OAuth 2.0 Authorization Server Metadata as per RFC 8414 by querying the `/.well-known/oauth-authorization-server` endpoint on the Issuer. - **authorizationServerMetadataUrl** (string): AuthorizationServerMetadataURL is the URL the controller fetches the OAuth 2.0 Authorization Server Metadata document from, as defined in RFC 8414. When set, it replaces the well-known URIs derived from Issuer. Set this when the metadata lives somewhere the issuer does not lead to, for example an issuer of `https://example.com/api/idp/authn` whose document is served only at `https://example.com/api/idp/v4/authn/.well-known/openid-configuration`. Issuer is unaffected by this field. It continues to identify the authorization server in the protected resource metadata the gateway publishes, and to derive the well-known URIs when this field is unset. When JWKS is not set, the JWKS URI is discovered from the document fetched here. - **protectedResourceMetadata** ([ProtectedResourceMetadata](#github-com-envoyproxy-ai-gateway-api-v1alpha1-protectedresourcemetadata), required): ProtectedResourceMetadata defines the OAuth 2.0 Resource Server Metadata as per RFC 8414. This is used to expose the metadata endpoint for mcp clients to discover the authorization servers, supported scopes, and JWKS URI. - **claimToHeaders** (ClaimToHeader array): ClaimToHeaders specifies JWT claims to extract and forward as HTTP headers to backend MCP servers. This enables backends to access user identity for authorization, auditing, or personalization. Security considerations: - Any client-provided headers matching the configured header names will be stripped to prevent forgery - Only the specified claims are extracted; the full JWT is not forwarded to backends - Consider using a header prefix (e.g., `X-Jwt-Claim-`) to avoid conflicts with other headers #### MCPRouteSecurityPolicy **Appears in:** - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) MCPRouteSecurityPolicy defines the security policy for a MCPRoute. ##### Fields - **oauth** ([MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteoauth)): OAuth defines the configuration for the MCP spec compatible OAuth authentication. - **apiKeyAuth** ([APIKeyAuth](https://gateway.envoyproxy.io/docs/api/extension_types/#apikeyauth)): APIKeyAuth defines the configuration for the API Key Authentication. - **extAuth** ([ExtAuth](#github-com-envoyproxy-gateway-api-v1alpha1-extauth)): ExtAuth defines the configuration for External Authorization. - **authorization** ([MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorization)): Authorization defines the configuration for the MCP spec compatible authorization. - **mergeType** ([MergeType](#github-com-envoyproxy-gateway-api-v1alpha1-mergetype)): MergeType determines how the auto-generated SecurityPolicy for this MCPRoute (which enforces OAuth/JWT, API key, or external authorization) is merged with other SecurityPolicy configurations targeting the same route or a parent Gateway/Listener. When set, the auto-generated SecurityPolicy will merge with the closest parent SecurityPolicy in the route's attachment hierarchy (for example, one targeting a Gateway, Gateway listener, ListenerSet, or ListenerSet listener) instead of overriding it entirely, per Envoy Gateway's merge semantics. If unset, no merging occurs, and the auto-generated SecurityPolicy fully overrides any other SecurityPolicy targeting the same route. See: https://gateway.envoyproxy.io/docs/tasks/security/apikey-authn/ #### MCPRouteSpec **Appears in:** - [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproute) MCPRouteSpec details the MCPRoute configuration. ##### Fields - **parentRefs** ([ParentReference](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#parentreference) array, required): ParentRefs are the names of the Gateway resources this MCPRoute is being attached to. Cross namespace references are not supported. In other words, the Gateway resources must be in the same namespace as the MCPRoute. Currently, each reference's Kind must be Gateway. - **path** (string): Path is the HTTP endpoint path that serves MCP requests over the Streamable HTTP transport. If not specified, the default is `/mcp`. - **headers** ([HTTPHeaderMatch](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#httpheadermatch) array): Headers are HTTP headers that must match for this route to be selected. Multiple match values are ANDed together, meaning, a request must match all the specified headers to select the route. - **hostnames** (Hostname array): Hostnames is a list of hostnames matched against the HTTP Host header to select this MCPRoute. This is equivalent to the Hostnames field in the Gateway API HTTPRouteSpec. When specified, the generated HTTPRoute includes these hostnames, so the MCP route lands in the per-host virtual host instead of the wildcard vhost (the same scoping AIGatewayRoute.Hostnames provides). See https://gateway-api.sigs.k8s.io/reference/api-types/httproute/#hostnames for the details of the Hostnames field in the Gateway API. - **backendRefs** ([MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendref) array, required): BackendRefs is a list of backend references to the MCP servers. These MCP servers will be aggregated and exposed as a single MCP endpoint to the clients. From the client's perspective, they only need to configure a single MCP server URL, e.g. `https://api.example.com/mcp`, and the Envoy AI Gateway will route the requests to the appropriate MCP server based on the requests. All names must be unique within this list to avoid potential tools, resources, etc. name collisions. Also, cross-namespace references are not supported. In other words, the backend MCP servers must be in the same namespace as the MCPRoute. - **securityPolicy** ([MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutesecuritypolicy)): SecurityPolicy defines the security policy for this MCPRoute. - **backendTrafficPolicy** ([MCPRouteBackendTrafficPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendtrafficpolicy)): BackendTrafficPolicy configures the auto-generated Envoy Gateway BackendTrafficPolicy resources for this MCPRoute, for example the OAuth protected resource metadata policy created when SecurityPolicy.OAuth is configured. - **backendSelector** ([MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendselector)): BackendSelector restricts which of this route's backends a given request may fan out to, evaluated once per candidate backend when a client session is initialized. If unspecified, all backends on the route are considered. Security note: a backendSelector rule whose CEL expression reads request.auth.jwt (claims or scopes) only reflects verified JWT data when SecurityPolicy.OAuth is configured for this route; the XValidation below enforces that pairing so that unauthenticated or forged bearer tokens can't influence backend selection. #### MCPRouteStatus **Appears in:** - [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproute) MCPRouteStatus contains the conditions by the reconciliation result. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array, required): Conditions is the list of conditions by the reconciliation result. Currently, at most one condition is set. Known .status.conditions.type are: `Accepted`, `NotAccepted`. #### MCPToolFilter **Appears in:** - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendref) MCPToolFilter filters tools using include and exclude patterns with exact matches or regular expressions. Exclude rules take precedence over include rules (deny-wins). When both include and exclude are specified, a tool must match an include rule AND not match any exclude rule to be allowed. ##### Fields - **include** (string array): Include is a list of tool names to include. Only the specified tools will be available. - **includeRegex** (string array): IncludeRegex is a list of RE2-compatible regular expressions that, when matched, include the tool. Only tools matching these patterns will be available. - **exclude** (string array): Exclude is a list of tool names to exclude. The specified tools will not be available. Exclude rules take precedence over include rules. - **excludeRegex** (string array): ExcludeRegex is a list of RE2-compatible regular expressions that, when matched, exclude the tool. Tools matching these patterns will not be available. Exclude rules take precedence over include rules. #### PerModelQuota **Appears in:** - [QuotaPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicyspec) ##### Fields - **modelName** (string, required): Model name for which the quota is specified. This must match the ModelNameOverride set in the AIGatewayRoute's BackendRef for the quota policy to apply to that backend. - **quota** ([QuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotadefinition), required): Expression for computing request cost and rules for matching requests to quota buckets. #### ProtectedResourceMetadata **Appears in:** - [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteoauth) ProtectedResourceMetadata represents the Protected Resource Metadata of the MCP server as per RFC 9728. References: * https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#authorization-server-location * https://datatracker.ietf.org/doc/html/rfc9728#name-protected-resource-metadata ##### Fields - **resource** (string, required): Resource is the identifier of the protected resource. This should match the MCPRoute's URL. For example, if the MCPRoute's URL is `https://api.example.com/mcp`, the Resource should be `https://api.example.com/mcp`. - **resourceName** (string): ResourceName is a human-readable name for the protected resource. - **scopesSupported** (string array): ScopesSupported defines the minimal set of scopes required for the basic functionality of the MCPRoute. It should avoid broad or overly permissive scopes to prevent clients from requesting tokens with excessive privileges. If an operation requires additional scopes that are not present in the access token, the client will receive a 403 Forbidden response that includes the required scopes in the `scope` field of the `WWW-Authenticate` header. This enables incremental privilege elevation through targeted `WWW-Authenticate: scope=`...`` challenges when privileged operations are first attempted. - **resourceSigningAlgValuesSupported** (string array): ResourceSigningAlgValuesSupported is a list of JWS signing algorithms supported by the resource server. These algorithms are used in the `alg` field of the JOSE header in signed tokens. - **resourceDocumentation** (string): ResourceDocumentation is a URL that provides human-readable documentation for the resource. - **resourcePolicyUri** (string): ResourcePolicyURI is a URL that points to the resource server's policy document. #### QuotaBucketMode **Underlying type:** string **Appears in:** - [QuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotadefinition) QuotaBucketMode specifies whether the default and per request buckets values are exclusive or inclusive. ##### Possible Values - **Shared** (enum): #### QuotaDefinition **Appears in:** - [PerModelQuota](#github-com-envoyproxy-ai-gateway-api-v1alpha1-permodelquota) QuotaDefinition specified expression for computing request cost and rules for matching requests to quota buckets. ##### Fields - **costExpression** (string): CostExpression specifies a CEL expression for computing the quota burndown of the LLM-related request. If no expression is specified the `total_tokens` value is used. For example: `input_tokens + cached_input_tokens + output_tokens` - **mode** ([QuotaBucketMode](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotabucketmode)): The `Mode` determines how quota is charged to the `DefaultBucket` and matching `BucketRules`. In the `shared` mode the quota is charged to all matching `BucketRules` AND the `DefaultBucket` and request is allowed only if the quota is available in at least one matching buckets. Defaults to `Shared`. - **defaultBucket** ([QuotaValue](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotavalue)): Quota applicable to all traffic. This value can be overridden for specific classes of requests using the `BucketRules` configuration. - **bucketRules** ([QuotaRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotarule) array): BucketRules are a list of client selectors and quotas. If a request matches multiple rules, each of their associated quotas get applied, so a single request might burn down the quota for multiple rules. Client selectors that match under the same model / service backend will be combined with the first limit taking precedence. #### QuotaPolicySpec **Appears in:** - [QuotaPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicy) QuotaPolicySpec specifies rules for computing token based costs of requests. ##### Fields - **targetRefs** ([LocalPolicyTargetReference](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1alpha2.LocalPolicyTargetReference) array): TargetRefs are the names of the AIServiceBackend resources this QuotaPolicy is being attached to. - **serviceQuota** ([ServiceQuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-servicequotadefinition)): Quota for all models served by AIServiceBackend(s). This value can be overridden for specific models using the `PerModelQuotas` configuration. Currently, the rate limit configuration is properly set up, but the descriptor set is not being set in the Envoy Configuration - **perModelQuotas** ([PerModelQuota](#github-com-envoyproxy-ai-gateway-api-v1alpha1-permodelquota) array): PerModelQuotas specifies quota for different models served by the AIServiceBackend(s) where this policy is attached. When multiple QuotaPolicies define the same Model for the same AIServiceBackend, the policy whose namespace/name is alphabetically first takes precedence. Keys are sorted to ensure deterministic snapshot generation. #### QuotaPolicyStatus **Appears in:** - [QuotaPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicy) QuotaPolicyStatus contains the conditions by the reconciliation result. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array, required): Conditions is the list of conditions by the reconciliation result. Currently, at most one condition is set. Known .status.conditions.type are: `Accepted`, `NotAccepted`. #### QuotaRule **Appears in:** - [QuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotadefinition) ##### Fields - **clientSelectors** (RateLimitSelectCondition array): ClientSelectors holds the list of conditions to select specific clients using attributes from the traffic flow. All individual select conditions must hold True for this rule and its limit to be applied. Currently, only header types are honored. If no client selectors are specified, the rule applies to all traffic of the targeted AIServiceBackend. - **quota** ([QuotaValue](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotavalue), required): Quota value for given client selectors. This quota is applied for traffic flows when the selectors compute to True, causing the request to be counted towards the limit. A response with 429 HTTP status code is sent back to the client when the selected requests have exceeded the quota. - **shadowMode** (boolean): ShadowMode indicates whether this quota rule runs in shadow mode. When enabled, all quota checks are performed (cache lookups, counter updates, telemetry generation), but the outcome is never enforced. The request always succeeds, even if the configured quota is exceeded. #### QuotaValue **Appears in:** - [QuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotadefinition) - [QuotaRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotarule) - [ServiceQuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-servicequotadefinition) QuotaValue defines the quota limits using sliding window. ##### Fields - **limit** (integer, required): The limit alloted for a specified time window. - **duration** (string, required): Time window. Must be exactly one of: `1s` (1 second), `1m` (1 minute), `1h` (1 hour), or `1d` (1 day). #### ServiceQuotaDefinition **Appears in:** - [QuotaPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicyspec) ##### Fields - **costExpression** (string): CostExpression specifies a CEL expression for computing the quota burndown of the LLM-related request. If no expression is specified the `total_tokens` value is used. For example: `input_tokens + cached_input_tokens + output_tokens` - **quota** ([QuotaValue](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotavalue), required): Quota value applicable to all requests. A response with 429 HTTP status code is sent back to the client when the selected requests have exceeded the quota. #### ToolCall **Appears in:** - [MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationtarget) ToolCall represents a tool call in the MCP authorization target. ##### Fields - **backend** (string, required): Backend is the name of the backend this tool belongs to. - **tool** (string, required): Tool is the name of the tool. #### VersionedAPISchema **Appears in:** - [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendspec) VersionedAPISchema defines the API schema of either AIGatewayRoute (the input) or AIServiceBackend (the output). This allows the ai-gateway to understand the input and perform the necessary transformation depending on the API schema pair (input, output). Note that this is vendor specific, and the stability of the API schema is not guaranteed by the ai-gateway, but by the vendor via proper versioning. ##### Fields - **name** ([APISchema](#github-com-envoyproxy-ai-gateway-api-v1alpha1-apischema), required): Name is the name of the API schema of the AIGatewayRoute or AIServiceBackend. - **version** (string): Version is the version of the API schema. When the name is set to AzureOpenAI, this version maps to `API Version` in the Azure OpenAI API documentation (https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#rest-api-versioning). This field is ignored for OpenAI, AWSBedrock, GCPVertexAI, and Anthropic. For OpenAI and Anthropic, use prefix to configure custom request paths. See https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-providers for details. - **prefix** (string): Prefix is the prefix for the API. When the name is set to `OpenAI`, `chat completions` API endpoint will be `$\{this_field\}/chat/completions`. When the name is set to `Anthropic`, the `messages` API endpoint will be `$\{this_field\}/messages`. It can be with or without a leading slash (`/`). This field is ignored for AWSAnthropic and GCPAnthropic. This is especially useful when routing to a backend that has an OpenAI or Anthropic compatible API but has a different prefix. For example, Gemini OpenAI compatible API (https://ai.google.dev/gemini-api/docs/openai) uses `/v1beta/openai` prefix. Another example is that Cohere AI (https://docs.cohere.com/v2/docs/compatibility-api) uses `/compatibility/v1` prefix. On the other hand, DeepSeek (https://api-docs.deepseek.com/) doesn't use prefix, so you can leave this field unset. See https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-providers for details. ## aigateway.envoyproxy.io/v1beta1 Package v1beta1 contains API schema definitions for the aigateway.envoyproxy.io API group. This is the beta version of the API, preferred over v1alpha1 for new resources. ## Resource Kinds ### Available Kinds - [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroute) - [AIGatewayRouteList](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutelist) - [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackend) - [AIServiceBackendList](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendlist) - [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicy) - [BackendSecurityPolicyList](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicylist) - [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfig) - [GatewayConfigList](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfiglist) - [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproute) - [MCPRouteList](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutelist) ### Kind Definitions #### AIGatewayRoute **Appears in:** - [AIGatewayRouteList](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutelist) AIGatewayRoute combines multiple AIServiceBackends and attaching them to Gateway(s) resources. This serves as a way to define a "unified" AI API for a Gateway which allows downstream clients to use a single schema API to interact with multiple AI backends. Envoy AI Gateway will generate the following k8s resources corresponding to the AIGatewayRoute: - HTTPRoute of the Gateway API as a top-level resource to bind all backends. The name of the HTTPRoute is the same as the AIGatewayRoute. - HTTPRouteFilter of the Envoy Gateway API per namespace for automatic hostname rewrite. The name of the HTTPRouteFilter is `ai-eg-host-rewrite-${AIGatewayRoute.Name}`. All of these resources are created in the same namespace as the AIGatewayRoute. Note that this is the implementation detail subject to change. If you want to customize the default behavior of the Envoy AI Gateway, you can use these resources as a reference and create your own resources. Alternatively, you can use EnvoyPatchPolicy API of the Envoy Gateway to patch the generated resources. For example, you can configure the retry fallback behavior by attaching BackendTrafficPolicy API of Envoy Gateway to the generated HTTPRoute. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `AIGatewayRoute` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutespec), required): Spec defines the details of the AIGatewayRoute. - **status** ([AIGatewayRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutestatus), required): Status defines the status details of the AIGatewayRoute. #### AIGatewayRouteList AIGatewayRouteList contains a list of AIGatewayRoute. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `AIGatewayRouteList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroute) array, required): #### AIServiceBackend **Appears in:** - [AIServiceBackendList](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendlist) AIServiceBackend is a resource that represents a single backend for AIGatewayRoute. A backend is a service that handles traffic with a concrete API specification. A AIServiceBackend is "attached" to a Backend which is either a k8s Service or a Backend resource of the Envoy Gateway. When a backend with an attached AIServiceBackend is used as a routing target in the AIGatewayRoute (more precisely, the HTTPRouteSpec defined in the AIGatewayRoute), the ai-gateway will generate the necessary configuration to do the backend specific logic in the final HTTPRoute. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `AIServiceBackend` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec), required): Spec defines the details of AIServiceBackend. - **status** ([AIServiceBackendStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendstatus), required): Status defines the status details of the AIServiceBackend. #### AIServiceBackendList AIServiceBackendList contains a list of AIServiceBackends. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `AIServiceBackendList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackend) array, required): #### BackendSecurityPolicy **Appears in:** - [BackendSecurityPolicyList](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicylist) BackendSecurityPolicy specifies configuration for authentication and authorization rules on the traffic exiting the gateway to the backend. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `BackendSecurityPolicy` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec), required): - **status** ([BackendSecurityPolicyStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicystatus), required): Status defines the status details of the BackendSecurityPolicy. #### BackendSecurityPolicyList BackendSecurityPolicyList contains a list of BackendSecurityPolicy ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `BackendSecurityPolicyList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicy) array, required): #### GatewayConfig **Appears in:** - [GatewayConfigList](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfiglist) GatewayConfig provides configuration for the AI Gateway external processor container that is deployed alongside the Gateway. A GatewayConfig is referenced by a Gateway via the annotation "aigateway.envoyproxy.io/gateway-config". The GatewayConfig must be in the same namespace as the Gateway that references it. This allows gateway-level configuration of the external processor, including environment variables (e.g., for tracing configuration) and resource requirements. Multiple Gateways can reference the same GatewayConfig to share configuration. Environment Variable Precedence: When merging environment variables, the following precedence applies (highest to lowest): 1. GatewayConfig.Spec.ExtProc.Kubernetes.Env (this resource) 2. Global controller flags (extProcExtraEnvVars) If the same environment variable name exists in both sources, the GatewayConfig value takes precedence. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `GatewayConfig` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigspec), required): Spec defines the configuration for the external processor. - **status** ([GatewayConfigStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigstatus), required): Status defines the status of the GatewayConfig. #### GatewayConfigList GatewayConfigList contains a list of GatewayConfig. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `GatewayConfigList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfig) array, required): #### MCPRoute **Appears in:** - [MCPRouteList](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutelist) MCPRoute defines how to route MCP requests to the backend MCP servers. This serves as a way to define a "unified" AI API for a Gateway which allows downstream clients to use a single schema API to interact with multiple MCP backends. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `MCPRoute` resource - **metadata** ([ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#objectmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **spec** ([MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec), required): Spec defines the details of the MCPRoute. - **status** ([MCPRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutestatus), required): Status defines the status details of the MCPRoute. #### MCPRouteList MCPRouteList contains a list of MCPRoute. ##### Fields - **apiVersion** (String, required): We are on version `aigateway.envoyproxy.io/v1beta1` of the API. - **kind** (String, required): This is a `MCPRouteList` resource - **metadata** ([ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#listmeta-v1-meta), required): Refer to Kubernetes API documentation for fields of `metadata`. - **items** ([MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproute) array, required): ## Supporting Types ### Available Types - [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterule) - [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulebackendref) - [AIGatewayRouteRuleMatch](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulematch) - [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutespec) - [AIGatewayRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutestatus) - [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) - [AIServiceBackendStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendstatus) - [APISchema](#github-com-envoyproxy-ai-gateway-api-v1beta1-apischema) - [AWSCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1beta1-awscredentialsfile) - [AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-awsoidcexchangetoken) - [AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-azureoidcexchangetoken) - [BackendSecurityPolicyAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyapikey) - [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyawscredentials) - [BackendSecurityPolicyAnthropicAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyanthropicapikey) - [BackendSecurityPolicyAzureAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyazureapikey) - [BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyazurecredentials) - [BackendSecurityPolicyCredentialOverride](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicycredentialoverride) - [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicygcpcredentials) - [BackendSecurityPolicyOIDC](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyoidc) - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) - [BackendSecurityPolicyStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicystatus) - [BackendSecurityPolicyType](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicytype) - [CredentialOverrideFromDynamicMetadata](#github-com-envoyproxy-ai-gateway-api-v1beta1-credentialoverridefromdynamicmetadata) - [CredentialOverrideFromRequestHeaders](#github-com-envoyproxy-ai-gateway-api-v1beta1-credentialoverridefromrequestheaders) - [GCPCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpcredentialsfile) - [GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpoidcexchangetoken) - [GCPServiceAccountImpersonationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpserviceaccountimpersonationconfig) - [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityfederationconfig) - [GCPWorkloadIdentityProvider](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityprovider) - [GatewayConfigExtProc](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigextproc) - [GatewayConfigForwardProxy](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigforwardproxy) - [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigspec) - [GatewayConfigStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigstatus) - [HTTPBodyField](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpbodyfield) - [HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpbodymutation) - [HTTPHeaderMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadermutation) - [HTTPHeaderValueFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadervaluefilter) - [HTTPHeaderValueFilterMode](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadervaluefiltermode) - [JWKS](#github-com-envoyproxy-ai-gateway-api-v1beta1-jwks) - [JWTSource](#github-com-envoyproxy-ai-gateway-api-v1beta1-jwtsource) - [LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1beta1-llmrequestcost) - [LLMRequestCostType](#github-com-envoyproxy-ai-gateway-api-v1beta1-llmrequestcosttype) - [MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationsource) - [MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationtarget) - [MCPBackendAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendapikey) - [MCPBackendAPIKeyInjectionPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendapikeyinjectionpolicy) - [MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendsecuritypolicy) - [MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendselector) - [MCPBackendSelectorRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendselectorrule) - [MCPHeaderForward](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpheaderforward) - [MCPPromptFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcppromptfilter) - [MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorization) - [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorizationrule) - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) - [MCPRouteBackendTrafficPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendtrafficpolicy) - [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteoauth) - [MCPRoutePrefixMode](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteprefixmode) - [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutesecuritypolicy) - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) - [MCPRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutestatus) - [MCPToolFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcptoolfilter) - [ProtectedResourceMetadata](#github-com-envoyproxy-ai-gateway-api-v1beta1-protectedresourcemetadata) - [ToolCall](#github-com-envoyproxy-ai-gateway-api-v1beta1-toolcall) - [VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1beta1-versionedapischema) ### Type Definitions #### AIGatewayRouteRule **Appears in:** - [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutespec) AIGatewayRouteRule is a rule that defines the routing behavior of the AIGatewayRoute. ##### Fields - **name** ([SectionName](#sigs-k8s-io-gateway-api-apis-v1-sectionname)): Name is the name of the route rule. This name must be unique within the route. When specified, it is copied to the generated HTTPRoute rule name. - **backendRefs** ([AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulebackendref) array): BackendRefs is the list of backends that this rule will route the traffic to. Each backend can have a weight that determines the traffic distribution. The namespace of each backend defaults to the same namespace as the AIGatewayRoute when not specified. Cross-namespace references are supported by specifying the namespace field. When a namespace different than the AIGatewayRoute's namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. BackendRefs can reference either AIServiceBackend resources (default) or InferencePool resources from the Gateway API Inference Extension. When referencing InferencePool resources: - Only one InferencePool backend is allowed per rule - Cannot mix InferencePool with AIServiceBackend references in the same rule - Fallback behavior is handled by the InferencePool's endpoint picker For AIServiceBackend references, you can achieve fallback behavior by configuring multiple backends combined with the BackendTrafficPolicy of Envoy Gateway. Please refer to https://gateway.envoyproxy.io/docs/tasks/traffic/failover/ as well as https://gateway.envoyproxy.io/docs/tasks/traffic/retry/. - **matches** ([AIGatewayRouteRuleMatch](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulematch) array): Matches is the list of AIGatewayRouteMatch that this rule will match the traffic to. This is a subset of the HTTPRouteMatch in the Gateway API. See for the details: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRouteMatch - **timeouts** ([HTTPRouteTimeouts](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#httproutetimeouts)): Timeouts defines the timeouts that can be configured for an HTTP request. If this field is not set, or the timeout.requestTimeout is nil, Envoy AI Gateway defaults to set 60s for the request timeout as opposed to 15s of the Envoy Gateway's default value. For streaming responses (like chat completions with stream=true), consider setting longer timeouts as the response may take time until the completion. Timeouts.Request acts as the maximum total time the gateway will wait for the entire response, including all streamed chunks. - **streamIdleTimeout** ([Duration](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.Duration)): StreamIdleTimeout is the maximum time Envoy will wait without receiving any bytes from the upstream. If the timer fires before the first response byte arrives, Envoy resets the upstream stream and a retry policy can fall over to the next backend. If it fires mid-stream after bytes have already arrived, the stream is cut and the client receives a 504. The AI Gateway extension server sets route.retry_policy.per_try_idle_timeout to this value on every xDS route generated from this rule before it is sent to the data plane. Pair this field with Timeouts.Request, which acts as the overall deadline. If this field is not set, no per-try idle timeout is applied. - **modelsOwnedBy** (string): ModelsOwnedBy represents the owner of the running models serving by the backends, which will be exported as the field of `OwnedBy` in openai-compatible API `/models`. This is used only when this rule contains `x-ai-eg-model` in its header matching where the header value will be recognized as a `model` in `/models` endpoint. All the matched models will share the same owner. Default to `Envoy AI Gateway` if not set. - **modelsCreatedAt** ([Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#time-v1-meta)): ModelsCreatedAt represents the creation timestamp of the running models serving by the backends, which will be exported as the field of `Created` in openai-compatible API `/models`. It follows the format of RFC 3339, for example `2024-05-21T10:00:00Z`. This is used only when this rule contains `x-ai-eg-model` in its header matching where the header value will be recognized as a `model` in `/models` endpoint. All the matched models will share the same creation time. Default to the creation timestamp of the AIGatewayRoute if not set. - **excludeFromModelsEndpoint** (boolean): ExcludeFromModelsEndpoint prevents models matched by this rule from being returned by the OpenAI-compatible `/models` endpoint. Requests still match this rule and apply its configured transformations, including model name overrides. This is useful for internal or backwards-compatible aliases, as well as duplicate model definitions with different schemas. #### AIGatewayRouteRuleBackendRef **Appears in:** - [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterule) AIGatewayRouteRuleBackendRef is a reference to a backend with a weight. It can reference either an AIServiceBackend or an InferencePool resource. ##### Fields - **name** (string, required): Name is the name of the backend resource. When Group and Kind are not specified, this refers to an AIServiceBackend. When Group and Kind are specified, this refers to the resource of the specified type. - **namespace** ([Namespace](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#namespace)): Namespace is the namespace of the backend resource. When unspecified (or empty string), this refers to the local namespace of the AIGatewayRoute. Note that when a namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. - **group** (string): Group is the group of the backend resource. When not specified, defaults to aigateway.envoyproxy.io (AIServiceBackend). Currently, only `inference.networking.k8s.io` is supported for InferencePool resources. - **kind** (string): Kind is the kind of the backend resource. When not specified, defaults to AIServiceBackend. Currently, only `InferencePool` is supported when Group is specified. - **modelNameOverride** (string): Name of the model in the backend. If provided this will override the name provided in the request. This field is ignored when referencing InferencePool resources. - **headerMutation** ([HTTPHeaderMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadermutation)): HeaderMutation defines the request header mutation to be applied to this backend. When both route-level and backend-level HeaderMutation are defined, route-level takes precedence over backend-level for conflicting operations. This field is ignored when referencing InferencePool resources. - **bodyMutation** ([HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpbodymutation)): BodyMutation defines the request body mutation to be applied to this backend. This allows modification of JSON fields in the request body before sending to the backend. When both route-level and backend-level BodyMutation are defined, route-level takes precedence over backend-level for conflicting operations. This field is ignored when referencing InferencePool resources. - **weight** (integer): Weight is the weight of the backend. This is exactly the same as the weight in the BackendRef in the Gateway API. See for the details: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.BackendRef Default is 1. - **priority** (integer): Priority is the priority of the backend. This sets the priority on the underlying endpoints. See: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/priority Note: This will override the `faillback` property of the underlying Envoy Gateway Backend This field is ignored when referencing InferencePool resources. Default is 0. #### AIGatewayRouteRuleMatch **Appears in:** - [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterule) ##### Fields - **headers** ([HTTPHeaderMatch](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#httpheadermatch) array): Headers specifies HTTP request header matchers. See HeaderMatch in the Gateway API for the details: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPHeaderMatch #### AIGatewayRouteSpec **Appears in:** - [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroute) AIGatewayRouteSpec details the AIGatewayRoute configuration. ##### Fields - **parentRefs** ([ParentReference](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#parentreference) array): ParentRefs are the names of the Gateway resources this AIGatewayRoute is being attached to. Currently, each reference's Kind must be Gateway. - **hostnames** (Hostname array): Hostnames is a list of hostnames matched against the HTTP Host header to select an AIGatewayRoute used to process the request. This is equivalent to the Hostnames field in the Gateway API HTTPRouteSpec. When specified, the generated HTTPRoute will include these hostnames for hostname-based filtering. See https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRouteSpec for the details of the Hostnames field in the Gateway API. - **rules** ([AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterule) array, required): Rules is the list of AIGatewayRouteRule that this AIGatewayRoute will match the traffic to. Each rule is a subset of the HTTPRoute in the Gateway API (https://gateway-api.sigs.k8s.io/api-types/httproute/). AI Gateway controller will generate a HTTPRoute based on the configuration given here with the additional modifications to achieve the necessary jobs, notably inserting the AI Gateway filter responsible for the transformation of the request and response, etc. In the matching conditions in the AIGatewayRouteRule, `x-ai-eg-model` header is available if we want to describe the routing behavior based on the model name. The model name is extracted from the request content before the routing decision. How multiple rules are matched is the same as the Gateway API. See for the details: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRoute At most 15 rules are allowed per AIGatewayRoute, corresponding to the Gateway API's limit on HTTPRoute.spec.rules (one slot is reserved for a controller-injected catch-all rule). To configure more rules on the same Gateway, split them across multiple AIGatewayRoute resources. - **llmRequestCosts** ([LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1beta1-llmrequestcost) array): LLMRequestCosts specifies how to capture the cost of the LLM-related request, notably the token usage. The AI Gateway filter will capture each specified number and store it in the Envoy's dynamic metadata per HTTP request. The namespaced key is `io.envoy.ai_gateway`. These route-level costs override any global defaults defined in GatewayConfig.Spec.GlobalLLMRequestCosts for the same metadataKey. If a metadataKey is not defined in either place, no cost is calculated for it. This allows you to define common cost formulas once at the gateway level (e.g., via GatewayConfig) and only override them in specific routes when needed (e.g., premium routes with different pricing). For example, let's say we have the following LLMRequestCosts configuration: ```yaml llmRequestCosts: - metadataKey: llm_input_token type: InputToken - metadataKey: llm_output_token type: OutputToken - metadataKey: llm_total_token type: TotalToken - metadataKey: llm_cached_input_token type: CachedInputToken - metadataKey: llm_cache_creation_input_token type: CacheCreationInputToken ``` Then, with the following BackendTrafficPolicy of Envoy Gateway, you can have three rate limit buckets for each unique x-tenant-id header value. One bucket is for the input token, the other is for the output token, and the last one is for the total token. Each bucket will be reduced by the corresponding token usage captured by the AI Gateway filter. ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: some-example-token-rate-limit namespace: default spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: usage-rate-limit rateLimit: type: Global global: rules: - clientSelectors: # Do the rate limiting based on the x-tenant-id header. - headers: - name: x-tenant-id type: Distinct limit: # Configures the number of `tokens` allowed per hour. requests: 10000 unit: Hour cost: request: from: Number # Setting the request cost to zero allows to only check the rate limit budget, # and not consume the budget on the request path. number: 0 # This specifies the cost of the response retrieved from the dynamic metadata set by the AI Gateway filter. # The extracted value will be used to consume the rate limit budget, and subsequent requests will be rate limited # if the budget is exhausted. response: from: Metadata metadata: namespace: io.envoy.ai_gateway key: llm_input_token - clientSelectors: - headers: - name: x-tenant-id type: Distinct limit: requests: 10000 unit: Hour cost: request: from: Number number: 0 response: from: Metadata metadata: namespace: io.envoy.ai_gateway key: llm_output_token - clientSelectors: - headers: - name: x-tenant-id type: Distinct limit: requests: 10000 unit: Hour cost: request: from: Number number: 0 response: from: Metadata metadata: namespace: io.envoy.ai_gateway key: llm_total_token ``` Note that when multiple AIGatewayRoute resources are attached to the same Gateway, and different costs are configured for the same metadata key, each route's rule is carried in the filter configuration with the route identity; the data plane selects the matching rule per request (by route), so each route can define its own cost for the same metadata key. #### AIGatewayRouteStatus **Appears in:** - [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroute) AIGatewayRouteStatus contains the conditions by the reconciliation result. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array, required): Conditions is the list of conditions by the reconciliation result. Currently, at most one condition is set. Known .status.conditions.type are: `Accepted`, `NotAccepted`. #### AIServiceBackendSpec **Appears in:** - [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackend) AIServiceBackendSpec details the AIServiceBackend configuration. ##### Fields - **schema** ([VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1beta1-versionedapischema), required): APISchema specifies the API schema of the output format of requests from Envoy that this AIServiceBackend can accept as incoming requests. Based on this schema, the ai-gateway will perform the necessary transformation for the pair of AIGatewayRouteSpec.APISchema and AIServiceBackendSpec.APISchema. This is required to be set. - **backendRef** ([BackendObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.BackendObjectReference), required): BackendRef is the reference to the Backend resource that this AIServiceBackend corresponds to. A backend must be a Backend resource of Envoy Gateway. Note that k8s Service will be supported as a backend in the future. See https://github.com/envoyproxy/ai-gateway/issues/902 for more details. This is required to be set. - **headerMutation** ([HTTPHeaderMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadermutation)): HeaderMutation defines the mutation of HTTP headers that will be applied to the request before sending it to the backend. - **bodyMutation** ([HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpbodymutation)): BodyMutation defines the mutation of HTTP request body JSON fields that will be applied to the request before sending it to the backend. - **headerValueFilters** ([HTTPHeaderValueFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadervaluefilter) array): HeaderValueFilters filter individual values out of multi-valued request headers before sending the request to this backend, so that one value an upstream provider does not accept does not fail the whole request. At most one filter per header name. Only GCPAnthropic and AWSAnthropic backends honor this, and today only for the `anthropic-beta` header — a filter naming any other header is accepted but has no effect. Setting it on a backend with any other schema is rejected. #### AIServiceBackendStatus **Appears in:** - [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackend) AIServiceBackendStatus contains the conditions by the reconciliation result. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array, required): Conditions is the list of conditions by the reconciliation result. Currently, at most one condition is set. Known .status.conditions.type are: `Accepted`, `NotAccepted`. #### APISchema **Underlying type:** string **Appears in:** - [VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1beta1-versionedapischema) APISchema defines the API schema. ##### Possible Values - **OpenAI** (enum): APISchemaOpenAI is the OpenAI schema. https://github.com/openai/openai-openapi - **Cohere** (enum): APISchemaCohere is the Cohere schema. https://docs.cohere.com/v2 - **AWSBedrock** (enum): APISchemaAWSBedrock is the AWS Bedrock schema. https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations_Amazon_Bedrock_Runtime.html - **AzureOpenAI** (enum): APISchemaAzureOpenAI APISchemaAzure is the Azure OpenAI schema. https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#api-specs - **GCPVertexAI** (enum): APISchemaGCPVertexAI is the schema followed by Gemini models hosted on GCP's Vertex AI platform. Note: Using this schema requires a BackendSecurityPolicy to be configured and attached, as the transformation will use the gcp-region and project-name from the BackendSecurityPolicy. https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.endpoints/generateContent?hl=en - **GCPAnthropic** (enum): APISchemaGCPAnthropic is the schema for Anthropic models hosted on GCP's Vertex AI platform. Returns native Anthropic format responses for seamless integration. https://docs.anthropic.com/en/api/claude-on-vertex-ai - **Anthropic** (enum): APISchemaAnthropic is the native Anthropic API schema. https://docs.claude.com/en/home - **AWSAnthropic** (enum): APISchemaAWSAnthropic is the schema for Anthropic models hosted on AWS Bedrock. Uses the native Anthropic Messages API format for requests and responses. When used with /v1/chat/completions endpoint, translates OpenAI format to Anthropic. When used with /v1/messages endpoint, passes through native Anthropic format. https://aws.amazon.com/bedrock/anthropic/ https://docs.claude.com/en/api/claude-on-amazon-bedrock - **AWSOpenAI** (enum): APISchemaAWSOpenAI is the OpenAI-compatible API schema provided by AWS. - **TypeSafe** (enum): APISchemaTypeSafe is the native TypeSafe AI schema used by the Jev System One model. Requests to /v1/systemone are passed through unchanged, and the version defaults to v1. https://docs.typesafe.ai/api.md #### AWSCredentialsFile **Appears in:** - [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyawscredentials) AWSCredentialsFile specifies the credentials file to use for the AWS provider. Envoy reads the secret file, and the profile to use is specified by the Profile field. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the credential file. The secret should contain the AWS credentials file keyed on `credentials`. - **profile** (string, required): Profile is the profile to use in the credentials file. #### AWSOIDCExchangeToken **Appears in:** - [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyawscredentials) AWSOIDCExchangeToken specifies credentials to obtain oidc token from a sso server. For AWS, the controller will query STS to obtain AWS AccessKeyId, SecretAccessKey, and SessionToken, and store them in a temporary credentials file. ##### Fields - **oidc** ([OIDC](https://gateway.envoyproxy.io/docs/api/extension_types/#oidc), required): OIDC is used to obtain oidc tokens via an SSO server which will be used to exchange for provider credentials. - **grantType** (string): GrantType is the method application gets access token. - **aud** (string): Aud defines the audience that this ID Token is intended for. - **awsRoleArn** (string, required): AwsRoleArn is the AWS IAM Role with the permission to use specific resources in AWS account which maps to the temporary AWS security credentials exchanged using the authentication token issued by OIDC provider. #### AzureOIDCExchangeToken **Appears in:** - [BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyazurecredentials) AzureOIDCExchangeToken specifies credentials to obtain oidc token from a sso server. For Azure, the controller will query Azure Entra ID to get an Azure Access Token, and store them in a secret. ##### Fields - **oidc** ([OIDC](https://gateway.envoyproxy.io/docs/api/extension_types/#oidc), required): OIDC is used to obtain oidc tokens via an SSO server which will be used to exchange for provider credentials. - **grantType** (string): GrantType is the method application gets access token. - **aud** (string): Aud defines the audience that this ID Token is intended for. #### BackendSecurityPolicyAPIKey **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) BackendSecurityPolicyAPIKey specifies the API key. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the secret containing the API key. ai-gateway must be given the permission to read this secret. The key of the secret should be `apiKey`. #### BackendSecurityPolicyAWSCredentials **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) BackendSecurityPolicyAWSCredentials contains the supported authentication mechanisms to access AWS. When neither CredentialsFile nor OIDCExchangeToken is specified, the AWS SDK's default credential chain will be used. This automatically supports various authentication methods in the following order: 1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) 2. EKS Pod Identity - automatically rotates credentials for pods in EKS clusters 3. IAM Roles for Service Accounts (IRSA) - injects credentials via mounted service account tokens 4. EC2 instance metadata (IAM instance roles) 5. ECS task roles The default credential chain is recommended for Kubernetes deployments as it supports automatic credential rotation without manual configuration. Credentials are refreshed automatically when they approach expiration (typically hourly for IRSA and Pod Identity). ##### Fields - **region** (string, required): Region specifies the AWS region associated with the policy. - **credentialsFile** ([AWSCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1beta1-awscredentialsfile)): CredentialsFile specifies the credentials file to use for the AWS provider. When specified, this takes precedence over the default credential chain. - **oidcExchangeToken** ([AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-awsoidcexchangetoken)): OIDCExchangeToken specifies the oidc configurations used to obtain an oidc token. The oidc token will be used to obtain temporary credentials to access AWS. When specified, this takes precedence over the default credential chain. #### BackendSecurityPolicyAnthropicAPIKey **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) BackendSecurityPolicyAnthropicAPIKey specifies the Anthropic API key. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the secret containing the Anthropic API key. ai-gateway must be given the permission to read this secret. The key of the secret should be `apiKey`. #### BackendSecurityPolicyAzureAPIKey **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) BackendSecurityPolicyAzureAPIKey specifies the Azure OpenAI API key. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the secret containing the Azure API key. ai-gateway must be given the permission to read this secret. The key of the secret should be `apiKey`. #### BackendSecurityPolicyAzureCredentials **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) BackendSecurityPolicyAzureCredentials contains the supported authentication mechanisms to access Azure. Only one of ClientSecretRef or OIDCExchangeToken must be specified. Credentials will not be generated if neither are set. ##### Fields - **clientID** (string, required): ClientID is a unique identifier for an application in Azure. - **tenantID** (string, required): TenantId is a unique identifier for an Azure Active Directory instance. - **clientSecretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference)): ClientSecretRef is the reference to the secret containing the Azure client secret. ai-gateway must be given the permission to read this secret. The key of secret should be `client-secret`. - **oidcExchangeToken** ([AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-azureoidcexchangetoken)): OIDCExchangeToken specifies the oidc configurations used to obtain an oidc token. The oidc token will be used to obtain temporary credentials to access Azure. #### BackendSecurityPolicyCredentialOverride **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) BackendSecurityPolicyCredentialOverride configures per-request credential sourcing. A trusted upstream filter (ext_authz or a preceding ext_proc) resolves the caller's credential and delivers it via one of the two sources below. Exactly one source must be configured. ##### Fields - **fromRequestHeaders** ([CredentialOverrideFromRequestHeaders](#github-com-envoyproxy-ai-gateway-api-v1beta1-credentialoverridefromrequestheaders)): FromRequestHeaders sources the credential from a request header injected by a trusted ingress filter. The gateway strips the header before forwarding to the upstream backend. Because any client can set request headers, this source is only safe when a trusted filter removes any client-supplied copy and writes the canonical value. - **fromDynamicMetadata** ([CredentialOverrideFromDynamicMetadata](#github-com-envoyproxy-ai-gateway-api-v1beta1-credentialoverridefromdynamicmetadata)): FromDynamicMetadata sources the credential from Envoy dynamic metadata set by a trusted filter (e.g. ext_authz). Metadata cannot be forged by the client and is the preferred source. #### BackendSecurityPolicyGCPCredentials **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) BackendSecurityPolicyGCPCredentials contains the supported authentication mechanisms to access GCP. When neither CredentialsFile nor WorkloadIdentityFederationConfig is specified, Application Default Credentials (ADC) will be used. This supports GKE Workload Identity, environment variables, and default service account credentials when running on GCP. ##### Fields - **projectName** (string, required): ProjectName is the GCP project name. - **region** (string, required): Region is the GCP region associated with the policy. - **credentialsFile** ([GCPCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpcredentialsfile)): CredentialsFile specifies the service account credentials file to use for the GCP provider. - **workloadIdentityFederationConfig** ([GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityfederationconfig)): WorkloadIdentityFederationConfig is the configuration for the GCP Workload Identity Federation. #### BackendSecurityPolicyOIDC **Appears in:** - [AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-awsoidcexchangetoken) - [AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-azureoidcexchangetoken) - [GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpoidcexchangetoken) - [GCPWorkloadIdentityProvider](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityprovider) BackendSecurityPolicyOIDC specifies OIDC related fields. ##### Fields - **oidc** ([OIDC](https://gateway.envoyproxy.io/docs/api/extension_types/#oidc), required): OIDC is used to obtain oidc tokens via an SSO server which will be used to exchange for provider credentials. - **grantType** (string): GrantType is the method application gets access token. - **aud** (string): Aud defines the audience that this ID Token is intended for. #### BackendSecurityPolicySpec **Appears in:** - [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicy) BackendSecurityPolicySpec specifies authentication rules on access the provider from the Gateway. Only one mechanism to access a backend(s) can be specified. Only one type of BackendSecurityPolicy can be defined. ##### Fields - **targetRefs** ([LocalPolicyTargetReference](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1alpha2.LocalPolicyTargetReference) array): TargetRefs are the names of the AIServiceBackend or InferencePool resources this BackendSecurityPolicy is being attached to. Attaching multiple BackendSecurityPolicies to the same resource is invalid and will result in an error during the reconciliation of the resource. - **type** ([BackendSecurityPolicyType](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicytype), required): Type specifies the type of the backend security policy. - **apiKey** ([BackendSecurityPolicyAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyapikey)): APIKey is a mechanism to access a backend(s). The API key will be injected into the Authorization header. - **awsCredentials** ([BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyawscredentials)): AWSCredentials is a mechanism to access a backend(s). AWS specific logic will be applied. - **azureAPIKey** ([BackendSecurityPolicyAzureAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyazureapikey)): AzureAPIKey is a mechanism to access Azure OpenAI backend(s). The API key will be injected into the api-key header. - **azureCredentials** ([BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyazurecredentials)): AzureCredentials is a mechanism to access a backend(s). Azure OpenAI specific logic will be applied. - **gcpCredentials** ([BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicygcpcredentials)): GCPCredentials is a mechanism to access a backend(s). GCP specific logic will be applied. - **anthropicAPIKey** ([BackendSecurityPolicyAnthropicAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyanthropicapikey)): AnthropicAPIKey is a mechanism to access Anthropic backend(s). The API key will be injected into the `x-api-key` header. https://docs.claude.com/en/api/overview#authentication - **credentialOverride** ([BackendSecurityPolicyCredentialOverride](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicycredentialoverride)): CredentialOverride, when set, sources the upstream credential per-request instead of using the static credential configured above. For AWSCredentials the credential is the three-part SigV4 input rather than a single string; see the source types for how each carries it. #### BackendSecurityPolicyStatus **Appears in:** - [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicy) BackendSecurityPolicyStatus contains the conditions by the reconciliation result. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array, required): Conditions is the list of conditions by the reconciliation result. Currently, at most one condition is set. Known .status.conditions.type are: `Accepted`, `NotAccepted`. #### BackendSecurityPolicyType **Underlying type:** string **Appears in:** - [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) BackendSecurityPolicyType specifies the type of auth mechanism used to access a backend. ##### Possible Values - **APIKey** (enum): - **AWSCredentials** (enum): - **AzureAPIKey** (enum): - **AnthropicAPIKey** (enum): - **AzureCredentials** (enum): - **GCPCredentials** (enum): #### CredentialOverrideFromDynamicMetadata **Appears in:** - [BackendSecurityPolicyCredentialOverride](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicycredentialoverride) CredentialOverrideFromDynamicMetadata sources the per-request credential from Envoy dynamic metadata produced by a trusted filter running earlier in the filter chain. ##### Fields - **namespace** (string, required): Namespace is the Envoy metadata namespace written by the trusted filter. Example: `envoy.filters.http.ext_authz` List the namespace in the Gateway's GatewayConfig under extProc.metadataForwardingNamespaces. Envoy forwards only the namespaces listed there, and only untyped metadata is read. - **key** (string): Key is the metadata key within the namespace. Defaults to the x-aigw-* name for the auth type. For every type except AWSCredentials the value must be a string holding the credential. For AWSCredentials it must be a struct, since SigV4 needs three inputs: accessKeyId secretAccessKey sessionToken (optional; omit for long-lived credentials) Field names are fixed. accessKeyId without secretAccessKey, or the reverse, fails the request instead of falling back. Defaults to `x-aigw-aws-credentials`. - **fallbackToConfigured** (boolean): FallbackToConfigured controls behaviour when the metadata key is absent. true (default) falls back to the static credential configured in the policy. false returns a 401 to the client without forwarding the request. #### CredentialOverrideFromRequestHeaders **Appears in:** - [BackendSecurityPolicyCredentialOverride](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicycredentialoverride) CredentialOverrideFromRequestHeaders sources the per-request credential from a request header. ##### Fields - **header** (string): Header is the name of the request header that carries the credential. Defaults to the x-aigw-* header for the configured auth type: APIKey → x-aigw-api-key AnthropicAPIKey → x-aigw-anthropic-api-key AzureAPIKey → x-aigw-azure-api-key AzureCredentials → x-aigw-azure-access-token GCPCredentials → x-aigw-gcp-access-token For AWSCredentials this is a prefix, not a single header name, since SigV4 needs three inputs. Defaults to `x-aigw-aws-`, yielding: x-aigw-aws-access-key-id x-aigw-aws-secret-access-key x-aigw-aws-session-token (optional; omit for long-lived credentials) All three are stripped before the backend sees them. An access key ID without a secret access key, or the reverse, fails the request instead of falling back. Prefer FromDynamicMetadata for AWSCredentials. Headers are client-settable, and a secret access key on the wire is a worse leak than an API key. - **fallbackToConfigured** (boolean): FallbackToConfigured controls behaviour when the header is absent. true (default) falls back to the static credential configured in the policy. false returns a 401 to the client without forwarding the request. #### GCPCredentialsFile **Appears in:** - [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicygcpcredentials) GCPCredentialsFile specifies the service account key json file to authenticate with GCP provider. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference), required): SecretRef is the reference to the credential file. The secret should contain the GCP service account credentials file keyed on `service_account.json`. #### GCPOIDCExchangeToken **Appears in:** - [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityfederationconfig) ##### Fields - **oidc** ([OIDC](https://gateway.envoyproxy.io/docs/api/extension_types/#oidc), required): OIDC is used to obtain oidc tokens via an SSO server which will be used to exchange for provider credentials. - **grantType** (string): GrantType is the method application gets access token. - **aud** (string): Aud defines the audience that this ID Token is intended for. #### GCPServiceAccountImpersonationConfig **Appears in:** - [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityfederationconfig) ##### Fields - **serviceAccountName** (string, required): ServiceAccountName is the name of the service account to impersonate. #### GCPWorkloadIdentityFederationConfig **Appears in:** - [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicygcpcredentials) ##### Fields - **projectID** (string, required): ProjectID is the GCP project ID. - **workloadIdentityProviderName** (string, required): WorkloadIdentityProviderName is the name of the external identity provider as registered on Google Cloud Platform. - **oidcExchangeToken** ([GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpoidcexchangetoken), required): OIDCExchangeToken specifies the oidc configurations used to obtain an oidc token. The oidc token will be used to obtain temporary credentials to access GCP. - **workloadIdentityPoolName** (string, required): WorkloadIdentityPoolName is the name of the workload identity pool defined in GCP. https://cloud.google.com/iam/docs/workload-identity-federation?hl=en - **serviceAccountImpersonation** ([GCPServiceAccountImpersonationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpserviceaccountimpersonationconfig)): ServiceAccountImpersonation is the service account impersonation configuration. This is used to impersonate a service account when getting access token. #### GatewayConfigExtProc **Appears in:** - [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigspec) GatewayConfigExtProc holds configuration for the external processor. ##### Fields - **kubernetes** ([KubernetesContainerSpec](https://gateway.envoyproxy.io/docs/api/extension_types/#kubernetescontainerspec)): Kubernetes defines the configuration for running the external processor as a Kubernetes container. - **metadataForwardingNamespaces** (string array): MetadataForwardingNamespaces lists the untyped dynamic metadata namespaces Envoy forwards to the external processor for this Gateway's AI traffic. A BackendSecurityPolicy using credentialOverride.fromDynamicMetadata receives metadata only from a namespace listed here. Under Envoy Gateway's mergeGateways, every Gateway in the class shares one Envoy, so a namespace any one of them declares is forwarded for all of their AI traffic. #### GatewayConfigForwardProxy **Appears in:** - [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigspec) GatewayConfigForwardProxy configures an HTTP CONNECT forward proxy for upstream egress. ##### Fields - **address** (string, required): Address is the `host:port` of the HTTP CONNECT proxy. The host may be a hostname or an IP address; the port is required. Upstream connections are tunnelled through this proxy via Envoy's http_11_proxy transport socket, preserving the upstream TLS session. #### GatewayConfigSpec **Appears in:** - [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfig) GatewayConfigSpec defines the configuration for the AI Gateway. ##### Fields - **extProc** ([GatewayConfigExtProc](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigextproc)): ExtProc defines the configuration for the external processor container. - **globalLLMRequestCosts** ([LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1beta1-llmrequestcost) array): GlobalLLMRequestCosts defines default LLM request costs that apply to all routes referencing this GatewayConfig. These costs can be overridden on a per-route basis via AIGatewayRoute.Spec.LLMRequestCosts. When a request matches a route, the cost calculation proceeds as follows: 1. If the route defines LLMRequestCosts with a matching metadataKey, use that. 2. Otherwise, fall back to the global cost with that metadataKey (if defined here). 3. If neither exists, the cost is not calculated for that metadataKey. This allows you to define common cost formulas once at the gateway level (e.g., billing_charges = input_tokens + output_tokens) and only override them in specific routes when needed (e.g., premium routes with different pricing). - **forwardProxy** ([GatewayConfigForwardProxy](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigforwardproxy)): ForwardProxy routes all upstream AI/LLM traffic from Gateways referencing this GatewayConfig through an HTTP CONNECT forward proxy. This is intended for data planes in locked-down networks where direct egress to providers is not permitted and all outbound traffic must traverse a proxy. #### GatewayConfigStatus **Appears in:** - [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfig) GatewayConfigStatus defines the observed state of GatewayConfig. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array): Conditions describe the current conditions of the GatewayConfig. #### HTTPBodyField **Appears in:** - [HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpbodymutation) HTTPBodyField represents a JSON field name and value for body mutation ##### Fields - **path** (string, required): Path is the top-level field name to set in the request body. Examples: `service_tier`, `max_tokens`, `temperature` - **value** (string, required): Value is the JSON value to set at the specified field. This can be any valid JSON value: string, number, boolean, object, array, or null. The value will be parsed as JSON and inserted at the specified field. Examples: - `\`scale\`` (string) - `42` (number) - `true` (boolean) - `\{\`key\`: \`value\`\}` (object) - `[1, 2, 3]` (array) - `null` (null) #### HTTPBodyMutation **Appears in:** - [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulebackendref) - [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) HTTPBodyMutation defines the mutation of HTTP request body JSON fields that will be applied to the request ##### Fields - **set** ([HTTPBodyField](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpbodyfield) array): Set overwrites/adds the request body with the given JSON field (name, value) before sending to the backend. Only top-level fields are currently supported. Input: \{ `model`: `gpt-4`, `service_tier`: `default` \} Config: set: - path: `service_tier` value: `scale` Output: \{ `model`: `gpt-4`, `service_tier`: `scale` \} - **remove** (string array): Remove the given JSON field(s) from the HTTP request body before sending to the backend. The value of Remove is a list of top-level field names to remove. Input: \{ `model`: `gpt-4`, `service_tier`: `default`, `internal_flag`: true \} Config: remove: [`service_tier`, `internal_flag`] Output: \{ `model`: `gpt-4` \} #### HTTPHeaderMutation **Appears in:** - [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulebackendref) - [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) HTTPHeaderMutation defines the mutation of HTTP headers that will be applied to the request ##### Fields - **set** ([HTTPHeader](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#httpheader) array): Set overwrites/adds the request with the given header (name, value) before the action. Input: GET /foo HTTP/1.1 my-header: foo Config: set: - name: `my-header` value: `bar` Output: GET /foo HTTP/1.1 my-header: bar - **remove** (string array): Remove the given header(s) from the HTTP request before the action. The value of Remove is a list of HTTP header names. Note that the header names are case-insensitive (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). Input: GET /foo HTTP/1.1 my-header1: foo my-header2: bar my-header3: baz Config: remove: [`my-header1`, `my-header3`] Output: GET /foo HTTP/1.1 my-header2: bar #### HTTPHeaderValueFilter **Appears in:** - [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) HTTPHeaderValueFilter filters individual values out of a multi-valued request header before the request is forwarded to this backend. This drops (or restricts to) specific values from a comma-separated header while forwarding the rest untouched, so one unsupported value does not fail the whole request. Only comma-delimited headers are supported. Values are matched exactly, after surrounding whitespace is trimmed. To remove a header entirely, use HeaderMutation.Remove instead. ##### Fields - **name** ([HTTPHeaderName](#sigs-k8s-io-gateway-api-apis-v1-httpheadername), required): Name is the name of the header whose values are filtered. Header names are case-insensitive (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). - **mode** ([HTTPHeaderValueFilterMode](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadervaluefiltermode)): Mode selects the filtering semantics. Defaults to Denylist. - **values** (string array): Values is the list of header values to drop (Denylist) or keep (Allowlist). An empty list leaves the header unchanged. #### HTTPHeaderValueFilterMode **Underlying type:** string **Appears in:** - [HTTPHeaderValueFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadervaluefilter) HTTPHeaderValueFilterMode specifies the filtering semantics of an HTTPHeaderValueFilter. ##### Possible Values - **Denylist** (enum): HTTPHeaderValueFilterModeDenylist drops the values listed in Values, keeping everything else. New values the gateway has not been configured for pass through untouched. - **Allowlist** (enum): HTTPHeaderValueFilterModeAllowlist keeps only the values listed in Values, dropping everything else. New values the gateway has not been configured for are dropped. #### JWKS **Appears in:** - [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteoauth) JWKS defines how to obtain JSON Web Key Sets (JWKS) either from a remote HTTP/HTTPS endpoint or from a local source. ##### Fields - **remoteJWKS** ([RemoteJWKS](https://gateway.envoyproxy.io/docs/api/extension_types/#remotejwks)): RemoteJWKS defines how to fetch and cache JSON Web Key Sets (JWKS) from a remote HTTP/HTTPS endpoint. - **localJWKS** ([LocalJWKS](https://gateway.envoyproxy.io/docs/api/extension_types/#localjwks)): LocalJWKS defines how to get the JSON Web Key Sets (JWKS) from a local source. #### JWTSource **Appears in:** - [MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationsource) JWTSource defines the MCP authorization source for JWT tokens. At least one of scopes or claims must be provided. Scopes and claims are AND-ed: when both are specified, both sets must match. ##### Fields - **scopes** (JWTScope array): Scopes defines the list of JWT scopes required for the rule. If multiple scopes are specified, all scopes must be present in the JWT for the rule to match. - **claims** (JWTClaim array): Claims defines the list of JWT claims required for the rule. Each claim must exist on the token and have at least one of the expected values. Use to enforce tenant or subject-based access. #### LLMRequestCost **Appears in:** - [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutespec) - [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigspec) LLMRequestCost configures each request cost. ##### Fields - **metadataKey** (string, required): MetadataKey is the key of the metadata to store this cost of the request. - **type** ([LLMRequestCostType](#github-com-envoyproxy-ai-gateway-api-v1beta1-llmrequestcosttype), required): Type specifies the type of the request cost. The default is `OutputToken`, and it uses `output token` as the cost. The other types are `InputToken`, `TotalToken`, `CachedInputToken`, `CacheCreationInputToken`, `ReasoningToken`, and `CEL`. - **cel** (string): CEL is the CEL expression to calculate the cost of the request. The CEL expression must return a signed or unsigned integer. If the return value is negative, it will be error. The expression can use the following variables: * model: the model name extracted from the request content. Type: string. * backend: the backend name in the form of `name.namespace`. Type: string. * input_tokens: the number of input tokens. Type: unsigned integer. * cached_input_tokens: the number of cached read input tokens. Type: unsigned integer. * cache_creation_input_tokens: the number of cache creation input tokens. Type: unsigned integer. * output_tokens: the number of output tokens. Type: unsigned integer. * total_tokens: the total number of tokens. Type: unsigned integer. * reasoning_tokens: the number of reasoning tokens. Type: unsigned integer. For example, the following expressions are valid: * `model == 'llama' ? input_tokens + output_token * 0.5 : total_tokens` * `backend == 'foo.default' ? input_tokens + output_tokens : total_tokens` * `backend == 'bar.default' ? (input_tokens - cached_input_tokens) + cached_input_tokens * 0.1 + cache_creation_input_tokens * 1.25 + output_tokens : total_tokens` * `input_tokens + output_tokens + total_tokens` * `input_tokens * output_tokens` #### LLMRequestCostType **Underlying type:** string **Appears in:** - [LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1beta1-llmrequestcost) LLMRequestCostType specifies the type of the LLMRequestCost. ##### Possible Values - **InputToken** (enum): LLMRequestCostTypeInputToken is the cost type of the input token. - **CachedInputToken** (enum): LLMRequestCostTypeCachedInputToken is the cost type of the cached input token. - **CacheCreationInputToken** (enum): LLMRequestCostTypeCacheCreationInputToken is the cost type of the cached input token. - **OutputToken** (enum): LLMRequestCostTypeOutputToken is the cost type of the output token. - **TotalToken** (enum): LLMRequestCostTypeTotalToken is the cost type of the total token. - **ReasoningToken** (enum): LLMRequestCostTypeReasoningToken is the cost type of the reasoning token. - **CEL** (enum): LLMRequestCostTypeCEL is for calculating the cost using the CEL expression. #### MCPAuthorizationSource **Appears in:** - [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorizationrule) MCPAuthorizationSource defines the source of an authorization rule. ##### Fields - **jwt** ([JWTSource](#github-com-envoyproxy-ai-gateway-api-v1beta1-jwtsource), required): JWT defines the JWT scopes required for this rule to match. #### MCPAuthorizationTarget **Appears in:** - [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorizationrule) MCPAuthorizationTarget defines the target of an authorization rule. ##### Fields - **tools** ([ToolCall](#github-com-envoyproxy-ai-gateway-api-v1beta1-toolcall) array, required): Tools defines the list of tools this rule applies to. #### MCPBackendAPIKey **Appears in:** - [MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendsecuritypolicy) MCPBackendAPIKey defines the configuration for the API Key Authentication to a backend. When both `header` and `queryParam` are unspecified, the API key will be injected into the "Authorization" header by default. ##### Fields - **secretRef** ([SecretObjectReference](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.SecretObjectReference)): secretRef is the Kubernetes secret which contains the API keys. The key of the secret should be `apiKey`. - **inline** (string): Inline contains the API key as an inline string. - **header** (string): Header is the HTTP header to inject the API key into. If not specified, defaults to `Authorization`. When the header is `Authorization`, the injected header value will be prefixed with `Bearer `. Either one of Header or QueryParam can be specified to inject the API key. - **queryParam** (string): QueryParam is the HTTP query parameter to inject the API key into. For example, if QueryParam is set to `api_key`, and the API key is `mysecretkey`, the request URL will be modified to include `?api_key=mysecretkey`. Either one of Header or QueryParam can be specified to inject the API key. Note: Embedding credentials in URLs (including query parameters) is generally not recommended because URLs can be exposed in logs and intermediary systems; prefer header-based injection when possible. - **injectionPolicy** ([MCPBackendAPIKeyInjectionPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendapikeyinjectionpolicy)): InjectionPolicy controls when the configured API key is written onto the target header. Always (the default) writes the credential, replacing any existing value, including values populated by forwardHeaders. IfNotPresent writes the API key only if the target header is absent, so a caller-supplied token forwarded onto the same header is preserved. InjectionPolicy applies only to header injection. It must not be set to IfNotPresent when queryParam is used, because query-parameter injection always rewrites the backend URL. #### MCPBackendAPIKeyInjectionPolicy **Underlying type:** string **Appears in:** - [MCPBackendAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendapikey) MCPBackendAPIKeyInjectionPolicy controls when the configured API key is written onto the request. ##### Possible Values - **Always** (enum): MCPBackendAPIKeyInjectionAlways writes the configured key onto the target header, replacing any existing value. This is the default and preserves existing behavior. - **IfNotPresent** (enum): MCPBackendAPIKeyInjectionIfNotPresent writes the configured key only when the target header is absent, so a value already set (for example by forwardHeaders) is preserved. #### MCPBackendSecurityPolicy **Appears in:** - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) MCPBackendSecurityPolicy defines the security policy for a backend MCP server. ##### Fields - **apiKey** ([MCPBackendAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendapikey)): APIKey is a mechanism to access a backend. The API key will be injected into the request headers. #### MCPBackendSelector **Appears in:** - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) MCPBackendSelector defines which of a MCPRoute's backends a request may fan out to. ##### Fields - **defaultAction** ([AuthorizationAction](#github-com-envoyproxy-gateway-api-v1alpha1-authorizationaction)): DefaultAction is the action to take when no rules match. If unspecified, defaults to Deny. - **rules** ([MCPBackendSelectorRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendselectorrule) array): Rules defines a list of backend selection rules. These rules are evaluated in order, the first matching rule will be applied, and the rest will be skipped. If no rules are defined, DefaultAction is applied to every candidate backend (defaults to Deny). #### MCPBackendSelectorRule **Appears in:** - [MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendselector) MCPBackendSelectorRule defines a single backend selection rule. ##### Fields - **cel** (string): CEL specifies a Common Expression Language (CEL) expression, evaluated once for each backend already listed in backendRefs. It does not parse a list of backends out of the request. Each evaluation binds request.mcp.backend to the name of the one candidate backend under test, so the expression should answer `is this backend allowed`, not `which backends should be used`. The expression must return a boolean; evaluation errors or non-boolean results are treated as `no match`. A match means the rule applies, and Action below decides whether that's an Allow or a Deny for that backend. Example CEL expressions: * `request.mcp.backend in request.auth.jwt.claims.mcp_backends` * `(`,` + request.headers[`x-ai-eg-mcp-backend-subset`] + `,`).contains(`,` + request.mcp.backend + `,`)` Available attributes are the same as documented on MCPRouteAuthorizationRule.CEL, except request.mcp.method, request.mcp.tool, and request.mcp.params are not populated (no MCP method has been selected yet at backend-selection time). Note: The CEL expression support is experimental, and the attributes available to the expression may change in future releases. - **action** ([AuthorizationAction](#github-com-envoyproxy-gateway-api-v1alpha1-authorizationaction)): Action is the decision for matching backends. If unspecified, defaults to Allow. #### MCPHeaderForward **Appears in:** - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) MCPHeaderForward specifies a header to extract from the incoming request and forward to a backend. ##### Fields - **name** (string, required): Name is the header name to extract from the incoming client request. - **backendHeader** (string): BackendHeader is the header name to use when forwarding to the backend. If not specified, the original header name is used. #### MCPPromptFilter **Appears in:** - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) MCPPromptFilter filters prompts using include and exclude patterns with exact matches or regular expressions. Exclude rules take precedence over include rules (deny-wins). When both include and exclude are specified, a prompt must match an include rule AND not match any exclude rule to be allowed. ##### Fields - **include** (string array): Include is a list of prompt names to include. Only the specified prompts will be available. - **includeRegex** (string array): IncludeRegex is a list of RE2-compatible regular expressions that, when matched, include the prompt. Only prompts matching these patterns will be available. - **exclude** (string array): Exclude is a list of prompt names to exclude. The specified prompts will not be available. Exclude rules take precedence over include rules. - **excludeRegex** (string array): ExcludeRegex is a list of RE2-compatible regular expressions that, when matched, exclude the prompt. Prompts matching these patterns will not be available. Exclude rules take precedence over include rules. #### MCPRouteAuthorization **Appears in:** - [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutesecuritypolicy) MCPRouteAuthorization defines the authorization configuration for a MCPRoute. ##### Fields - **defaultAction** ([AuthorizationAction](#github-com-envoyproxy-gateway-api-v1alpha1-authorizationaction)): DefaultAction is the action to take when no rules match. If unspecified, defaults to Deny. - **rules** ([MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorizationrule) array): Rules defines a list of authorization rules. These rules are evaluated in order, the first matching rule will be applied, and the rest will be skipped. If no rules are defined, the default action will be applied to all requests. #### MCPRouteAuthorizationRule **Appears in:** - [MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorization) MCPRouteAuthorizationRule defines an authorization rule for MCPRoute based on the MCP authorization spec. Reference: https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-challenge-handling ##### Fields - **source** ([MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationsource), required): Source defines the authorization source for this rule. If not specified, the rule will match all sources. - **target** ([MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationtarget), required): Target defines the authorization target for this rule. If not specified, the rule will match all targets. - **cel** (string): CEL specifies a Common Expression Language (CEL) expression evaluated for this rule. The expression must return a boolean; evaluation errors or non-boolean results are treated as `no match`. Example CEL expressions: * `request.method == `POST`` * `request.headers[`x-custom-header`] == `AllowedValue`` * `request.mcp.tool in [`toolA`, `toolB`]` Available attributes in the CEL expression: * request.method: HTTP method such as GET or POST. Type: string. * request.headers: map of headers with lowercased keys, first value only. Type: map[string]string. * request.headers_all: map of headers with lowercased keys, all values. Type: map[string][]string. * request.path: request path such as /mcp. Type: string. * request.auth.jwt.claims: JWT claims when a bearer JWT is present. Type: map[string]any. * request.auth.jwt.scopes: JWT scopes when a bearer JWT is present. Type: []string. * request.mcp.method: MCP method such as tools/list or tools/call. Type: string. * request.mcp.backend: upstream backend name (for example, `kiwi` or `github`). Type: string. * request.mcp.tool: tool name without backend prefix (for example, `list_issues`). Type: string. * request.mcp.params: parameters of the MCP method, including keys like `_meta` and `arguments`. Type: object. Note: The CEL expression support is experimental, and the attributes available to the expression may change in future releases. - **action** ([AuthorizationAction](#github-com-envoyproxy-gateway-api-v1alpha1-authorizationaction)): Action is the authorization decision for matching requests. If unspecified, defaults to Allow. #### MCPRouteBackendRef **Appears in:** - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) MCPRouteBackendRef wraps a EG's BackendObjectReference to reference an MCP server. ##### Fields - **group** ([Group](#sigs-k8s-io-gateway-api-apis-v1-group)): Group is the group of the referent. For example, `gateway.networking.k8s.io`. When unspecified or empty string, core API group is inferred. - **kind** ([Kind](#sigs-k8s-io-gateway-api-apis-v1-kind)): Kind is the Kubernetes resource kind of the referent. For example `Service`. Defaults to `Service` when not specified. ExternalName services can refer to CNAME DNS records that may live outside of the cluster and as such are difficult to reason about in terms of conformance. They also may not be safe to forward to (see CVE-2021-25740 for more information). Implementations SHOULD NOT support ExternalName Services. Support: Core (Services with a type other than ExternalName) Support: Implementation-specific (Services with type ExternalName) - **name** ([ObjectName](#sigs-k8s-io-gateway-api-apis-v1-objectname), required): Name is the name of the referent. - **namespace** ([Namespace](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#namespace)): Namespace is the namespace of the backend. When unspecified, the local namespace is inferred. Note that when a namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. Support: Core - **port** ([PortNumber](#sigs-k8s-io-gateway-api-apis-v1-portnumber)): Port specifies the destination port number to use for this resource. Port is required when the referent is a Kubernetes Service. In this case, the port number is the service port number, not the target port. For other resources, destination port might be derived from the referent resource or this field. - **path** (string): Path is the HTTP endpoint path of the backend MCP server. If not specified, the default is `/mcp`. - **toolSelector** ([MCPToolFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcptoolfilter)): ToolSelector filters the tools exposed by this MCP server. Supports exact matches and RE2-compatible regular expressions for both include and exclude patterns. If not specified, all tools from the MCP server are exposed. - **promptSelector** ([MCPPromptFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcppromptfilter)): PromptSelector filters the prompts exposed by this MCP server, analogous to ToolSelector. If not specified, all prompts from the MCP server are exposed. This is also used to determine which prompt names can be safely exposed unprefixed when this backend's effective PrefixMode is Never: only prompts named in Include are validated for cross-backend uniqueness and exposed bare. If unset while PrefixMode is Never, this backend's prompts are not exposed bare (they keep the `__` prefix) because their names cannot be proven unique at admission time. - **securityPolicy** ([MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendsecuritypolicy)): SecurityPolicy is the security policy to apply to this MCP server. - **forwardHeaders** ([MCPHeaderForward](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpheaderforward) array): ForwardHeaders specifies HTTP headers to extract from the incoming client request and forward to this backend MCP server. This enables per-user authentication passthrough (e.g., personal access tokens) without requiring OAuth configuration. Each entry specifies a header name to extract and an optional rename for the backend. - **prefixMode** ([MCPRoutePrefixMode](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteprefixmode)): PrefixMode controls how tool and prompt names from this backend are prefixed. Takes precedence over the route-level PrefixMode when set. #### MCPRouteBackendTrafficPolicy **Appears in:** - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) MCPRouteBackendTrafficPolicy configures how the Envoy Gateway BackendTrafficPolicy resources auto-generated for a MCPRoute interact with other BackendTrafficPolicy resources (e.g. an operator-defined rate limiting policy) targeting the same HTTPRoute. ##### Fields - **mergeType** ([MergeType](#github-com-envoyproxy-gateway-api-v1alpha1-mergetype)): MergeType determines how the auto-generated BackendTrafficPolicy (for example, the OAuth protected resource metadata policy) is merged with other BackendTrafficPolicy configurations targeting the same route or a parent Gateway/Listener. When set, the auto-generated BackendTrafficPolicy will merge with the closest parent BackendTrafficPolicy in the route's attachment hierarchy instead of overriding it entirely, per Envoy Gateway's merge semantics. If unset, no merging occurs, and the auto-generated BackendTrafficPolicy fully overrides any other BackendTrafficPolicy targeting the same route. See: https://gateway.envoyproxy.io/docs/tasks/traffic/backend-traffic-policy/ #### MCPRouteOAuth **Appears in:** - [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutesecuritypolicy) MCPRouteOAuth defines a MCP spec compatible OAuth authentication configuration for a MCPRoute. Reference: https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization ##### Fields - **issuer** (string, required): Issuer is the authorization server's issuer identity. - **audiences** (string array): Audiences is a list of JWT audiences allowed access. It is recommended to set this field for token audience validation, as it is a security best practice to prevent token misuse. Reference: https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#token-audience-binding-and-validation - **jwks** ([JWKS](#github-com-envoyproxy-ai-gateway-api-v1beta1-jwks)): JWKS defines how a JSON Web Key Sets (JWKS) can be obtained to verify the access tokens presented by the clients. If not specified, the JWKS URI will be discovered from the OAuth 2.0 Authorization Server Metadata as per RFC 8414 by querying the `/.well-known/oauth-authorization-server` endpoint on the Issuer. - **authorizationServerMetadataUrl** (string): AuthorizationServerMetadataURL is the URL the controller fetches the OAuth 2.0 Authorization Server Metadata document from, as defined in RFC 8414. When set, it replaces the well-known URIs derived from Issuer. Set this when the metadata lives somewhere the issuer does not lead to, for example an issuer of `https://example.com/api/idp/authn` whose document is served only at `https://example.com/api/idp/v4/authn/.well-known/openid-configuration`. Issuer is unaffected by this field. It continues to identify the authorization server in the protected resource metadata the gateway publishes, and to derive the well-known URIs when this field is unset. When JWKS is not set, the JWKS URI is discovered from the document fetched here. - **protectedResourceMetadata** ([ProtectedResourceMetadata](#github-com-envoyproxy-ai-gateway-api-v1beta1-protectedresourcemetadata), required): ProtectedResourceMetadata defines the OAuth 2.0 Resource Server Metadata as per RFC 8414. This is used to expose the metadata endpoint for mcp clients to discover the authorization servers, supported scopes, and JWKS URI. - **claimToHeaders** (ClaimToHeader array): ClaimToHeaders specifies JWT claims to extract and forward as HTTP headers to backend MCP servers. This enables backends to access user identity for authorization, auditing, or personalization. Security considerations: - Any client-provided headers matching the configured header names will be stripped to prevent forgery - Only the specified claims are extracted; the full JWT is not forwarded to backends - Consider using a header prefix (e.g., `X-Jwt-Claim-`) to avoid conflicts with other headers #### MCPRoutePrefixMode **Underlying type:** string **Appears in:** - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) MCPRoutePrefixMode controls how tool and prompt names are prefixed when exposed to clients. PrefixMode can be set at the route level (MCPRouteSpec) and overridden per backend (MCPRouteBackendRef); the per-backend value takes precedence when set. A single route can therefore mix Always and Never backends. This is intentional: it lets an operator keep the route-level default (Always is recommended for most routes, since it guarantees a collision-free catalog without any extra configuration) while opting specific, well-known backends into bare names, e.g. to match tool names an existing client or interactive MCP App already hardcodes. Per-backend override is meant as a targeted escape hatch, not a replacement for choosing one naming policy for the route as a whole. ##### Possible Values - **Always** (enum): MCPRoutePrefixModeAlways prefixes tool/prompt names with `__`. This is the default and preserves existing behavior. - **Never** (enum): MCPRoutePrefixModeNever exposes tool/prompt names unprefixed. Tool names must be unique across all Never-mode backends on the route. #### MCPRouteSecurityPolicy **Appears in:** - [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) MCPRouteSecurityPolicy defines the security policy for a MCPRoute. ##### Fields - **oauth** ([MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteoauth)): OAuth defines the configuration for the MCP spec compatible OAuth authentication. - **apiKeyAuth** ([APIKeyAuth](https://gateway.envoyproxy.io/docs/api/extension_types/#apikeyauth)): APIKeyAuth defines the configuration for the API Key Authentication. - **extAuth** ([ExtAuth](#github-com-envoyproxy-gateway-api-v1alpha1-extauth)): ExtAuth defines the configuration for External Authorization. - **authorization** ([MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorization)): Authorization defines the configuration for the MCP spec compatible authorization. - **mergeType** ([MergeType](#github-com-envoyproxy-gateway-api-v1alpha1-mergetype)): MergeType determines how the auto-generated SecurityPolicy for this MCPRoute (which enforces OAuth/JWT, API key, or external authorization) is merged with other SecurityPolicy configurations targeting the same route or a parent Gateway/Listener. When set, the auto-generated SecurityPolicy will merge with the closest parent SecurityPolicy in the route's attachment hierarchy (for example, one targeting a Gateway, Gateway listener, ListenerSet, or ListenerSet listener) instead of overriding it entirely, per Envoy Gateway's merge semantics. If unset, no merging occurs, and the auto-generated SecurityPolicy fully overrides any other SecurityPolicy targeting the same route. See: https://gateway.envoyproxy.io/docs/tasks/security/apikey-authn/ #### MCPRouteSpec **Appears in:** - [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproute) MCPRouteSpec details the MCPRoute configuration. ##### Fields - **parentRefs** ([ParentReference](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#parentreference) array, required): ParentRefs are the names of the Gateway resources this MCPRoute is being attached to. Cross namespace references are not supported. In other words, the Gateway resources must be in the same namespace as the MCPRoute. Currently, each reference's Kind must be Gateway. - **path** (string): Path is the HTTP endpoint path that serves MCP requests over the Streamable HTTP transport. If not specified, the default is `/mcp`. - **headers** ([HTTPHeaderMatch](https://gateway-api.sigs.k8s.io/reference/spec/?h=httproutetimeouts#httpheadermatch) array): Headers are HTTP headers that must match for this route to be selected. Multiple match values are ANDed together, meaning, a request must match all the specified headers to select the route. - **hostnames** (Hostname array): Hostnames is a list of hostnames matched against the HTTP Host header to select this MCPRoute. This is equivalent to the Hostnames field in the Gateway API HTTPRouteSpec. When specified, the generated HTTPRoute includes these hostnames, so the MCP route lands in the per-host virtual host instead of the wildcard vhost (the same scoping AIGatewayRoute.Hostnames provides). See https://gateway-api.sigs.k8s.io/reference/api-types/httproute/#hostnames for the details of the Hostnames field in the Gateway API. - **backendRefs** ([MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) array, required): BackendRefs is a list of backend references to the MCP servers. These MCP servers will be aggregated and exposed as a single MCP endpoint to the clients. From the client's perspective, they only need to configure a single MCP server URL, e.g. `https://api.example.com/mcp`, and the Envoy AI Gateway will route the requests to the appropriate MCP server based on the requests. All names must be unique within this list to avoid potential tools, resources, etc. name collisions. Also, cross-namespace references are not supported. In other words, the backend MCP servers must be in the same namespace as the MCPRoute. - **securityPolicy** ([MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutesecuritypolicy)): SecurityPolicy defines the security policy for this MCPRoute. - **backendTrafficPolicy** ([MCPRouteBackendTrafficPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendtrafficpolicy)): BackendTrafficPolicy configures the auto-generated Envoy Gateway BackendTrafficPolicy resources for this MCPRoute, for example the OAuth protected resource metadata policy created when SecurityPolicy.OAuth is configured. - **backendSelector** ([MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendselector)): BackendSelector restricts which of this route's backends a given request may fan out to, evaluated once per candidate backend when a client session is initialized. If unspecified, all backends on the route are considered. Security note: a backendSelector rule whose CEL expression reads request.auth.jwt (claims or scopes) only reflects verified JWT data when SecurityPolicy.OAuth is configured for this route; the XValidation below enforces that pairing so that unauthenticated or forged bearer tokens can't influence backend selection. - **prefixMode** ([MCPRoutePrefixMode](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteprefixmode)): PrefixMode controls how tool and prompt names from all backends on this route are prefixed. #### MCPRouteStatus **Appears in:** - [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproute) MCPRouteStatus contains the conditions by the reconciliation result. ##### Fields - **conditions** ([Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#condition-v1-meta) array, required): Conditions is the list of conditions by the reconciliation result. Currently, at most one condition is set. Known .status.conditions.type are: `Accepted`, `NotAccepted`. #### MCPToolFilter **Appears in:** - [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) MCPToolFilter filters tools using include and exclude patterns with exact matches or regular expressions. Exclude rules take precedence over include rules (deny-wins). When both include and exclude are specified, a tool must match an include rule AND not match any exclude rule to be allowed. ##### Fields - **include** (string array): Include is a list of tool names to include. Only the specified tools will be available. - **includeRegex** (string array): IncludeRegex is a list of RE2-compatible regular expressions that, when matched, include the tool. Only tools matching these patterns will be available. - **exclude** (string array): Exclude is a list of tool names to exclude. The specified tools will not be available. Exclude rules take precedence over include rules. - **excludeRegex** (string array): ExcludeRegex is a list of RE2-compatible regular expressions that, when matched, exclude the tool. Tools matching these patterns will not be available. Exclude rules take precedence over include rules. #### ProtectedResourceMetadata **Appears in:** - [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteoauth) ProtectedResourceMetadata represents the Protected Resource Metadata of the MCP server as per RFC 9728. References: * https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#authorization-server-location * https://datatracker.ietf.org/doc/html/rfc9728#name-protected-resource-metadata ##### Fields - **resource** (string, required): Resource is the identifier of the protected resource. This should match the MCPRoute's URL. For example, if the MCPRoute's URL is `https://api.example.com/mcp`, the Resource should be `https://api.example.com/mcp`. - **resourceName** (string): ResourceName is a human-readable name for the protected resource. - **scopesSupported** (string array): ScopesSupported defines the minimal set of scopes required for the basic functionality of the MCPRoute. It should avoid broad or overly permissive scopes to prevent clients from requesting tokens with excessive privileges. If an operation requires additional scopes that are not present in the access token, the client will receive a 403 Forbidden response that includes the required scopes in the `scope` field of the `WWW-Authenticate` header. This enables incremental privilege elevation through targeted `WWW-Authenticate: scope=`...`` challenges when privileged operations are first attempted. - **resourceSigningAlgValuesSupported** (string array): ResourceSigningAlgValuesSupported is a list of JWS signing algorithms supported by the resource server. These algorithms are used in the `alg` field of the JOSE header in signed tokens. - **resourceDocumentation** (string): ResourceDocumentation is a URL that provides human-readable documentation for the resource. - **resourcePolicyUri** (string): ResourcePolicyURI is a URL that points to the resource server's policy document. #### ToolCall **Appears in:** - [MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationtarget) ToolCall represents a tool call in the MCP authorization target. ##### Fields - **backend** (string, required): Backend is the name of the backend this tool belongs to. - **tool** (string, required): Tool is the name of the tool. #### VersionedAPISchema **Appears in:** - [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) VersionedAPISchema defines the API schema of either AIGatewayRoute (the input) or AIServiceBackend (the output). This allows the ai-gateway to understand the input and perform the necessary transformation depending on the API schema pair (input, output). Note that this is vendor specific, and the stability of the API schema is not guaranteed by the ai-gateway, but by the vendor via proper versioning. ##### Fields - **name** ([APISchema](#github-com-envoyproxy-ai-gateway-api-v1beta1-apischema), required): Name is the name of the API schema of the AIGatewayRoute or AIServiceBackend. - **version** (string): Version is the version of the API schema. When the name is set to AzureOpenAI, this version maps to `API Version` in the Azure OpenAI API documentation (https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#rest-api-versioning). This field is ignored for OpenAI, AWSBedrock, GCPVertexAI, and Anthropic. For OpenAI and Anthropic, use prefix to configure custom request paths. See https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-providers for details. - **prefix** (string): Prefix is the prefix for the API. When the name is set to `OpenAI`, `chat completions` API endpoint will be `$\{this_field\}/chat/completions`. When the name is set to `Anthropic`, the `messages` API endpoint will be `$\{this_field\}/messages`. It can be with or without a leading slash (`/`). This field is ignored for AWSAnthropic and GCPAnthropic. This is especially useful when routing to a backend that has an OpenAI or Anthropic compatible API but has a different prefix. For example, Gemini OpenAI compatible API (https://ai.google.dev/gemini-api/docs/openai) uses `/v1beta/openai` prefix. Another example is that Cohere AI (https://docs.cohere.com/v2/docs/compatibility-api) uses `/compatibility/v1` prefix. On the other hand, DeepSeek (https://api-docs.deepseek.com/) doesn't use prefix, so you can leave this field unset. See https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-providers for details. --- # Agent Router 1.2: safe to share across users and teams Source: https://theagentrouter.ai/blog/v1.2-release-announcement Date: 2026-10-09 ![Agent Router 1.2: safe to share across users and teams](https://theagentrouter.ai/img/og/blog/v1.2-release-announcement.png) **Agent Router 1.2 is out.** It is the first release since Envoy AI Gateway became Agent Router, and nothing you deploy is renamed: same CRDs, same Helm charts, same `aigw` CLI. The short version: 1.2 makes the gateway safe to share. Safe to put your MCP tools in front of real users. Safe to run one gateway for many teams. And easier to trust when something goes wrong in production. ## Put MCP tools in front of real users When an agent calls a tool through the gateway, the gateway decides who the caller is and what they can touch. 1.2 tightens every step of that decision: - **Only verified identity counts.** Authorization rules read JWT claims only from tokens Envoy has verified, and the token's issuer must match the one you configured. - **Broken rules deny.** A CEL policy that errors at runtime now blocks the call instead of being skipped. - **Sessions belong to one user.** An MCP session is bound to the user who started it, so someone else can't reuse its session ID. - **No shared default key.** The Helm chart generates a random session encryption seed instead of shipping a well-known one. ## MCP that fits the clients you already have - **Clean tool names.** Clients that hardcode tool names, such as MCP Apps, can see `search_issues` instead of `github__search_issues` (`prefixMode: Never`). - **Bring your own token.** Users can forward a personal token, such as a GitHub PAT, and fall back to a shared service key when they don't (`injectionPolicy: IfNotPresent`). - **Keep your Gateway-wide policies.** Gateway-level rate limits and ext-auth can now stay in force alongside MCP OAuth instead of being replaced (`mergeType`). - **More identity providers.** Point the gateway at the exact OAuth metadata URL when your provider doesn't serve it at the standard path. ## One gateway, many teams When several teams share a gateway, one team's config should never reach another team's credentials. In 1.2, a BackendSecurityPolicy can use a Secret in another namespace only when that namespace allows it with a ReferenceGrant, for every credential type. A grant that names an object now covers only that object, and revoking a grant takes effect right away. ## Fewer surprises in production - **Rotated credentials apply immediately.** This fixes a 1.1.0 regression where upstream calls could fail with a stale key until the next reconcile. - **Compare providers side by side.** GenAI metrics now carry the backend, so you can compare latency, time to first token, and token usage for the same model across providers. - **Per-tenant token limits charge the right tenant**, and streaming usage is recorded even when the client disconnects right after the last chunk. - **Clear errors.** Calling an endpoint a provider doesn't support returns 422, not 500. - **Smoother rollouts.** The controller reports Ready only after its caches sync, and the webhook certificate survives `helm upgrade` and Argo CD syncs. On the provider side, 1.2 adds Amazon Bedrock's OpenAI-compatible endpoint (no translation hop) and TypeSafe System One. It also filters `anthropic-beta` flags per backend, so one unsupported flag no longer fails the whole request, and counts OpenAI prompt-cache writes in your cost metrics. ## Before you upgrade Some of these fixes change behavior. Upgrade from 1.1.x (not directly from 1.0.x), and check these first: 1. **Upgrade Envoy Gateway to v1.9.2 and the Gateway API CRDs to v1.6.** 2. **Expect MCP clients to reconnect.** The new random seed ends active sessions. A temporary fallback seed keeps them alive. 3. **Match `oauth.issuer` to your identity provider's `iss` exactly**, trailing slash included. 4. **Add `securityPolicy.oauth`** to any MCPRoute whose CEL reads `auth.jwt`, and make CEL null-safe. 5. **Add ReferenceGrants** for credential Secrets in other namespaces. 6. **Review per-tenant token limits.** They are now enforced correctly, so tenants may start seeing 429s. The [v1.2 release notes](https://theagentrouter.ai/release-notes/v1.2) have the full list and step-by-step upgrade guidance. ## Get involved 1.2 came from 39 contributors, many of them contributing for the first time. The next release can include you. **Get involved:** - **Come to the weekly community meeting.** We meet every Monday. [Join on Zoom](https://zoom-lfx.platform.linuxfoundation.org/meeting/91546415944?password=61fd5a5d-41e9-4b0c-86ea-b607c4513e37) and add your questions, use cases, or demos to the [agenda](https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w/edit?tab=t.0). - **Join the [Agent Router Discord](https://discord.gg/xuxtPq43gZ)** to ask questions and talk to the maintainers. - **Help build what's next.** Work is under way on the stateless 2026-07-28 MCP specification, [OAuth token exchange for MCP backends](https://github.com/theagentrouter/agent-router/issues/2036), a dedicated MCPBackend CRD, [longer quota windows](https://github.com/theagentrouter/agent-router/issues/2469), and [agent-to-agent (A2A) traffic](https://github.com/theagentrouter/agent-router/issues/2070). Share your requirements on the issues, or pick one up. - **Tell us how you use Agent Router.** [Add your logo](https://github.com/theagentrouter/agent-router/edit/main/site/src/data/adopters/adopters.json) to the adopters page, or share your story in Discord. - **Send your first pull request.** Start with [CONTRIBUTING.md](https://github.com/theagentrouter/agent-router/blob/main/CONTRIBUTING.md). Thank you to everyone who wrote code, reviewed pull requests, reported bugs, and joined the community meetings during the move to Agent Router. **[Get started with Agent Router 1.2](https://theagentrouter.ai/docs/getting-started/)** and [star the repository](https://github.com/theagentrouter/agent-router). --- # Envoy AI Gateway is becoming Agent Router, an Agentic AI Foundation project Source: https://theagentrouter.ai/blog/envoy-ai-gateway-is-now-agent-router Date: 2026-09-09 ![Agent Router. Formerly Envoy AI Gateway. Joining the Agentic AI Foundation on 10 September.](https://theagentrouter.ai/img/og/blog/envoy-ai-gateway-is-now-agent-router.png) On 10 September, **Agent Router**, previously **Envoy AI Gateway**, officially joins the **Agentic AI Foundation**. Same code. Same maintainers. New home. ## A new home Envoy AI Gateway started as an Envoy sub-project within the CNCF. On 10 September it becomes a standalone project and moves into the Agentic AI Foundation. This is great news for the community. The AAIF is home to the agentic stack ecosystem, and being part of it puts Agent Router closer to the projects and people building that stack. We get to collaborate more directly, and keep building the solutions that power agents together with everyone else in the foundation. ## What does not change Nothing you deploy is renamed. The product name changes in prose only. - **CRDs and API group stay as they are.** `AIGatewayRoute`, `AIServiceBackend`, `BackendSecurityPolicy`, and the `aigateway.envoyproxy.io` API group are not being renamed. - **The CLI is still `aigw`**, and the `envoy-ai-gateway-system` namespace stays the same. - **Same maintainers, same release cadence, same Apache 2.0 license.** - **Envoy stays underneath.** Agent Router is still built on Envoy and Envoy Gateway. Your manifests from yesterday apply tomorrow. There is nothing to migrate. ## What does change - **The name.** Envoy AI Gateway becomes Agent Router. - **The repository** moves to [`theagentrouter/agent-router`](https://github.com/theagentrouter/agent-router). Old `envoyproxy/ai-gateway` links redirect. - **The website** moves to [theagentrouter.ai](https://theagentrouter.ai). Every `aigateway.envoyproxy.io` link redirects to the same page. - **Community chat** moves to the [Agent Router Discord](https://discord.gg/xuxtPq43gZ). The weekly community meeting continues as before. ## Where to go next - [Get started](https://theagentrouter.ai/docs/getting-started/) or try it with one command on your laptop: `OPENAI_API_KEY=sk-... aigw run` - [Star the repository](https://github.com/theagentrouter/agent-router) and open issues and discussions there from now on. - [Join the Discord](https://discord.gg/xuxtPq43gZ) and the [weekly community meeting](https://zoom-lfx.platform.linuxfoundation.org/meeting/91546415944?password=61fd5a5d-41e9-4b0c-86ea-b607c4513e37). Thank you to everyone who built, adopted, reviewed, and championed Envoy AI Gateway over the past two years. The name on the door is new. The people behind it, and the code, are the same. **Agent Router configures. Envoy handles the traffic.** --- # Multi-User MCP in Production: Per-User Identity at the Gateway Layer Source: https://theagentrouter.ai/blog/multi-user-mcp-header-forwarding Date: 2026-07-05 ![Per-user identity at the gateway layer](https://theagentrouter.ai/img/og/blog/multi-user-mcp-header-forwarding.png) When teams first deploy MCP servers in production, the path of least resistance is a shared service account: one API token for Jira, one for Slack, one for GitHub — stored as Kubernetes Secrets and used for every request, regardless of who triggered the agent. This works until it doesn't. Comments post under the wrong name, users see data their own accounts can't access, and the RBAC rules carefully configured in your enterprise tools are silently bypassed. Envoy AI Gateway ships per-backend header forwarding in the `MCPRoute` API — available since v0.6 and part of the now-GA v1.0 — which solves this for a common enterprise case with a single YAML addition. This post covers how it works, how it plays out in a real production deployment, and — just as important — when you should reach for OAuth or token exchange instead. ## The Problem with Shared Service Accounts Consider an internal agent platform routing dozens of concurrent agents through Envoy AI Gateway, each serving a different engineer or SRE. With a shared Jira service account, every agent has the same access regardless of the triggering user's actual permissions. Slack tools can reach channels the triggering engineer isn't a member of. Jira comments appear under the service account's name, not the engineer's. Nothing is technically broken — and that is the problem. The permission model of every downstream tool is being quietly flattened into "whatever the bot can do." The instinct is to reach for OAuth. The reality is more nuanced, and it helps to first untangle what "authentication" actually means in an MCP deployment. ## Who Authenticates What: The Two Questions Every multi-user MCP setup has to answer two _separate_ questions, and most confusion comes from mixing them together: 1. **Does the gateway trust the incoming request?** — the front door. 2. **Whose name does the backend tool see?** — what happens inside. The first question is perimeter authentication. The gateway should verify that a request comes from someone (or something) in your organization before doing anything else. Envoy AI Gateway supports this through the `MCPRoute` security policy (API key or OAuth) or Envoy Gateway's `SecurityPolicy` (OIDC/JWT). You want this in production regardless of everything below. The second question is where multi-user identity lives. Once a request is inside, each backend MCP server needs _a_ credential — and each backend decides what the caller can see based on _who that credential says they are_. There are four answers, and each is right for a different situation: - **A shared service account.** The backend sees the same bot identity for every user. Right when the tool has no meaningful per-user permissions — a weather API, a public docs search. Wrong everywhere RBAC matters. - **Forward the user's own credential.** The user already holds a credential the backend accepts — a personal access token, or a corporate SSO token for internal tools. The gateway passes it through, and the backend sees the actual user. This is the pattern this post covers. - **The agent as its own identity.** A nightly automation that files summary tickets shouldn't impersonate anyone — it should _be_ a machine identity with narrow, fixed permissions (OAuth client credentials). Right for headless automation acting on its own behalf. - **Token exchange (RFC 8693).** The user's corporate token means nothing to an external SaaS on a different identity provider. A token service exchanges it for a downstream token that still represents the same user. Right for per-user identity _across_ an IdP boundary — at the cost of real infrastructure. There is also the full OAuth authorization code flow with browser consent, which the MCP specification mandates for interactive clients talking to external servers. It is the right answer when a human is present — and architecturally impossible for a headless agent firing at 3am with no browser. Lined up side by side, here is how the three headless per-user options compare against the interactive OAuth flow above — the shared service account is left out since it does not carry user identity at all: | Pattern | Headless? | Per-user RBAC? | New infrastructure? | Right when | | ------------------------- | --------- | ---------------- | ------------------------------------------- | ------------------------------------------ | | OAuth auth code + PKCE | ❌ | ✅ | OAuth Authorization Server (AS) per tool | Interactive clients, external SaaS | | Client credentials | ✅ | Fixed scope only | App registration | Agent acts as itself | | Token exchange (RFC 8693) | ✅ | ✅ | Security Token Service (STS) + trust config | Cross-IdP boundary | | **Header forwarding** | ✅ | ✅ | **None** | User's credential already works downstream | ![Decision path for choosing a downstream identity pattern](https://theagentrouter.ai/img/blog/multi-user-mcp-decision.png) ## How Header Forwarding Works The key design property is what _doesn't_ happen: Envoy AI Gateway's MCP proxy builds a fresh HTTP request for every upstream call. Client headers do not propagate to backend MCP servers by default — at all. Forwarding is an explicit, per-backend opt-in via `forwardHeaders` on each backend reference. ![Fan-out with per-backend header scoping](https://theagentrouter.ai/img/blog/multi-user-mcp-fanout.png) This matters most during fan-out. A single `tools/list` call fans out to every backend in the route, but each backend receives only the headers it explicitly opted into. A per-user Atlassian token configured for the Atlassian backend is never seen by any other backend in the same route. The absence of `forwardHeaders` on a backend means no identity propagation to that service, full stop. Each entry can also rename the header on the way through: `name` selects the inbound header, and an optional `backendHeader` sets the name used toward the backend — useful when a backend expects a vendor-specific header. ## Level 1: PAT Passthrough — Zero Infrastructure This is the simplest production-ready configuration. Many enterprise MCP servers — Atlassian, Glean, Sourcegraph — accept per-user authentication via custom HTTP headers carrying a personal access token (PAT). Every engineer already has one. The gateway just needs to deliver it to the right backend, and only that one: ```yaml apiVersion: aigateway.envoyproxy.io/v1beta1 kind: MCPRoute metadata: name: internal-tools namespace: default spec: parentRefs: - name: envoy-ai-gateway kind: Gateway group: gateway.networking.k8s.io path: "/mcp" backendRefs: # Atlassian MCP — forward each user's personal access tokens. - name: atlassian-mcp kind: Backend group: gateway.envoyproxy.io path: "/mcp" forwardHeaders: - name: X-Atlassian-Jira-Personal-Token - name: X-Atlassian-Confluence-Personal-Token # Public weather API — no forwarding, no user identity. - name: weather-mcp kind: Backend group: gateway.envoyproxy.io path: "/mcp" securityPolicy: apiKey: secretRef: name: weather-api-key ``` The agent's MCP client is configured once with the user's PAT headers, and every request carries them automatically. The gateway forwards them to the Atlassian backend only; the weather backend keeps using its shared API key and never sees a user token. When a backend expects a different header name than your clients send, rename in transit: ```yaml forwardHeaders: - name: Authorization # extract from the inbound request backendHeader: X-User-Token # forward under this name ``` The rename is also how per-user identity and a service-account credential coexist on one backend: forward the user's token under a dedicated header while the backend's `securityPolicy` injects the service credential into `Authorization`. Note that credentials injected by a backend `securityPolicy` are applied at the Envoy layer and overwrite any forwarded header of the same name — so give the forwarded user identity its own header rather than expecting it to take precedence. Two honest caveats. First, in this mode the gateway is not validating the PATs — the backend is. Keep inbound authentication at the front door (an `MCPRoute` security policy, or OIDC at the gateway) so the gateway itself is never an open proxy. Second, forward identity headers only to backends inside your trust perimeter; if a backend cannot validate the credential, it has no business receiving it. ## Level 2: OIDC + JWT Forwarding — The Same-IdP Pattern PAT passthrough needs no identity provider at all, which is exactly why it is the easiest place to start. Once your platform has corporate SSO wired up, the same `forwardHeaders` mechanism supports a stronger variant: validate the user's corporate JWT at the perimeter, then forward it to internal tools that already trust the same IdP. ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: SecurityPolicy metadata: name: corporate-oidc spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: envoy-ai-gateway oidc: clientID: "${OIDC_CLIENT_ID}" clientSecret: name: oidc-client-secret redirectURL: "https://gateway.company.com/oauth2/callback" provider: issuer: "https://company.okta.com" ``` With this in place, the gateway rejects unauthenticated requests at the door, and a backend opting into `forwardHeaders: [{name: Authorization}]` receives the same corporate JWT the internal tool already accepts through SSO — short-lived, validated at the perimeter, revocable at the IdP, with zero configuration in the downstream tool. This is the natural hardening step for internal tools that natively validate your corporate identity. ## Stateless Passthrough vs. Token Broker Gateway products in this space often take a different approach: the user completes a consent flow or pastes a token once, the gateway stores it server-side, and clients afterwards just connect to a URL. It is fair to ask whether Envoy AI Gateway's model — the client sends the credential on every request — is a step backwards. It isn't; it is a different point in the design space. Sending a credential per request is the normal model of HTTP authentication — every GitHub API call carries a token, every browser request carries cookies, and the MCP specification itself has clients send `Authorization` on each HTTP request. A broker doesn't eliminate that; it moves the sending from client→backend to gateway→backend, and in exchange the gateway becomes a stateful token vault: encrypted storage, rotation, and a high-value breach target — with central revocation as the payoff. Passthrough keeps the gateway stateless. There is nothing to steal at rest, nothing to rotate, and nothing to migrate; the credential rests with the client platform, which for an enterprise agent platform is typically already holding the user's session. Short-lived, IdP-issued JWTs favor passthrough strongly. Long-lived tokens for external SaaS — where storage, refresh, and revocation genuinely need management — favor a broker or token exchange. The two models are complementary, and the same route can serve both. ## What This Looks Like in Practice Picture a platform aggregating a dozen MCP backends behind one route — Jira, Confluence, Slack, GitHub Enterprise, and internal observability tools, collectively exposing well over a hundred tools. Enabling per-user identity for the backends that need it is a YAML addition to the existing `MCPRoute`. No OAuth app registrations, no token storage, no consent flows, no changes to the backends themselves. The results are exactly what the RBAC model promises: Jira comments attribute to the triggering engineer, not a bot. Confluence and Jira queries return only what each engineer's own account can see. Agents triggered by different engineers surface different data, matching each user's actual permissions. And because forwarding is per-backend, the user tokens never touch the public-API backends sharing the same route. ## What's Next Header forwarding covers the case where the user's credential already works downstream. For the cross-IdP case — calling Atlassian Cloud or GitHub.com with per-user identity — RFC 8693 OAuth token exchange support is actively in review. Between the two, the gateway layer is becoming the natural place where agent traffic picks up the right identity for each tool it touches. Learn more: 1. Header forwarding documentation: https://aigateway.envoyproxy.io/docs/capabilities/mcp/#header-forwarding 2. Feature PR: https://github.com/envoyproxy/ai-gateway/pull/2047 3. Design discussion: https://github.com/envoyproxy/ai-gateway/issues/1966 4. OAuth token exchange (in review): https://github.com/envoyproxy/ai-gateway/pull/2092 5. MCP authorization specification: https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization --- # Announcing Envoy AI Gateway 1.0 — A Stable, Production-Ready AI Gateway Source: https://theagentrouter.ai/blog/v1.0-release-announcement Date: 2026-06-23 ![Announcing Envoy AI Gateway 1.0](https://theagentrouter.ai/img/og/blog/v1.0-release-announcement.png) Today we're thrilled to announce **Envoy AI Gateway 1.0** — the first **stable, generally available** release of the open source AI gateway built on **CNCF's [Envoy Gateway](https://gateway.envoyproxy.io/)**. When we [shipped v0.1 in February 2025](https://aigateway.envoyproxy.io/blog/01-release-announcement), we closed that post with three words: **"Onward to 1.0!"** Sixteen months and many releases later, backed by a community of maintainers and adopters across the industry, we're here. 1.0 means you can build on Envoy AI Gateway with confidence: a control-plane API we're committing to keep stable, running on the same battle-tested Envoy foundation that already powers production traffic at the world's largest companies. ## 🔒 What 1.0 Means: Stability You Can Build On The headline of 1.0 isn't a single new feature — it's a **promise**. Our [release policy](https://github.com/envoyproxy/ai-gateway/blob/main/RELEASES.md) has always said that we'd cut the major v1.0.0 release once we had a _first stable control-plane API_. That moment is here. For stable APIs, the commitment is simple and strong: > **We will never break the APIs unless there is a critical security issue**, and we will always provide > a migration path in the release notes if we ever must. Concretely, that means: | Guarantee | What it means for you | | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Stable CRDs** | The resources you author — `AIGatewayRoute`, `AIServiceBackend`, `BackendSecurityPolicy`, `GatewayConfig`, `MCPRoute`, all served at `v1beta1` — won't break under you. | | **Predictable upgrades** | Upgrading the controller won't break a valid, migrated configuration. | | **Documented migrations** | Any future change that requires action will ship with a clear, documented upgrade path. | This is the foundation enterprises have been asking for — the ability to standardize on a single, provider-agnostic AI gateway without betting on a moving target. ## 🛣️ The Road from 0.1 to 1.0 v0.1 launched with a unified API in front of **two** providers and the essentials: upstream authorization and token-based rate limiting. 1.0 is a different animal. Here's how far the project has come: | Capability | v0.1 (Feb 2025) | 1.0 | | -------------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------- | | **AI providers** | 2 (OpenAI, AWS Bedrock) | **16**, with cross-provider request/response translation | | **API surface** | Chat completions | Chat, completions, embeddings, image generation, audio (transcription / translation / speech), and the OpenAI Responses API | | **MCP (Model Context Protocol)** | — | A full **MCP gateway**: server multiplexing, tool routing & filtering, and fine-grained authorization | | **Multimodal** | — | Image, audio, and video inputs across supported providers | | **Observability** | Basic metrics | OpenTelemetry tracing, OpenInference, GenAI token metrics, separate reasoning-token accounting | | **Multi-tenancy & routing** | Token rate limiting | Hostname-based routing, model virtualization, and quota-aware rate limiting | | **Control-plane API** | v1alpha1 (experimental) | **v1beta1 (stable)** | Sixteen providers, integrated through a single OpenAI-compatible interface, including OpenAI, Azure OpenAI, Google Gemini, Google Vertex AI, AWS Bedrock, Anthropic, Mistral, Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok, and the Tetrate Agent Router Service. ## ✨ What's in 1.0 ### One API, every provider Point your application at a single OpenAI-compatible endpoint and let the gateway handle provider-specific translation, authentication, and routing. Switch or mix providers without touching application code — and use **model virtualization** to keep your app code stable while routing changes underneath: ```yaml backendRefs: - name: openai-backend modelNameOverride: "gpt-4o" - name: anthropic-backend modelNameOverride: "claude-opus-4" ``` This is the key to A/B testing, gradual migrations, multi-provider strategies, and safeguarding against vendor lock-in. ### Provider authentication, handled for you `BackendSecurityPolicy` keeps provider credentials out of your application and centralizes upstream auth — API keys, AWS, Azure, and GCP cloud-native identity (including Workload Identity), all managed at the gateway. ### An MCP gateway for the agent era Aggregate multiple **Model Context Protocol** servers behind one endpoint, route and filter the tools clients can see, and enforce **fine-grained, CEL-based authorization** — so `tools/list` only ever returns what a caller is actually allowed to use. ### Enterprise observability, built in Token-aware metrics, OpenTelemetry tracing with OpenInference compatibility (for evaluation tools like Arize Phoenix), and separate accounting for reasoning tokens give you the cost control and visibility that AI workloads demand. ### Standards all the way down Built on the Kubernetes **Gateway API** and the [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/), Envoy AI Gateway is an additive layer on Envoy Gateway — it expands what Envoy can do for GenAI traffic without changing how you already deploy and operate it. ## 🌍 Built by a Community, on CNCF Envoy 1.0 is the work of a genuinely cross-industry community. Maintainers come from **Tetrate, Bloomberg, Tencent, Netflix, and Nutanix**, alongside a growing roster of independent contributors who join our **weekly community meetings**, file issues, and ship code. Just as importantly, the project has been shaped by real-world use. Our thanks to the organizations who used, tested, and shared feedback along the way — including **LY Corporation, Alan by Comma Soft, and the National Research Platform**. [![Star History Chart](https://api.star-history.com/svg?repos=envoyproxy/ai-gateway&type=Date)](https://www.star-history.com/#envoyproxy/ai-gateway&Date) [Visit us on GitHub](https://github.com/envoyproxy/ai-gateway) and star the repo to show your support. ## 🔭 What's Next, Beyond 1.0 A stable API is a starting line, not a finish line. On the roadmap: - **A dedicated `MCPBackend` CRD**, decoupling MCP backend configuration from `MCPRoute`. - **Deeper MCP authorization** across tools, resources, and prompts. - **Fuller quota-aware routing** that automatically steers around rate-limited upstreams. - **More provider translation paths** and **expanded multimodal** support. The roadmap is community-driven — and we'd love your help shaping it. ## 🚀 Get Involved | Action | Resource | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ | | **Try 1.0 today** | [Download the release](https://github.com/envoyproxy/ai-gateway/releases) | Get the latest release and start exploring | | | [Follow the getting started guide](https://aigateway.envoyproxy.io/docs/getting-started/) | Step-by-step setup instructions | | | [Explore the examples](https://github.com/envoyproxy/ai-gateway/tree/main/examples) | Real-world configuration examples | | **Join the community** | [Weekly community meetings](https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w/edit?tab=t.0#heading=h.6nxfjwmrm5g6) | Add it to your calendar | | | [Slack #envoy-ai-gateway](https://envoyproxy.slack.com/channels/envoy-ai-gateway) | Join the conversation on Envoy Slack | | | [GitHub Discussions](https://github.com/envoyproxy/ai-gateway/discussions) | Share experiences and ask questions | | **Contribute** | [Report issues](https://github.com/envoyproxy/ai-gateway/issues/new?template=bug_report.md) | Help us improve by reporting bugs | | | [Request features](https://github.com/envoyproxy/ai-gateway/issues/new?template=feature_request.md) | Tell us what you need next | | | [Submit code](https://github.com/envoyproxy/ai-gateway/blob/main/CONTRIBUTING.md) | Build the next release with us | ## 🙏 Acknowledgments 1.0 belongs to everyone who got us here: the maintainers and contributors who wrote the code and reviews, the early adopters who tested pre-releases and told us what broke, and the broader Gateway API, Envoy, and CNCF communities whose standards we build on. Thank you. The future of AI infrastructure is **open, stable, and community-driven**. We can't wait to see what you build on it. 🚀 **This is just the beginning.** --- _Envoy AI Gateway 1.0 is available now. For detailed release notes, API changes, and upgrade guidance, visit our [release notes page](https://aigateway.envoyproxy.io/release-notes/v1.0)._ --- # Benchmarking Envoy AI Gateway Control Plane Scaling Source: https://theagentrouter.ai/blog/benchmarking-control-plane-scaling Date: 2026-02-23 ![](https://theagentrouter.ai/img/og/blog/benchmarking-control-plane-scaling.png) **How many AIGatewayRoute resources can Envoy AI Gateway handle?** In this post, I'll walk through how I benchmarked the control plane scaling of Envoy AI Gateway, the architecture of the test, and the results from scaling to 2,000 routes. ## The Experiment: What are we testing? This benchmark focuses on **Configuration Scale**. The goal is to generate a massive number of AI Gateway routes, apply them to the cluster, and verify that the Gateway can successfully route traffic for every single one of them. ### High-Level Workflow The benchmark operates in two distinct phases: - **Provisioning Phase:** We programmatically create thousands of Kubernetes Custom Resources (CRDs) to flood the Gateway's configuration. - **Testing Phase:** We send inference requests to verify that the Gateway has actually registered these routes and is forwarding traffic correctly. ## The Setup To perform this test without incurring massive bills from OpenAI or Anthropic, I used a **Mock Cassette Server**. This acts as our "AI Provider," recording and replaying static responses. ### Prerequisites To reproduce this, you need a Kubernetes cluster with: - **Envoy Gateway:** Both the control plane and data plane installed. - **Envoy AI Gateway CRDs:** Specifically `AIServiceBackend` and `AIGatewayRoute`. - **A Mock Backend:** A reachable service (DNS name) that mimics an LLM provider API. ### The Resources (The "Scale Knobs") To stress-test the control plane, our automation script creates three categories of resources: 1. **Gateway Infrastructure:** Standard Kubernetes Gateway API resources (`GatewayClass` and `Gateway`). This is the entry point for our traffic. 2. **Backend Infrastructure:** These represent the upstream targets. We configure `Backend` and `AIServiceBackend` resources pointing to our mock cassette server. We created multiple backends to ensure the load-balancing logic was also active. 3. **Route Infrastructure (The Stress Test):** This is the main lever we pulled. We generated thousands of `AIGatewayRoute` objects. ### Methodology I wrote a Go CLI tool to automate the process: 1. **Build:** It loops through and creates the defined number of `AIGatewayRoute` CRDs. 2. **Validate:** It iterates through every created route and sends an inference request. 3. **Retry:** It retries up until the inference endpoint is ready. ## The Results & The Journey As we pushed the route count higher, we encountered a configuration bottleneck that needed to be addressed before we could reach our target. ### Tuning the gRPC Configuration Pipeline The Envoy Gateway Control Plane communicates with the AI Gateway Extension Server via gRPC. As the number of routes grew, the xDS configuration payload sent between these components exceeded the default gRPC message size of 4MB. To support large-scale configurations, we increased the gRPC message size to 25MB. This required updates in two places to ensure both the sender and receiver were aligned: **1. Envoy Gateway Configuration (`envoy-gateway-values.yaml`)** ```yaml extensionManager: maxMessageSize: 25Mi backendResources: # ... other resource details ``` **2. AI Gateway Controller Configuration (`values.yaml`)** ```yaml controller: maxRecvMsgSize: "26214400" ``` With this tuning in place, the configuration pipeline could handle the full scale of our test. ### Verified: 2,000 AIGatewayRoutes After applying the gRPC configuration tuning, we successfully provisioned and validated **2,000 AIGatewayRoutes**. Every single route was confirmed to be actively routing inference traffic to the correct backend. The system handled this scale cleanly — all routes were created, picked up by the configuration watcher, and serving traffic within the expected readiness window. > **Note on Header Mutation:** > These benchmark runs were performed without using `headerMutation` in `AIServiceBackend`. In Envoy AI Gateway, header mutation contributes additional per-backend configuration to the filter config. With header mutation enabled (especially if verbose or repeated across backends), the configuration payload per route would be larger, which is worth considering when planning for scale. > **Note on Filter Configuration Secret Size:** > The AI Gateway controller aggregates route and backend configuration from `AIGatewayRoute` and `AIServiceBackend` resources into a single Kubernetes Secret that the extproc server consumes at runtime. Since Kubernetes (etcd) enforces a ~1MB size limit on individual objects, operators scaling to a high number of routes should be mindful of the total configuration size. Features like `headerMutation` in `AIServiceBackend` add to this payload, so the effective route ceiling depends on the per-route configuration complexity. ## Performance Deep Dive: Latency & Resource Usage Beyond scaling, I also wanted to verify the performance characteristics of the system at 2,000 routes. ### 1. Route Readiness Latency (The 5-Second Rule) I specifically tested how long it takes for a newly created `AIGatewayRoute` to become "ready" (successfully routing traffic). - **Observation:** Across all 2,000 routes, there was a consistent delay of approximately **5 seconds** from creation to readiness. - **The "Why":** This isn't a performance degradation; it's by design. The extproc server uses a configuration watcher (`filterapi.StartConfigWatcher`) that polls for changes at a default interval of 5 seconds to dynamically apply updates without restarting the pod. - **Verdict:** No degradation in inference performance. Once the route is picked up by the watcher, traffic flows instantly with no added overhead from the scale. ![Consistent 5-second "Time to First Success" across 2,000 routes, confirming the deterministic polling interval.](https://theagentrouter.ai/img/blog/benchmarking-route-readiness-latency.png) ### 2. Resource Utilization (The Cost of Configuration) I monitored the key components during the provisioning phase to understand the resource cost of adding thousands of routes. #### The "Ramp-Up" Pattern Unlike the flat line of a steady state, resource usage during the provisioning phase tells a different story. As we added routes one by one (up to 2,000), we observed a distinct linear increase in both CPU and Memory usage. - **During Creation:** Every new route requires processing, validation, and xDS translation. This cost accumulates linearly — the more routes you have, the more CPU cycles are burned to compute the new configuration snapshot, and more memory is allocated to store the state. - **After Completion:** - **CPU:** Usage drops sharply back to normal levels once the provisioning script finishes. This confirms that the high CPU load is a transient cost of processing the configuration changes. - **Memory:** Usage drops only slightly but remains effectively at the new, higher baseline. This is expected, as the controllers and the proxy must actively hold the 2,000 route objects and their associated xDS configuration in memory to serve traffic. #### Envoy Gateway & AI Gateway Controllers Both controllers exhibited this pattern: a steady, linear climb in resources as the configuration set grew. ![AI Gateway Controller CPU usage rises linearly as routes are added, then drops to baseline.](https://theagentrouter.ai/img/blog/benchmarking-aigw-controller-cpu.png) ![Memory usage follows the linear accumulation trend and stays elevated to store the route state.](https://theagentrouter.ai/img/blog/benchmarking-aigw-controller-memory.png) #### Envoy Proxy Pod The data plane (Envoy Proxy) showed a similar trend. The memory footprint grew as it loaded the expanding configuration, and CPU usage tracked the processing effort required to apply these updates. ![Envoy Proxy CPU usage scales with the rate of configuration updates.](https://theagentrouter.ai/img/blog/benchmarking-envoy-proxy-cpu.png) ![Envoy Proxy memory usage grows to accommodate the route configuration and persists at the higher level.](https://theagentrouter.ai/img/blog/benchmarking-envoy-proxy-memory.png) ## Conclusion Scaling an AI Gateway isn't just about how many tokens per second you can process; it's about how many distinct routing rules you can manage. We successfully tested Envoy AI Gateway with **2,000 AIGatewayRoutes**, and the system performed well across the board — consistent route readiness latency, linear and predictable resource growth, and zero routing failures. The only tuning required was increasing the gRPC message size from the default 4MB to 25MB to accommodate the larger xDS configuration payloads. With that configuration in place, the gateway handled 2,000 routes without any issues. ## Get Involved Envoy AI Gateway is an open-source, community-driven project, and we'd love your help to continue improving it. Whether it's running your own benchmarks, contributing code, or sharing feedback, there are many ways to get involved: - **Join us on Slack** -- [Register for Envoy Slack](https://communityinviter.com/apps/envoyproxy/envoy) and join the **#envoy-ai-gateway** channel to connect with the community. - **Attend the Weekly Community Meeting** -- We meet every week to discuss roadmap, features, and open issues. Check the [meeting notes](https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w/edit?tab=t.0#heading=h.6nxfjwmrm5g6) for the schedule and agenda. - **Contribute on GitHub** -- Raise issues, suggest improvements, and submit PRs on the [GitHub repository](https://github.com/envoyproxy/ai-gateway). - **Share Your Experience** -- Run your own benchmarks, try the gateway in your environment, and share your results with the community in [GitHub Discussions](https://github.com/envoyproxy/ai-gateway/discussions). We're building this together -- come help us push the boundaries of what an AI Gateway can do. --- # The Reality and Performance of MCP Traffic Routing with Envoy AI Gateway Source: https://theagentrouter.ai/blog/mcp-in-envoy-ai-gateway Date: 2025-12-08 ![](https://theagentrouter.ai/img/og/blog/mcp-in-envoy-ai-gateway.png) Envoy AI Gateway (AIGW) provides a production-ready bridge between AI agents and their tools by handling Model Context Protocol (MCP) traffic. As teams adopt MCP, questions about scale, performance, and architecture naturally arise. This post addresses those questions by first clearing up common misunderstandings, then diving into the actual architecture, the design choices we made (and why), and how you can test and evaluate whether it is the right solution for your system. **This post will give you the context you need to:** - Evaluate MCP routing in Envoy AI Gateway with realistic expectations - Understand the design decisions, their impact, and how they impact you - Learn about how you can configure and tune MCP routing in Envoy AI Gateway to meet your needs ## **Part 1: Common Misconceptions** Before discussing how it works, let’s address how people _think_ it works. There are a few frequent misconceptions about how Envoy AI Gateway handles MCP traffic. ### **Misconception 1:** "AI Gateway's MCP implementation is slow." #### Reality The AIGW MCP implementation offers performance comparable to other cloud-native solutions while providing full access to all Envoy traffic-handling features. #### The Nuance The default configuration settings balance performance, functionality, and security. If you want to further reduce latency based on your use case, you can easily adjust the configuration to prioritize raw speed in internal or low-latency environments. ### **Misconception 2:** "Envoy AI Gateway is a separate project that ignores core Envoy." #### Reality Envoy AI Gateway is deeply integrated into the Envoy Ecosystem. It extends the Envoy Gateway control plane and leverages Envoy Proxy data-plane extensions. It is not a fork or a side project running in isolation; it is designed to feed proven implementations back into Envoy core, accelerating the adoption of emerging patterns like MCP that require faster iteration than the core release cycle typically allows. ### **Misconception 3:** “Envoy is not able to handle AI traffic.” #### Reality The Envoy proxy's architecture is ideally suited for managing AI traffic, including use cases like MCP. Both LLM and MCP traffic ultimately rely on HTTP. As a cloud-native, battle-tested HTTP proxy utilized at scale for nearly a decade, Envoy is a proven solution. Furthermore, the [Envoy community is actively working](https://github.com/search?q=repo%3Aenvoyproxy%2Fenvoy+mcp&type=pullrequests&s=created&o=desc) toward making the core Envoy proxy fully MCP-compliant. This future compatibility means Envoy AI Gateway users will be able to leverage the standard Envoy proxy for their data plane needs. #### The Nuance Currently, Envoy AI Gateway leverages Envoy Proxy extensions to fill functionality gaps and enable MCP routing via Envoy Proxy. This is thanks to Envoy Proxy's extensible architecture, which allows us to rapidly respond to and handle the various challenges the new era of AI gives us. --- ## **Part 2: The Actual Architecture** The core challenge of MCP is that it is currently a **stateful protocol**. A session ID must be reused across calls so the server can maintain context. When you place a gateway in the middle, a single client session might need to route to multiple upstream MCP servers (e.g., GitHub, Jira, local files), each maintaining its own independent session. ### The "Stateless" MCP Routing Design Instead of maintaining persistent session mappings in a centralized session store, Envoy AI Gateway uses a **token-encoding architecture**. 1. **Initialization:** When an agent initializes an MCP session, the gateway establishes upstream sessions with the backend servers (e.g., GitHub, Jira). 2. **Secure session encoding:** The gateway builds a compact description of these upstream sessions and wraps it into a safe, self-contained client session ID. This ensures the map is tamper-proof and that the client cannot see the internal topology. 3. **Routing:** The client receives this session ID. On subsequent requests, the client returns it. Any gateway replica can decode it and immediately route traffic to the correct upstream servers. ![](https://theagentrouter.ai/img/blog/mcp-routing.png) **Tune performance to your needs:** This design allows Envoy AI Gateway users to configure further and tune performance to meet their needs. ### Understanding and Tuning Performance The performance overhead users may observe stems from the key‑derivation function (KDF) used during session encryption, not from MCP routing itself. Envoy AI Gateway encrypts the session ID to prevent leaking internal details to MCP clients. The default settings favor "stronger" key derivation over raw speed, leading to two regimes: - **Default encryption settings (e.g., 100k KDF iterations):** Adds on the order of **tens of milliseconds** of overhead per new session. - **Tuned encryption settings (e.g., \~100 iterations):** Overhead drops to around **1–2 ms per session,** comparable to other MCP gateways. The following graph illustrates proxy performance across different Session Encryption values and you can run the benchmarks on your own hardware to explore what would work best for you. **Benchmark Setup:** These benchmarks were run on a MacBook Pro 17,1 (M1) laptop with 8 cores to demonstrate the impact of session encryption on overall performance. The benchmarks measure the time taken to call a simple “echo” tool directly vs. calling it via Envoy AI Gateway with different session encryption configurations. **Analysis of Proxy Performance with Session Encryption:** _Average execution time in milliseconds (lower is better)_ ![](https://theagentrouter.ai/img/blog/mcp-routing-benchmark-chart.png) **Tip:** You can run these benchmarks yourself to explore how configuring encryption settings meets your performance needs. --- ## **Part 3: How We Arrived at This Design (and Why)** When designing this, we faced a binary architectural choice: **Keep state in the gateway** or **Encode state in the client**. ![](https://theagentrouter.ai/img/blog/mcp-routing-options.png) ### The Alternative We Rejected: Centralized State One approach (Option One) would have been to return a UUID to the client and store the mapping (`UUID -> [Upstream_Session_1, Upstream_Session_2]`) in a shared store like Redis. - **The Problem:** - **More components to manage:** This introduces an additional component and dependency. To scale the gateway, you must also scale and manage a highly available session store. If that store fails, all traffic stops. - **High Availability Architecture Complexity:** To make the solution multi-region and highly available, you must ensure that the data store backing the gateway is replicated across regions. ### The Path We Chose: Encoded State We chose the second option—encoding the session information into the client session ID—for three specific reasons: 1. **Simple Horizontal Scaling:** Because the session information travels with the request, you can spin up new gateway replicas instantly. Any replica can handle any request without needing to sync with a database. 2. **Operational Simplicity:** This eliminates the need to provision, secure, and operate an entire component. Encoding session information into the client token is the **current** design and will evolve to meet the needs of AI traffic routing. Any evolution will follow the principles of focusing on real‑world production use cases, maintaining consistent configuration, and aligning with Envoy core. #### Addressing the Trade-offs This design choice creates a deliberate trade-off: we accept a negligible amount of compute overhead in exchange for operational simplicity to address current use cases and traffic routing needs. Instead of paying the cost of managing, scaling, and querying a database for every request, the gateway simply spends a few CPU cycles processing the token. This is an efficient exchange that keeps the architecture clean and stateless. --- ## **Part 4: Evaluation — Is it Right for You?** The best way to decide if this architecture meets your needs is to test it against your specific performance and security requirements. 1. **Configure an MCP route in Envoy AI Gateway:** Point your agent at AIGW and **experiment with encryption tuning.** 2. **Run the benchmark harness** that you can [find here](https://github.com/envoyproxy/ai-gateway/blob/main/tests/data-plane-mcp/bench/bench_test.go) in the Envoy AI Gateway repo. ```shell # Clone the repo and build the latest CLI git clone git@github.com:envoyproxy/ai-gateway.git cd ai-gateway make build.aigw # Run the benchmarks go test -timeout=15m -run='^$' -bench=. -count=10 ./tests/data-plane-mcp/bench/... ``` 3. **Share your experience:** Open an issue or join the community channels. ## **Part 5: The Envoy Ecosystem Alignment** ### Part of the Battle-Tested Envoy Ecosystem Envoy AI Gateway is part of the Envoy ecosystem, which offers key benefits: - **Battle‑tested data plane:** Routing traffic through AIGW is built on Envoy Proxy’s decade of experience as a high‑performance, production‑grade data plane. - **Ongoing core MCP work:** AIGW can quickly adapt to new AI patterns and feed proven implementations back into Envoy core. AIGW leverages the proven Envoy Proxy extension mechanisms to address traffic-handling needs for AI system builders rapidly. - **Shared investments and community:** Improvements to observability, security, and performance in Envoy generally benefit AI workloads. --- ## Summary and Conclusion Envoy AI Gateway (AIGW) addresses the challenges of routing Model Context Protocol (MCP) traffic, specifically the need to handle long-lived, stateful sessions at scale. ### Key Design Decisions | Feature | Design Choice in AIGW | Benefit | | :--------------------------- | :------------------------------------------------------------------------------------------------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Session State Management** | Encode session information into an encrypted client session ID (stateless gateway). | Simple horizontal scaling, failure isolation, and operational simplicity (no shared session store required). | | **Latency** | Conservative, security-heavy default encryption settings introduce latency during encryption and decryption. | High integrity and defense-in-depth, configurable to tune speed vs. security (tens of milliseconds default; tunable to 1-2ms). | | **Ecosystem Alignment** | Integrated with the battle-tested Envoy Proxy data plane, and investing in enhancing Envoy Proxy to support MCP and AI traffic. | Reliability, high performance, and rapid adaptation to new AI/MCP patterns. Allowing adopters of the AIGW control plane to benefit from Envoy Ecosystem advancements. | | **Best Practice** | Encourage agent context, appropriate routing patterns, and tool grouping. | Keeps session tokens compact, enables granular security policy enforcement, and improves LLM performance. | ## Conclusion Envoy AI Gateway is a production-ready solution for managing stateful MCP traffic, prioritizing horizontal scalability and operational simplicity through its session-encoding design. While the default cryptographic settings offer strong integrity and add slight latency, this overhead is explicitly configurable to meet various performance benchmarks. As part of the Envoy ecosystem, AIGW is positioned to evolve rapidly, incorporating features like optional session persistence and advanced policy based on MCP metadata, ensuring it remains the reliable control point for connecting LLM agents to their tools. Teams are encouraged to test performance with the provided benchmark harness and share real-world usage feedback. --- # Announcing Model Context Protocol Support in Envoy AI Gateway Source: https://theagentrouter.ai/blog/mcp-implementation Date: 2025-10-02 ![Hero feature image of title.](https://theagentrouter.ai/img/og/blog/mcp-implementation.png) We’re excited to announce that the next release of Envoy AI Gateway will introduce first-class support for [Model Context Protocol](https://modelcontextprotocol.io/) (MCP), cementing Envoy AI Gateway (EAIGW) as the universal gateway for modern production AI workloads. Envoy AI Gateway started in close collaboration with [Bloomberg](https://www.bloomberg.com/) and [Tetrate](https://tetrate.io/) to meet production-scale AI workload demands, combining real-world expertise and innovation from some of the industry’s largest adopters. Built upon the battle-tested [Envoy Proxy](https://www.envoyproxy.io/) data plane as the AI extension of Envoy Gateway, it is trusted for critical workloads by thousands of enterprises worldwide. EAIGW already provides unified LLM access, cost and quota enforcement, credential management, intelligent routing, resiliency, and robust observability for mission-critical AI traffic. With the addition of MCP, we have brought these features to the communication between Agents and external tools, making EAIGW even more versatile for enterprise-scale AI deployments. For a deeper look at the collaborative story and technical vision, see the [Bloomberg partnership announcement](https://tetrate.io/blog/tetrate-bloomberg-collaborating-on-envoy-ai-gateway), their [official release coverage](https://www.bloomberg.com/company/press/tetrate-and-bloomberg-release-open-source-envoy-ai-gateway-built-on-cncfs-envoy-gateway-project/), and previous [project announcements](https://theagentrouter.ai/blog/01-release-announcement). ## Why MCP Matters for AI Gateways MCP is rapidly becoming the industry’s open standard for enabling AI agents to securely and flexibly connect to external tools and data sources. As the AI ecosystem shifts from monolithic models to agentic architectures, **building robust, policy-driven, and observable pathways between AI and the rest of the enterprise stack has never been more critical**. Integrating MCP directly into Envoy AI Gateway means: - **Seamless interoperability** between AI agents, tools, and context providers, whether they’re third-party cloud LLMs or internal enterprise services. - **Consistent security and governance**: The gateway can now apply fine-grained authentication, authorization, and observability over tool invocations and data access flowing through MCP. - **Accelerated developmen**t: With MCP supported natively, teams can adopt the latest agent-based AI flows **on their existing Envoy infrastructure without custom or glue code.** ## Key Features in the First Implementation The initial implementation aims for a reliable implementation of the latest version of the spec, covering the full spectrum of features, not only focusing on tool calling: | **Feature** | **Description** | | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Streamable HTTP Transport** | Full support for MCP’s streamable HTTP transport, aligning with the [June 2025 MCP spec](https://modelcontextprotocol.io/specification/2025-06-18). Efficient handling of stateful sessions and multi-part JSON-RPC messaging over persistent HTTP connections. | | **OAuth Authorization** | Native enforcement of [OAuth authentication flows](https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization) for AI agents and services bridging via MCP, ensuring secure tool usage at scale. Backwards compatibility with the [previous version of the authorization spec](https://modelcontextprotocol.io/specification/2025-03-26/basic/authorization) to maximize compatibility with existing agents. | | **MCP Server multiplexing, Tool Routing, and Filtering** | Route tool calls and notifications to the right MCP backends, aggregating and filtering available tools based on gateway policy. Dynamically aggregate, merge, and filter messages and streaming notifications from multiple MCP servers, so agents receive a unified, policy-governed interface to all available services. | | **Upstream Authentication** | Built-in upstream authentication primitives to securely connect to external MCP servers, with support for credential injection and validation using existing Envoy Gateway patterns. | | **Full MCP Spec Coverage** | Complete [June 2025 MCP spec](https://modelcontextprotocol.io/specification/2025-06-18) compliance, including support for tool calls, notifications, prompts, resources, and bi-directional server-to-client requests. Robust session and stream management, including reconnection logic (e.g., Last-Event-ID for SSE), ensuring resilience and correctness for long-lived agent conversations. | | **Zero Friction Developer and Production UX** | All features are supported in standalone mode, allowing them to start the Envoy AI Gateway locally on their machine with a single command and start leveraging all MCP features. These configurations can be used as-is in production environments, as there is full compatibility between local standalone mode and Kubernetes. | | **Tested in Real Life** | The implementation has full protocol test coverage and is tested with real-world providers like GitHub and agents like [Goose](https://block.github.io/goose/), validating end-to-end functionality in the existing ecosystem. | ## Under the Hood Adding MCP support meant more than just passing bytes through the stack. Our approach **leverages Envoy’s architecture and existing features**, introducing a lightweight MCP Proxy that handles session management, multiplexes streams, and bridges the gap between the stateful JSON-RPC protocol and the broader Envoy extension mechanisms. Key design decisions included: - **Minimal Architectural Complexity**: The implementation does not add any additional component/complexity to the existing Envoy AI Gateway’s architecture - **Fully leverages Envoy's networking stack**: The MCP proxy harnesses Envoy’s proven networking stack for connection management, load balancing, circuit breaking, rate-limiting, observability, etc. - **Decoupled Iteration Velocity**: The MCP Proxy is implemented as a lightweight Go server to keep pace with the rapidly evolving MCP specification, while still relying on Envoy for networking primitives. **Info:** You can find more details about the architecture and design decisions in the design document of the [MCP contribution pull request](https://github.com/envoyproxy/ai-gateway/pull/1260). ## Getting Started One of the easiest ways to get started with the MCP features in Envoy AI Gateway is by using the standalone mode to run it on your machine with no friction. In standalone mode, you can start the MCP Gateway with your existing MCP servers configuration file used by agents like Claude Code. ### Use your existing MCP servers file In the following example, we’ll start Envoy AI Gateway to proxy the GitHub and Context7 MCP servers. First, define the servers in the mcp-servers.json file: ```json { "mcpServers": { "context7": { "type": "http", "url": "https://mcp.context7.com/mcp" }, "github": { "type": "http", "url": "https://api.githubcopilot.com/mcp/readonly", "headers": { "Authorization": "Bearer ${GITHUB_ACCESS_TOKEN}" } } } } ``` And then start Envoy AI Gateway: **Tip:** Refer to the [CLI installation instructions](https://theagentrouter.ai/docs/cli/aigwinstall) if you haven’t installed the CLI yet. ```shell $ aigw run --mcp-config mcp-servers.json ``` This will start the Envoy AI Gateway locally and start serving the MCP servers at [http://localhost:1975/mcp](http://localhost:1975/mcp). **You can point your preferred agent (Claude, Goose, etc) to this URL as a Streamable HTTP MCP server.** You can also use some features, like Tool Filtering, with the existing server file. To do it, you just need to add the list of tools to expose for each server (by default, all tools are exposed), like in the following example: ```json { "mcpServers": { "context7": { "type": "http", "url": "https://mcp.context7.com/mcp" }, "github": { "type": "http", "url": "https://api.githubcopilot.com/mcp/readonly", "headers": { "Authorization": "Bearer ${GITHUB_ACCESS_TOKEN}" }, "tools": ["issue_read", "list_issues"] } } } ``` ### Using the new MCPRoute API You can also use the new `MCPRoute` API, which will allow a more fine-grained configuration and will work in standalone mode and in Kubernetes as well. The following example defines a complete `MCPRoute` that shows how simple it is to configure: - MCP Server multiplexing. - MCP Authentication using OAuth. - Tool filtering. - MCP Server upstream authentication. ```yaml apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: MCPRoute metadata: name: mcp-route namespace: default spec: parentRefs: - name: aigw-run kind: Gateway group: gateway.networking.k8s.io backendRefs: - name: context7 kind: Backend group: gateway.envoyproxy.io path: "/mcp" - name: github kind: Backend group: gateway.envoyproxy.io path: "/mcp/readonly" # Use the radonly endpoint # Only expose certain tools toolSelector: includeRegex: - .*pull_requests?.* - .*issues?.* # Configure upstream authentication securityPolicy: apiKey: secretRef: name: github-access-token # Configure the gateway to enforce authentication using OAuth. securityPolicy: oauth: issuer: "https://auth-server.example.com" protectedResourceMetadata: # The URL here must match the URL of the Envoy AI Gateway resource: "http://localhost:1975/mcp" scopesSupported: - "profile" - "email" ``` This file can be used in your Kubernetes cluster and in standalone mode. You can quickly try it out with: ```shell $ aigw run mcp-route.yaml ``` And point your local agents again to [http://localhost:1975/mcp](http://localhost:1975/mcp). Once you’re happy with the configuration, you can apply it as-is on your Kubernetes cluster and production environments. ## What’s Next? This is just the beginning. As MCP and agentic architectures advance, we’ll continue to evolve and upstream features—ensuring Envoy AI Gateway remains the **universal,** **most reliable, policy-driven, and interoperable AI gateway available** modern production AI workloads. We are proud to contribute the implementation of the MCP protocol, and look forward to continue enhancing Envoy AI Gateway together with the community and bring more use cases and features to the project. We’d love your feedback as you start exploring MCP in your GenAI or agent infrastructure. Join our community meetings or file an issue to help shape the roadmap\! --- # Enhancing AI Gateway Observability - OpenTelemetry Tracing Arrives in Envoy AI Gateway Source: https://theagentrouter.ai/blog/openinference-for-ai-observability Date: 2025-08-25 ![Hero feature image of title.](https://theagentrouter.ai/img/og/blog/openinference-for-ai-observability.png) Aggregated metrics like latency, error rates, and throughput on their own won't reveal the source of _why_ a system's output was wrong, slow, or expensive. The **v0.3 release** of Envoy AI Gateway brings comprehensive **OpenTelemetry tracing support** with **OpenInference semantic conventions**, extending the existing metrics foundation to provide complete visibility into LLM application behavior. This enables you to improve the quality and safety of your AI-integrated applications by allowing you to understand the full context of a request journey, as your LLM traces will inform application improvements and guardrail needs. ## The Observability Challenges in AI Applications Traditional observability looks at request speed, request volume, and error rates. These work well for simple stateless HTTP services. But they are not enough for AI applications. **AI applications present unique challenges:** - LLM requests have complicated costs based on token usage - Responses can vary and may stream out tokens slowly - Semantic failures occur when the AI fails to understand or produce the correct answer - These issues don't show up as HTTP errors Envoy AI Gateway has addressed some of these challenges. It collects a set of robust metrics through OpenTelemetry. It tracks token use, request times, and provider performance data. The **v0.1.3 update** added GenAI-specific metrics. These include time-to-first-token, available via Prometheus endpoints. However, these metrics alone can't tell you the full story or the root cause of an issue; this is where **OpenTelemetry tracing with OpenInference conventions** comes in. ## Specifications Made for AI: OpenInference Semantic Conventions To enable you to gain traffic insights in the best way, Envoy AI Gateway chooses to stay close to standards. Instead of creating custom trace formats, Envoy AI Gateway uses **[OpenInference.](https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md)** This is a widely accepted standard for AI applications, compatible with OpenTelemetry. Many frameworks, like **[BeeAI](https://docs.beeai.dev/observability/custom-agent-traceability)** and **[HuggingFace SmolAgents,](https://huggingface.co/docs/smolagents/v1.21.1/en/tutorials/inspect_runs#inspecting-runs-with-opentelemetry)** support it. OpenInference sets rules for tracking how large language models work. It covers things like the prompt, model settings, tokens used, and the response. Key moments, such as the time to get the first token during streaming, are recorded as span events. These relate to the earlier discussed metrics. This OpenTelemetry approach uses spans and works well with common tracing systems. These systems usually link traces, not logs. For example, you can set up Envoy AI Gateway to **[work with Jaeger.](https://www.jaegertracing.io)** This setup is similar to how LLM evaluation systems like Arize Phoenix handle OpenInference directly. Redaction controls are available from the start. They help you manage your trace data and balance it with your evaluation needs. **Redaction controls** are available from the start. They help you manage your trace data and balance it with your evaluation needs. ## How it All Fits Together: Envoy AI Gateway OpenTelemetry Tracing Architecture ![Diagram showing how it all fits together](https://theagentrouter.ai/img/blog/Envoy%20AI%20gateway%20+%20Phoenix.drawio.png) Here's an example of a simple trace that includes both application and gateway spans, shown in Arize Phoenix: ![Screenshot of Phoenix UI](https://theagentrouter.ai/img/blog/phoenix.webp) ## Capture and Evaluate your Traffic: LLM Evaluation Tracing data isn't only for in-the-moment troubleshooting; this data is your key to optimizing your AI system. **OpenInference traces provide the structured data foundation necessary for comprehensive LLM evaluation:** - Capture the whole interaction context, including prompts, model parameters, and outputs - Leverage evaluation frameworks to identify patterns over time - Find optimization opportunities for performance, accuracy, and/or cost ### Evaluation Patterns To analyze the trace data, you can leverage an **[LLM-as-a-Judge](https://arize.com/llm-as-a-judge/)** evaluation pattern using your production trace data. Since your traces are in OpenInference format, your easiest option is to leverage a solution that can consume them. Platforms like **Arize Phoenix** consume OpenInference traces directly, enabling easy analysis of inference traffic captured through Envoy AI Gateway. ### Privacy Controls When capturing tracing data, you want to ensure you keep private data private. The gateway includes **configurable privacy controls** for sensitive environments: - Selectively redact content from spans - Limit data captured in multimodal interactions - Apply custom filtering based on organizational requirements ## Telemetry via the Gateway: Zero-Application-Change Integration **Gateway-level tracing requires no code changes in your applications.** As traffic is routed via Envoy AI Gateway, the OpenInference-compliant traces are automatically created for all requests to the OpenAI-compatible API. This happens regardless of whether the calling applications include OpenTelemetry instrumentation. ### Automatic Trace Propagation If your calling applications are already instrumented with OpenTelemetry, the client spans will automatically join the same distributed trace as the gateway spans, providing **end-to-end visibility**. This trace propagation offers a complete view of the AI interactions. ### Seamless Integration As it integrates with your current tools, you and your team can begin using this new feature without needing to learn and adopt new tooling and instrumentation patterns. ## Follow the Development Lifecycle: Deployment Flexibility As you move from local development, through dev, test, staging, and to production, you can capture and trace traffic in the same way. **The tracing capability works consistently across deployment modes:** - **Local development:** Standalone CLI tool - **Production:** Kubernetes Gateway - **All environments:** Full tracing support for OpenAI-compatible requests, including streaming responses and multimodal inputs This deployment consistency simplifies your observability integration throughout your development lifecycle. Teams can establish observability patterns during local development that integrate seamlessly into production environments without architectural changes. ## Looking Ahead: AI Applications Evolve Fast, and Infrastructure and Observability with it The new tracing capability is available with the **v0.3 Envoy AI Gateway release**. For complete configuration details and integration examples, see the **[tracing documentation](https://theagentrouter.ai/docs/capabilities/observability/tracing)** to get started. As AI infrastructure continues to evolve, comprehensive observability becomes essential for managing operational complexity and ensuring application quality. **OpenTelemetry tracing with OpenInference conventions** provides the foundation teams need to build reliable, observable AI systems. ### Get Involved Want to get involved in building Envoy AI Gateway and further improve the observability capabilities? - **Raise an issue** on our GitHub repository - **Join us on Slack** in the `#envoy-ai-gateway` channel - **Join our weekly community meetings** --- # Announcing the Envoy AI Gateway v0.3 Release Source: https://theagentrouter.ai/blog/v0.3-release-announcement Date: 2025-08-22 ![Envoy AI Gateway v0.3.0 Release](https://theagentrouter.ai/img/og/blog/v0.3-release-announcement.png) The **Envoy AI Gateway v0.3** release introduces **intelligent inference routing** through Endpoint Picker (EPP) integration, expands our provider ecosystem with **Google Vertex AI Production Support** as well as **Native Anthropic API**, and delivers **Enterprise-Grade Observability** with OpenInference tracing. ## **The Big Shifts in v0.3** Envoy AI Gateway v0.3 isn't just another feature release; it's a **fundamental shift** toward intelligent, production-ready AI infrastructure. This release addresses three critical challenges that have been holding back AI adoption in enterprise environments: ### 1\. From Static to Intelligent Routing Traditional load balancers treat AI inference endpoints like web servers, but AI workloads are fundamentally different. With **Endpoint Picker integration**, Envoy AI Gateway now makes intelligent routing decisions based on real-time AI-specific metrics like KV-cache usage, queue depth, and LoRA adapter information. **What this means for you:** | Benefit | Description | | ----------------------------------- | -------------------------------------------------------------------- | | **Latency reduction** | Optimal endpoint selection based on real-time AI metrics | | **Automatic resource optimization** | Intelligent resource allocation across your inference infrastructure | | **Zero manual intervention** | Automated endpoint management without operational overhead | ### 2\. Expanded Provider Ecosystem We've moved beyond experimental integrations to deliver **production-grade support** for the AI providers that matter most to enterprises. **Google Vertex AI** is now supported with complete streaming capabilities for Gemini models. **Anthropic on Vertex AI** moves from experimental to production-ready with multi-tool support and configurable API versions. **What this means for you:** | Benefit | Description | | --------------------------------- | ------------------------------------------------------------------------ | | **Unified OpenAI-compatible API** | Single interface across Google, Anthropic, AWS, and more providers | | **Enterprise-grade reliability** | Production-ready stability for mission-critical AI workloads | | **Provider flexibility** | Switch between providers without architectural changes or vendor lock-in | ### 3\. Enterprise Observability for AI AI workloads require specialized observability that traditional monitoring tools can't provide. v0.3 delivers comprehensive AI-specific monitoring across four key areas. **What this means for you:** | Observability Feature | Description | | ------------------------------- | ------------------------------------------------------------------------- | | **OpenInference tracing** | Complete request lifecycle visibility and evaluation system compatibility | | **Configurable metrics labels** | Granular monitoring based on request headers for custom filtering | | **Embeddings metrics support** | Comprehensive token usage tracking for accurate cost attribution | | **Enhanced GenAI metrics** | Improved accuracy with OpenTelemetry semantic conventions | ## **Notable New Features in v0.3** ### Endpoint Picker Provider: The Future of AI Load Balancing A highlight of v0.3 is our integration with the [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/), which allows intelligent endpoint selection that understands AI workloads. ```yaml # AIGatewayRoute with InferencePool apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: AIGatewayRoute metadata: name: intelligent-routing spec: rules: - matches: - headers: - name: x-ai-eg-model value: meta-llama/Llama-3.1-8B-Instruct backendRefs: - group: inference.networking.x-k8s.io kind: InferencePool name: vllm-llama3-pool ``` This isn't just about load balancing; it's about **intelligent infrastructure** that adapts to your AI workloads in real-time. ### Google Vertex AI: Enterprise AI at Scale Google Vertex AI support moves to production-ready status with: - **GCP Vertex AI Authentication** with Service Account Key or Workload Identity Federation. - **Complete Gemini Support** with OpenAI API compatibility for function calls, multimodal, reasoning and streaming. - **Complete Anthropic on Vertex AI Support** with OpenAI API compatibility for function calls, multimodal, extended thinking and streaming. - **Native Anthropic API** via GCP Vertex AI to unlock use case like ClaudeCode. - **Enterprise-grade reliability** for mission-critical deployments. This brings the power of Google's AI platform into your unified AI infrastructure, managed through a single, consistent API. ### Comprehensive AI Observability Traditional observability tools fall short when monitoring AI workloads. v0.3 delivers four significant observability enhancements: | Enhancement | Feature | Benefit | | ------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------- | | **OpenInference Tracing Integration** | Complete LLM request tracing with timing and token information | Deep visibility into AI request lifecycle | | **OpenInference Tracing Integration** | Evaluation system compatibility with tools like Arize Phoenix | Seamless integration with AI evaluation workflows | | **OpenInference Tracing Integration** | Full chat completion request/response data capture | Complete audit trail for debugging and analysis | | **Configurable Metrics Labels** | Custom labeling based on HTTP request headers | Flexible monitoring and alerting setup | | **Configurable Metrics Labels** | Granular monitoring by user ID, API version, or application context | Enhanced filtering and segmentation | | **Configurable Metrics Labels** | Enhanced filtering and alerting capabilities | More targeted monitoring and alerts | | **Embeddings Metrics Support** | Comprehensive token usage tracking for both chat and embeddings APIs | Better cost control and usage insights | | **Embeddings Metrics Support** | Accurate cost attribution across different operation types | Precise cost allocation and budgeting | | **Embeddings Metrics Support** | OpenTelemetry semantic conventions compliance | Standardized observability integration | | **Enhanced GenAI Metrics** | Improved error handling and attribute mapping | More reliable performance monitoring | | **Enhanced GenAI Metrics** | More accurate token latency measurements | Better performance analysis data | | **Enhanced GenAI Metrics** | Better performance analysis data | Improved optimization insights | ### Model Name Virtualization: Abstraction for Flexibility The new `modelNameOverride` field enables powerful model abstraction: ```yaml backendRefs: - name: openai-backend modelNameOverride: "gpt-4" - name: anthropic-backend modelNameOverride: "claude-3" ``` By abstracting away the model name, application developers can use standardized model names, while the gateway handles provider-specific routing. This is, for example, useful when doing A/B testing, gradual migrations, safeguarding against provider lock-in, and multi-provider strategies. ## Unified LLM and non-LLM APIs Enhanced Gateway resource management by allowing both standard HTTPRoute and AIGatewayRoute to be attached to the same Gateway object. This provides a unified routing configuration that supports both AI and non-AI traffic within a single gateway infrastructure, simplifying deployment and management. ## **Community Impact and Momentum** ### Growing Community The v0.3 release represents the collaborative effort of our **rapidly expanding community**: - **Contributors from Tetrate, Bloomberg, Tencent, Google, and Nutanix** - **Independent developers** driving innovation - **Enterprise adopters** providing real-world feedback This diversity of perspectives has shaped v0.3 into a release that serves both **bleeding-edge innovators** and **enterprise production needs**. [![Star History Chart](https://api.star-history.com/svg?repos=envoyproxy/ai-gateway&type=Date)](https://www.star-history.com/#envoyproxy/ai-gateway&Date) [Visit on GitHub](https://github.com/envoyproxy/ai-gateway) and Star the Repo to show your support. ### Standards Leadership Our integration with the **Gateway API Inference Extension** demonstrates our commitment to **open standards** and **vendor-neutral solutions**. By building on proven Gateway API patterns, we're ensuring that Envoy AI Gateway remains interoperable and future-proof. Enabling tracing through **OpenInference Tracing Integration** further cements and showcases our community's commitment to industry standards, collaboration, and ecosystem integration. ## What This Release Enables | Benefit | Impact | | --------------------------------------------------------- | --------------------------------------- | | Simplified model deployment with intelligent routing | Faster development cycles | | Performance optimization through real-time metrics | Better model performance | | Cost control with token-based rate limiting | More predictable operating costs | | Multi-model support in a single infrastructure | Reduced complexity and maintenance | | Unified AI infrastructure supporting diverse workloads | Scalable, future-proof architecture | | Standards-based architecture for long-term sustainability | Vendor-neutral, interoperable solutions | | Vendor flexibility without architectural changes | Reduced lock-in risk | | Enterprise observability for production confidence | Production-ready monitoring | | Reduced operational complexity through automation | Lower operational overhead | | Improved reliability with intelligent failover | Higher system reliability | | Better resource utilization across infrastructure | Optimized infrastructure costs | | Streamlined monitoring with AI-specific telemetry | Simplified troubleshooting | ## Get Involved: Join the AI Infrastructure Revolution The future of AI infrastructure is **open, collaborative, and community-driven**. Here's how you can be part of it: | Action | Resource | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ | | **🚀 Try v0.3 Today** | [Download the release](https://github.com/envoyproxy/ai-gateway/releases/tag/v0.3.0) | Get the latest release and start exploring | | | [Follow our getting started guide](https://aigateway.envoyproxy.io/docs/getting-started/) | Step-by-step setup instructions | | | [Explore the examples](https://github.com/envoyproxy/ai-gateway/tree/main/examples) | Real-world configuration examples | | **💬 Join the Community** | [Weekly Community Meetings](https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w/edit?tab=t.0#heading=h.6nxfjwmrm5g6) | Add to your calendar | | | [Slack Channel #envoy-ai-gateway](https://envoyproxy.slack.com/channels/envoy-ai-gateway) | Join the conversation on Envoy Slack | | | [GitHub Discussions](https://github.com/envoyproxy/ai-gateway/discussions) | Share experiences and ask questions | | **🛠️ Contribute to the Future** | [Report Issues](https://github.com/envoyproxy/ai-gateway/issues/new?template=bug_report.md) | Help us improve by reporting bugs | | | [Request Features](https://github.com/envoyproxy/ai-gateway/issues/new?template=feature_request.md) | Tell us what you need for future releases | | | [Submit Code](https://github.com/envoyproxy/ai-gateway/blob/main/CONTRIBUTING.md) | Contribute to the next release | ## Acknowledgments: The Power of Open Source v0.3 wouldn't exist without our incredible community. Special recognition goes to: - **Enterprise contributors** who provided production feedback and requirements - **Open source maintainers** from the Gateway API and CNCF communities - **Individual developers** who contributed code, documentation, and ideas - **Early adopters** who tested pre-releases and reported issues ## Get Started Today Ready to experience the future of AI infrastructure? **Get started with Envoy AI Gateway v0.3** and see how intelligent inference routing, expanded provider support, and enterprise observability can transform your AI deployments. The future of AI infrastructure is **open, intelligent, and community-driven**. Join us in building it. 🚀 [**Get Started with v0.3 →**](https://aigateway.envoyproxy.io/docs/getting-started/) --- _Envoy AI Gateway v0.3 is available now. For detailed release notes, API changes, and upgrade guidance, visit our [release notes page](https://aigateway.envoyproxy.io/release-notes/v0.3)._ --- # Envoy AI Gateway Introduces Endpoint Picker Support Source: https://theagentrouter.ai/blog/endpoint-picker-for-inference-routing Date: 2025-07-30 ![Reference Architecture for Envoy AI Gateway](https://theagentrouter.ai/img/og/blog/endpoint-picker-for-inference-routing.png) ## Introduction **Envoy AI Gateway now supports Endpoint Picker Provider (EPP) integration as per the** [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/). This feature enables you to leverage intelligent, dynamic routing for AI inference workloads through intelligent endpoint selection based on real-time metrics, including KV-cache usage, queued requests, and LoRA adapter information. When running AI inference at scale, this means your system can automatically select the optimal inference endpoint for each request, thereby optimizing resource utilization. ![An overview of Endpoint Picker together with Envoy AI Gateway](https://theagentrouter.ai/img/blog/epp-blog-overview.png) ## The Problem: Traditional Load Balancing Falls Short for AI Workloads As LLM inference workloads have a new set of characteristics compared to traditional API servers, informing the load balancer of which upstream target is the most optimal to serve the request requires a new set of information for making those decisions. Some of the unique characteristics of AI inference workloads: - **Variable processing times** based on model complexity and input size - **Different resource requirements** for different models and configurations - **Real-time performance metrics** that change constantly (KV-cache usage, queue depth, etc.) - **Specialized hardware requirements** (GPUs, TPUs, different model variants) Without taking these into account when routing, you’ll end up with: - Overloaded endpoints while others sit idle - Increased latency due to poor endpoint selection - Resource waste from suboptimal load distribution - Manual intervention is required for optimal performance ## The Solution: Endpoint Picker Provider Integration Envoy AI Gateway's new EPP support addresses these challenges by providing users with the option to integrate an Endpoint Picker. The endpoint picker will take these pieces of information into account and inform the gateway of the best upstream target to route the request to. ![An overview of Endpoint Picker together with Envoy AI Gateway](https://theagentrouter.ai/img/blog/epp-blog-step-by-step.png) ### 1\. Intelligent Endpoint Selection The Endpoint Picker (EPP) automatically routes requests to the most suitable backend by analyzing real-time metrics from your inference endpoints, such as: - Current KV-cache usage - Number of queued inference requests - LoRA adapter information - Endpoint health and performance metrics ### 2\. Dynamic Load Balancing Unlike static load balancers, the EPP continuously receives endpoint status, which affects routing decisions in real-time. This dynamic decision ensures optimal resource utilization across your entire inference infrastructure. ### 3\. Automatic Failover When an endpoint becomes unavailable or performance degrades, the EPP automatically routes traffic to healthy alternatives, ensuring high availability for your AI services. ### 4\. Extensible Architecture The EPP architecture supports custom endpoint picker providers, allowing you to implement domain-specific routing logic tailored to your unique requirements. ## How It Works: Two Integration Approaches Envoy AI Gateway supports EPP integration through two powerful approaches: ### HTTPRoute \+ InferencePool For simple inference workloads, you can use the standard Gateway API HTTPRoute with InferencePool. This approach offers basic intelligent routing, automatic load balancing, and straightforward configuration, making it ideal for simple use cases. ![Showing the relationship between HTTPRoute and InferencePool Kubernetes Resources](https://theagentrouter.ai/img/blog/epp-blog-http-route.png) ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: inference-pool-with-httproute namespace: default spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: inference-pool-with-httproute namespace: default rules: - backendRefs: - group: inference.networking.k8s.io kind: InferencePool name: vllm-llama3-8b-instruct namespace: default weight: 1 matches: - path: type: PathPrefix value: / timeouts: request: 60s ``` This approach provides: - Basic intelligent routing - Automatic load balancing - Simple configuration ### AIGatewayRoute \+ InferencePool For advanced AI-specific features, use Envoy AI Gateway's custom AIGatewayRoute. ![Showing the relationship between AIGatewayRoute and InferencePool Kubernetes Resources](https://theagentrouter.ai/img/blog/epp-blog-aigwroute.png) ```yaml apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: AIGatewayRoute metadata: name: inference-pool-with-aigwroute namespace: default spec: targetRefs: - name: inference-pool-with-aigwroute kind: Gateway group: gateway.networking.k8s.io rules: - matches: - headers: - type: Exact name: x-ai-eg-model value: meta-llama/Llama-3.1-8B-Instruct backendRefs: - group: inference.networking.k8s.io kind: InferencePool name: vllm-llama3-8b-instruct - matches: - headers: - type: Exact name: x-ai-eg-model value: mistral:latest backendRefs: - group: inference.networking.k8s.io kind: InferencePool name: mistral - matches: - headers: - type: Exact name: x-ai-eg-model value: some-cool-self-hosted-model backendRefs: - name: envoy-ai-gateway-basic-testupstream ``` This enhanced approach adds: - **Multi-model routing** based on the request body modelName - **Token-based rate limiting** for cost control in self-hosted Models - **Advanced LLM observability** across LLM Models ## Real-World Benefits ### For AI/ML Engineers - **Lower latency**: Requests automatically routed to optimal endpoints - **Better throughput**: Intelligent distribution prevents bottlenecks - **Cost optimization**: Efficient resource usage reduces infrastructure costs - **Enhanced observability**: Real-time metrics and performance insights ### For Platform Teams - **Standards compliance**: Built on the Gateway API Inference Extension - **Vendor flexibility**: Support for multiple EPP implementations - **Future-proof architecture**: Extensible design for evolving requirements - **Kubernetes-native**: Seamless integration with existing infrastructure ### For DevOps Teams - **Reduced operational overhead**: No more manual endpoint management - **Improved reliability**: Automatic failover and health monitoring - **Better resource utilization**: Dynamic load balancing maximizes efficiency - **Simplified scaling**: Add new endpoints without configuration changes ## Getting Started Setting up EPP support in Envoy AI Gateway is straightforward: ### 1\. Install InferencePool CRDs ```shell # Install Gateway API Inference Extension kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.5.1/manifests.yaml ``` ### 2\. Deploy Your Inference Backends ```shell # Deploy sample vLLM backend kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/v0.5.1/config/manifests/vllm/sim-deployment.yaml ``` ### 3\. Configure InferenceObjective and InferencePool ```yaml apiVersion: inference.networking.k8s.io/v1 kind: InferenceObjective metadata: name: base-model spec: modelName: meta-llama/Llama-3.1-8B-Instruct criticality: Critical poolRef: name: vllm-llama3-8b-instruct --- apiVersion: inference.networking.k8s.io/v1 kind: InferencePool metadata: name: vllm-llama3-8b-instruct spec: targetPortNumber: 8000 selector: app: vllm-llama3-8b-instruct extensionRef: name: vllm-llama3-8b-instruct-epp ``` ### 4\. Create Your Route Choose between HTTPRoute or AIGatewayRoute based on your needs, and you're ready to go\! ## What's Next The introduction of EPP support represents a significant milestone in Envoy AI Gateway's evolution. This feature enables intelligent routing for AI workloads, making it easier than ever to deploy and manage production AI inference services. ### Upcoming Enhancements - **Upstream Conformance Test**: Integrate with **Gateway API Inference Extension** Conformance Tests. - **Fallback Support**: This feature provides support for endpoint picker fallback when the Endpoint Picker is unavailable, based on the Envoy HostOverride LbPolicy. It ensures continuous service availability even in the event of Endpoint Picker failure. - **Internal Managed EPP**: This upcoming feature will provide support for an Envoy AI Gateway-managed Endpoint Picker, simplifying the management of your AI inference endpoints. - **Enhance End-to-End (E2E) Testing**: Add more E2E tests for InferencePool support. ### Community Contributions We're excited to see how the community leverages this capability. Whether you're building custom EPP implementations, contributing to the Gateway API Inference Extension, or sharing your deployment experiences, we'd love to hear from you. ## Conclusion Endpoint Picker Provider support enables Envoy AI Gateway to serve not only as an Egress AI Gateway but also as an Ingress AI Gateway for AI inference workloads. By automatically selecting optimal endpoints based on real-time metrics, this feature improves performance and maximizes resource utilization for hosted inference systems. Whether you're running a small AI service or a large-scale inference platform, EPP support provides the intelligent routing capabilities you need to deliver reliable, high-performance AI services to your users. Ready to get started with Envoy AI Gateway? Check out our [documentation](https://aigateway.envoyproxy.io/docs/capabilities/inference/) for guides and examples, join our [community discussions](https://github.com/envoyproxy/ai-gateway/discussions) to share your experiences, raise issues, request features, and learn from others. --- _Envoy AI Gateway continues to evolve as the premier solution for routing and managing AI workloads. Stay tuned for more exciting features and capabilities as we work to simplify, enhance, and improve the reliability and efficiency of AI deployment._ ## Resources - [Inference Optimization Documentation](https://theagentrouter.ai/docs/capabilities/inference/) - [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) - [AIGatewayRoute \+ InferencePool Guide](https://theagentrouter.ai/docs/capabilities/inference/aigatewayroute-inferencepool/) - [HTTPRoute \+ InferencePool Guide](https://theagentrouter.ai/docs/capabilities/inference/httproute-inferencepool/) - [GitHub Repository](https://github.com/envoyproxy/ai-gateway) --- # A Reference Architecture for Adopters of Envoy AI Gateway Source: https://theagentrouter.ai/blog/envoy-ai-gateway-reference-architecture Date: 2025-07-15 ![Reference Architecture for Envoy AI Gateway](https://theagentrouter.ai/img/og/blog/envoy-ai-gateway-reference-architecture.png) # Building a Scalable, Flexible, Cloud-Native GenAI Platform with Open Source Solutions AI workloads are complex, and unmanaged complexity kills velocity. Your architecture is the key to mastering it. As generative AI (GenAI) becomes foundational to modern software products, developers face a chaotic new reality, juggling different APIs from various providers while also attempting to deploy self-hosted open-source models. This leads to credential sprawl, inconsistent security policies, runaway costs, and an infrastructure that is difficult to scale and govern. **Your architecture doesn’t have to be this complex.** Platform engineering teams need a secure, scalable way to serve both internal and external LLMs to their users. That’s where Envoy AI Gateway comes in. This reference architecture outlines how to build a flexible GenAI platform using the open source solutions Envoy AI Gateway, KServe, and complementary tools. Whether you're self-hosting models or integrating with model-serving services on cloud providers, such as OpenAI and Anthropic, this architecture enables a unified, governable interface for LLM traffic. --- ## Core Architecture: Two-Tier Gateway Design The foundation of this platform is a Two-Tier Gateway Architecture: ### Tier One Gateway Deployed in a centralized Gateway Cluster. It serves as the main API traffic entry point for client applications. **Its Job:** To route traffic to external LLM providers (e.g., OpenAI, Anthropic, Bedrock, Vertex) or to the appropriate internal gateway to access an internal model-serving cluster. **Why It Matters:** This gateway provides a unified API for all application developers. They don't need to know or care if a model is hosted by a third party or in-house. It centralizes coarse-grained policies, such as authentication, top-level routing, and global rate limiting, simplifying the developer experience and providing a single control point for platform-wide governance. ### Tier Two Gateway Deployed as part of a self-hosted model serving cluster **Its Job:** To handle internal traffic routing, load balancing, and policy enforcement specific to self-hosted models running on platforms like KServe. **Why It Matters:** This empowers platform teams. They can manage model versions, conduct releases, and apply specific security rules for self-hosted models without needing to make changes to the primary, customer-facing gateway. This separation ensures that internal operational changes don't impact external clients. ### Design Benefits This design cleanly separates external access from internal implementation, giving teams the autonomy they need to move fast without breaking things, and provides: - Centralized credential management - Unified API access - Cost tracking and traffic governance ![](https://theagentrouter.ai/img/blog/aigw-ref.drawio.png) ### Routing and Traffic Management Envoy AI Gateway provides a consistent client interface and abstracts the complexity of accessing diverse GenAI backends. It supports: | Feature | Problem | Solution | | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Upstream Authentication with Credential Injection** | Application developers must manage, store, and rotate API keys for multiple external LLM providers. This is a security risk and an operational burden that slows down development. | The gateway injects the correct credentials per provider. Developers can make requests to the gateway using a single, internal authentication token, and the gateway handles attaching the appropriate third-party API key. **This decouples applications from external secrets, dramatically improving security and simplifying code.** | | **Token-Based Rate Limiting and Cost Optimization** | A single buggy script or a new feature can lead to unexpected spikes in usage, resulting in huge bills from LLM providers or overwhelming your self-hosted infrastructure. | The gateway acts as an intelligent financial and operational circuit breaker. You can enforce policies based on token usage, requests, or estimated cost. **This prevents abuse and ensures that you stay within budget, providing critical protection for the business.** | | **Observability Hooks for Usage Patterns and Latency** | When you use multiple model providers, it's nearly impossible to get a clear picture of usage. Which teams are spending the most? Which models have the highest latency? How many tokens is our new feature consuming? | By routing all traffic through a single point, the gateway provides a unified source of truth and built-in support for metrics, logs, and traces tailored for GenAI workloads. **This enables you to accurately track costs, identify performance bottlenecks, and understand usage patterns across your entire GenAI stack.** | **Tip:** Clients simply hit a single endpoint and let the Gateway handle routing to the appropriate backend—self-hosted or third-party. --- ## Self-Hosted Model Serving with KServe While many organizations start with external providers, self-hosting models offer advantages in terms of cost, privacy, and customization. If you are self-hosting models, KServe is a powerful addition. For many data scientists and ML engineers, turning a trained model into a production-ready API is a major hurdle that requires deep expertise in Kubernetes, networking, and infrastructure. KServe bridges this gap and eliminates the complexity. KServe is a model serving platform that automates the heavy lifting, allowing an engineer to simply provide a configuration file for their model while KServe builds the scalable, resilient API endpoint. **A few highlights of what KServe provides:** - Autoscaling (including token-based autoscaling for LLMs and scale-to-zero for GPUs) - Multi-node inference (via vLLM) - Support for OpenAI-compatible APIs and advanced runtime integrations (vLLM) - Built-in support for model and prompt caching Check out the [KServe Documentation](https://kserve.github.io/website/latest/) to learn more about all the capabilities. Use tools like Backstage to simplify model deployment and namespace management for teams. **Info:** Self-hosting is optional. Many adopters will initially rely on external providers and add internal hosting as requirements evolve. ![](https://theagentrouter.ai/img/blog/kserve-ref.png) --- ## Observability, Control, and Optimization for Production Readiness In a production GenAI platform environment, observability, control, and optimization become must-haves. Envoy AI Gateway and KServe offer integrations that cater to these needs. ### Observability Achieving visibility into your system enables the identification of bottlenecks, effective cost management, and ensures model reliability. - **Unified Metrics and Tracing:** - Envoy AI Gateway integrates seamlessly with **OpenTelemetry**, adhering to the GenAI Semantic Conventions. This delivers a unified view of requests, latency, token usage, and errors across all external and internal models. - Leverage specialized telemetry, like **OpenLLMetry**, for granular insights into LLM-specific metrics, ensuring you capture essential details like prompt and completion lengths, token throughput, and model-specific performance. - **Centralized Logging:** - Centralize logging across providers through Envoy AI Gateway, facilitating easier debugging, auditing, and compliance. ### Control Enforcing policies and controls ensures your platform remains secure, stable, and cost-effective: - **Policy Enforcement and Guardrails:** - Set usage-based guardrails directly in Envoy AI Gateway to prevent cost overruns, enforce compliance, and safeguard against prompt misuse or model hallucination. - Implement safety checks and output validation rules that enable your team to control quality and compliance centrally, rather than embedding these checks individually within applications. ### Optimization Maximizing the efficiency and responsiveness of your models can significantly enhance user experience and operational efficiency: - **Caching Strategies:** - KServe's **model caching** further optimizes inference, significantly lowering response times and improving model utilization. - **Disaggregated Serving:** - Optimize hardware resources by leveraging KServe’s support for **disaggregated inference**, which enables the separate management of compute-intensive inference stages and memory-intensive operations, thereby maximizing hardware efficiency and reducing costs. **Info:** Be sure to review the documentation for each project to learn more about preparing your setup for production. --- ## Pluggable and Flexible This is not a rigid, all-or-nothing platform. It’s a set of foundational components you can adopt and extend to fit your unique environment. This architecture works whether you're fully committed to Kubernetes or using a hybrid cloud. **You can:** - Start with externally hosted LLMs or self-hosted inference - Use Envoy AI Gateway with any compatible provider via a unified API - Add your own authorization logic and/or custom functionality via Envoy’s extension filters - Deploy your Gateways in different clusters and cloud providers, giving you flexibility of choice in hosting environments This is not a rigid platform. It’s a foundational component you can build on and extend. --- ## Summary The Envoy AI Gateway reference architecture gives platform teams a guide to: - Centralize access to GenAI models with a unified API - Enforce consistent traffic and security policies across model providers - Support both internal and external LLM usage without refactoring client applications - Scale safely and cost-effectively without reinventing core infrastructure Envoy AI Gateway sits at the heart of this design, an intelligent, extensible control point for all your GenAI traffic. --- # Announcing the first Envoy AI Gateway Release – A Community Milestone! Source: https://theagentrouter.ai/blog/01-release-announcement Date: 2025-02-25 ![Announcing the first Envoy AI Gateway Release](https://theagentrouter.ai/img/og/blog/01-release-announcement.png) Today, we're excited to announce the **0.1 release** of the **Envoy AI Gateway,** the first AI gateway built on **CNCF's [Envoy Gateway](https://gateway.envoyproxy.io/)** and backed by a thriving, growing community. The journey to the **Envoy AI Gateway** started with a simple but powerful vision: make it easier for enterprises to integrate and scale AI in their applications. ## Where We Are Now The **Envoy AI Gateway** is now available on GitHub and ready for developers to deploy and explore. It enables enterprises to integrate AI services through a unified API while managing authorization, cost control, and scalability with built-in features: - ✅ **Unified API** for seamless integration with multiple LLM providers (starting with **AWS Bedrock** and **OpenAI**). - ✅ **Upstream Authorization** to simplify authentication across AI providers. - ✅ **Usage Rate Limiting** based on word tokens to control costs and ensure operational efficiency. With this release, we're making AI adoption in cloud-native environments more straightforward and accessible to organizations of all sizes. ## The Power of Community This milestone wouldn't be possible without the incredible contributions and participation from across the industry. **A shout out to our community members from** Tetrate, Bloomberg, WSO2, RedHat, Google, and our independent contributors who have joined discussions, provided feedback, and helped shape the roadmap. 🐱 Even **the cat Mellow** has attended community meetings, proving that AI isn't just for humans\! The momentum behind **Envoy AI Gateway** speaks to the need for an open, **collaborative** approach to GenAI infrastructure. Our contributor's and early adopters' excitement and expertise drive **innovation forward.** ## Where We're Headed We're just getting started\! Here's a sneak peek at what's next: - **Google Gemini 2.0 integration** out-of-the-box. - **Provider and Model Fallback Logic** to ensure continued service availability. - **Prompt Templating** for consistent AI interactions. - **Semantic Caching** to optimize response efficiency and reduce costs. The **roadmap is community-driven**, and we'd love for **more contributors** to help shape the future of AI infrastructure\! ## Join the Movement Want to be part of the journey? Here's how you can get involved: - **Download & Try Envoy AI Gateway** → [GitHub Repo](https://github.com/envoyproxy/ai-gateway/releases/tag/v0.1.5) - **Join the Conversation** → [Attend our community meetings](https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w/edit?tab=t.0#heading=h.6nxfjwmrm5g6) (Mellow might show up too\!) - **Join us on Slack** → [Register for Envoy Slack](https://communityinviter.com/apps/envoyproxy/envoy?email=test) and join \#envoy-ai-gateway - **Contribute** → Raise issues, suggest improvements, and submit PRs on GitHub - **Meet Us In Person** → [Register for the Envoy AI Gateway Workshop in London on March 31st](https://www.eventbrite.com/e/hands-on-workshop-deploy-configure-and-useenvoy-ai-gateway-tickets-1255461000649?aff=oddtdtcreator) The future of GenAI infrastructure **is open and collaborative,** and we're excited to work with you to build it\! 🚀 **Onward to 1.0\!** --- # End User Keynote at KubeCon 2024 Source: https://theagentrouter.ai/blog/kubecon-end-user-keynote-2024 Date: 2024-11-14 At KubeCon North America 2024, Alexa Griffith had the opportunity to present the End User Keynote on **Centralizing & Simplifying Enterprise AI Workflows with Envoy AI Gateway**. ## About the presentation As Generative AI reshapes the industry, the demands on AI platforms have rapidly evolved. Organizations now require centralized infrastructure to manage and optimize access to self-trained, open source, and commercial AI models at scale. In this talk, we introduce the Envoy AI Gateway, a collaborative open source effort led by engineers from [Bloomberg](https://www.bloomberg.com/company/what-we-do/engineering-cto/) and [Tetrate](https://tetrate.io). Learn how the Envoy AI Gateway, which is built atop Envoy Gateway and Envoy Proxy, provides a unified, scalable solution for model access, usage limiting, and upstream authorization. --- # Introducing Envoy AI Gateway Source: https://theagentrouter.ai/blog/introducing-envoy-ai-gateway Date: 2024-10-18 **The industry is embracing Generative AI functionality, and we need to evolve how we handle traffic on an industry-wide scale. Keeping AI traffic handling features exclusive to enterprise licenses is counterproductive to the industry’s needs. This approach limits incentives to a single commercial entity and its customers. Even single-company open-source initiatives do not promote open multi-company collaboration.** A shared challenge like this presents an opportunity for open collaboration to build the necessary features. We believe bringing together different use cases and requirements through open collaboration will lead to better solutions and accelerate innovation. The industry will benefit from diverse expertise and experiences by openly collaborating on software across companies and industries. That is why Tetrate and Bloomberg have started an open collaboration to bring critical features for this new era of Gen AI integration. Collaborating openly in the Envoy community, bringing AI traffic handling features to Envoy, via Envoy Gateway and Envoy Proxy. ## Why we need AI traffic handling features What makes traffic to LLM models different from traditional API traffic? On the surface it appears similar. Traffic comes from a client app that is making an API request, and this request has to get to the provider that hosts the LLM model. However, it is different. Managing LLM traffic from multiple apps, to multiple LLM providers, introduces new and different challenges where traditional API Gateway features fall short. For example, traditional rate-limiting based on number of requests doesn’t work for controlling usage of LLM providers as they’re computationally complex services. To measure usage LLM providers tokenize the words in the request message and response message, and count the number of tokens used. This count gives a good approximation of the computational complexity and cost of serving the request. Beyond controlling usage of LLMs there are many more challenges relating to ease of integration and high-availability architectures. It’s no longer enough to just optimize for quality of service alone, adopters must consider costs of usage in real time. As adopters of Gen AI look for Gateway solutions to handle these challenges for their system, they often find the necessary features locked behind enterprise licenses. ## Three key MVP features Now, let’s look at how handling AI traffic poses new challenges for Gateways. There are several features we discussed together with our collaborators at Bloomberg, and together we decided on three key features for the MVP: - **Usage Limiting** – to control LLM usage based on word tokens - **Unified API** – to simplify client integration with multiple LLM providers - **Upstream Authorization** – to configure Authorization to multiple upstream LLM providers What other features are you looking for? Get in touch with us to share your use case and define the future of Envoy AI Gateway. We are really excited about these features being part of Envoy. They will benefit those integrating with LLM providers and, ultimately, also Gateway users for general API request traffic. When it comes to AI Gateway features, we have chosen to collaborate and build within the CNCF Envoy project because we believe multi-company, open-source projects benefit the entire industry by enabling innovation without creating single vendor risk.