Skip to content

Gateway and routing

Overview

The gateway-provider Pulumi stack is responsible for installing the networking foundation that all model-serving workloads depend on. It performs three distinct jobs:

  1. Install CRDs — Gateway API CRDs and Gateway API Inference Extension (GIE) CRDs must be present in the cluster before any Gateway or InferencePool resource can be created.
  2. Install Istio — Deploys istio-base and istiod via Helm, enabling Istio as the Gateway API implementation.
  3. Create shared Gateway — When infraStackRef (cloud) or gatewayHostname (on-prem) is configured, creates a shared-gateway resource in gateway-system backed by a cloud-managed or Istio/MetalLB load balancer. app_shaide and other stacks attach HTTPRoute resources to this gateway.

Why a shared Gateway instead of a LoadBalancer Service per workload?

Without a shared gateway, each workload (e.g. app_shaide) would need its own LoadBalancer Service. On GCP this means a separate L4 Network Load Balancer per service — each with its own external IP, its own GCP LB resource, and its own cost.

The shared gateway consolidates all external traffic behind a single entry point:

LoadBalancer per Service Shared Gateway
GCP LB resources One per service One for all services
External IPs One per service One static IP
TLS certificate One per service One GKE-managed cert
Cost Scales with number of services Fixed

On-prem the same principle applies — one MetalLB IP and one Istio proxy instead of one per service. The goal on on-prem is to mirror the cloud infrastructure as closely as possible, which is why shared-gateway is deployed on RKE2 as well.

CRD Installation

CRDs are installed via kustomize.NewDirectory — the kustomize rendering engine is embedded in the Pulumi Kubernetes provider. Each CRD is tracked as an individual Pulumi resource, enabling drift detection and eliminating the kubectl binary dependency.

SSA field-manager conflicts (GCP)

GKE clusters pre-install Gateway API CRDs via kube-addon-manager, which holds ownership of fields such as metadata.annotations.gateway.networking.k8s.io/bundle-version and spec.versions. Pulumi's Server-Side Apply would conflict with this ownership.

A patchForceCRDs transform is applied to both kustomize directories. It injects the pulumi.com/patchForce: "true" annotation on every CRD resource, equivalent to kubectl apply --force-conflicts. Pulumi takes ownership of the conflicting fields without an error. On RKE2, where no prior owner exists, the annotation has no effect.

kustomize.NewDirectory
    └── transform: patchForceCRDs
            └── sets metadata.annotations["pulumi.com/patchForce"] = "true"
                on every kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition

Traffic Flows

GCP

Client
    │
    ▼
GKE L7 LB (203.0.113.10)
    │
    ▼
shared-gateway  [gateway-system]        ← gateway-provider
    │
    └── HTTPRoute                       ← app_shaide
            └──▶ shaide-server (ClusterIP)  [app-shaide ns]
                     │
                     └──▶ infra-<model>-inference-gateway-istio (in-cluster FQDN)
                              │
                              ▼
                      per-model Gateway (istio, ClusterIP)   ← app_serving (llm-d-infra)
                              │
                              └── HTTPRoute                  ← app_serving
                                      └──▶ InferencePool
                                               └──▶ model pods

On-prem (RKE2)

Client
    │
    ▼
MetalLB L2 (DNS: shaide.internal.lan)
    │
    ▼
shared-gateway-istio Service (LoadBalancer)  [gateway-system]
    │
    ▼
Istio Envoy proxy
    │
    ▼
shared-gateway  [gateway-system]             ← gateway-provider
    │
    └── HTTPRoute                            ← app_shaide
            └──▶ shaide-server (ClusterIP)  [app-shaide ns]
                     │
                     └──▶ infra-<model>-inference-gateway-istio (in-cluster FQDN)
                              │
                              ▼
                      per-model Gateway (istio, ClusterIP)   ← app_serving (llm-d-infra)
                              │
                              └── HTTPRoute                  ← app_serving
                                      └──▶ InferencePool
                                               └──▶ model pods

Azure — AGC

Client
    │
    ▼
AGC frontend (*.alb.azure.com)
    │
    ▼
shared-gateway  [gateway-system]             ← gateway-provider
    │
    └── HTTPRoute                            ← app_shaide
            └──▶ shaide-server (ClusterIP)  [app-shaide ns]
                     │
                     └──▶ infra-<model>-inference-gateway-istio (in-cluster FQDN)
                              │
                              ▼
                      per-model Gateway (istio, ClusterIP)   ← app_serving (llm-d-infra)
                              │
                              └── HTTPRoute                  ← app_serving
                                      └──▶ InferencePool
                                               └──▶ model pods
  • Azure-managed L7 load balancer — Microsoft runs the infrastructure outside the cluster
  • AKS provides the azure-alb-external GatewayClass built-in
  • AGC assigns its own stable hostname (*.alb.azure.com); DNS uses CNAME, not A record
  • No static IP needed — the hostname is stable
  • Requires Microsoft.ServiceNetworking/trafficControllers
  • The ApplicationLoadBalancer CR + delegated ALB subnet (snet-alb-<stack>) connect AGC to the cluster
  • The subnet is asked for only when the azure-alb-external class is picked; any other class needs no ALB and no subnet. With AGC the installer requires the subnet resource ID and validates its shape. On an update it pre-fills the association already on the cluster; otherwise it suggests the nodes' VNet with the subnet name left open. An empty association makes the ALB controller delete the AGC and its frontend, taking the platform offline, so it is never accepted
  • Zero-ops: Microsoft manages scaling, patching, and availability

Azure — Istio

Supported, and used where AGC is not available: in regions without Application Gateway for Containers (polandcentral at the time of writing, e.g. aks-axem-dev-polandcentral), or on clusters without the ALB controller. Istio is installed by this stack anyway, for the per-model inference gateways, so the only extra cost is the shared-gateway-istio Envoy pod and an Azure Standard Load Balancer in front of it.

Client
    │
    ▼
Azure Standard Load Balancer (public IP, L4)
    │
    ▼
shared-gateway-istio (Envoy, gateway-system)    ← gateway-provider (Gateway, class istio)
    │
    └── HTTPRoute (app-shaide)  ──▶ shaide-server (ClusterIP)
  • Pick istio at the installer's Gateway class prompt. On a cluster whose shared-gateway already uses istio, it is pre-selected; on a cluster with no accepted class other than istio, it is used without asking.
  • No ApplicationLoadBalancer, ALB subnet or albName is created or asked for.
  • The stack annotates the generated LoadBalancer Service so the Azure Load Balancer probes Istio's status port (see Health probes).
  • A pre-allocated public IP can be bound with gatewayStaticIP. Without it, Azure assigns the Service an address, which changes if the Service is recreated.

Downstream stacks (app_shaide, app_serving) are unaffected — the Gateway and HTTPRoute resources are identical regardless of which implementation backs them.

Same Gateway API spec, different infrastructure behind it.

GKE's gateway controller provisions a Google Cloud L7 load balancer externally — no in-cluster Service is created. The IP is a GCP resource, not a Kubernetes Service.

Istio's gateway controller creates a physical Deployment + LoadBalancer Service in-cluster. MetalLB handles L2 ARP to advertise the IP on the LAN.

# GCP — no in-cluster Service:
$ kubectl get svc -n gateway-system
No resources found in gateway-system namespace.

$ kubectl get gateway -n gateway-system
NAME             CLASS                              ADDRESS          PROGRAMMED
shared-gateway   gke-l7-regional-external-managed   203.0.113.10   True

# On-prem — Istio creates a backing Service:
$ kubectl --kubeconfig ~/.kube/rke2-cluster.yaml get svc -n gateway-system
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)
shared-gateway-istio   LoadBalancer   10.43.100.50   10.0.10.50   15021:31238/TCP,80:30418/TCP

$ kubectl --kubeconfig ~/.kube/rke2-cluster.yaml get gateway -n gateway-system
NAME             CLASS   ADDRESS        PROGRAMMED
shared-gateway   istio   10.0.10.50   True

Azure AGC vs Azure Istio — Operational Differences

How traffic reaches the cluster

AGC operates entirely outside the AKS cluster. Microsoft provisions an Azure-managed L7 load balancer (the Application Gateway for Containers resource) in response to the ApplicationLoadBalancer CR. The ALB Controller pod runs inside the cluster and watches for Gateway and HTTPRoute resources, then programs the external AGC resource over the Azure control plane. No Envoy pod is created; traffic never touches an in-cluster proxy before reaching the backend pods.

Istio runs the data plane inside the cluster. The istiod control plane and an Envoy (shared-gateway-istio) pod are both deployed into gateway-system. An Azure Standard Load Balancer (L4) forwards external traffic to the Envoy pod's NodePort, and Envoy applies the Gateway / HTTPRoute routing rules before passing requests to the backend.

IP address and DNS

AGC does not expose a routable public IP. Instead it assigns a stable Azure-internal hostname of the form <hash>.fz52.alb.azure.com. DNS must use a CNAME record pointing to that hostname. The hostname is stable across Gateway updates, cluster upgrades, and node replacements.

Istio exposes a public IP through the shared-gateway-istio LoadBalancer Service. To prevent the IP from changing when the Service or cluster is recreated, a static PublicIPAddress resource is pre-allocated in the AKS node resource group (mc-aks-<stack>) by the 03_cluster stack and bound to the Gateway via spec.addresses (type: IPAddress). DNS uses an A record pointing to that static IP.

Failure domain

With AGC, a misconfigured Gateway or HTTPRoute affects only the Azure-side programming. The in-cluster workloads continue running; only external routing is disrupted.

With Istio, the Envoy pod is in the request path. If shared-gateway-istio crashes or is evicted, all external traffic stops. The Deployment has a single replica by default — consider scaling it for production workloads.

Health probes

AGC manages its own health checks against the backend pods and handles probe configuration automatically.

Istio relies on the Azure Load Balancer to health-check the Envoy pod. Azure defaults to an HTTP probe on port 80 with path /, which Envoy answers with 404 (no matching route). This is worked around by propagating the annotation service.beta.kubernetes.io/port_80_health-probe_port: "15021" via spec.infrastructure.annotations on the Gateway resource — Azure then probes Istio's dedicated status endpoint (15021/healthz/ready) instead.

Resource consumption

AGC Istio
In-cluster pods ALB Controller only istiod + Envoy (shared-gateway-istio)
Azure resources AGC frontend, ALB subnet Standard LB, Public IP
Cluster CPU/memory Minimal ~200m CPU, ~256Mi per istiod + Envoy
Required subnet Delegated /24 (snet-alb-<stack>) None

Gateway implementation by target

Target GatewayClass Backed by
GKE gke-l7-regional-external-managed Google Cloud L7 load balancer
AKS azure-alb-external Application Gateway for Containers
AKS (no AGC in the region) istio Istio Envoy behind an Azure Standard Load Balancer
On-prem istio Istio Envoy behind MetalLB

On AKS, prefer AGC where the region offers it. It offloads L7 routing to Azure, keeps in-cluster resource usage down, and needs no health-probe workaround. Use Istio where AGC is unavailable.