Issues setting up AWS ALB + Envoy Gateway

[romelBen]

I'm looking in setup AWS ALB + Envoy Gateway where the ALB can terminate and Envoy Gateway can route the traffic. We have decided in migrating our services of ingress-nginx to Envoy Gateway but it's proving difficult at least in this front. I'm still unclear on how to do this and on my AWS ALB I am receiving "Unhealthy: healthcheck failing". This is my setup:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Values.environment }}-eg-external-alb
  namespace: {{ .Release.Namespace }}
  annotations:
    alb.ingress.kubernetes.io/scheme: "internet-facing"
    alb.ingress.kubernetes.io/load-balancer-name: "{{ .Values.environment }}-eg-alb-external"
    alb.ingress.kubernetes.io/target-type: "instance"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/certificate-arn: "{{ .Values.certArn }}"
    alb.ingress.kubernetes.io/group.name: "{{ .Values.environment }}-eg-alb-external-group"
    alb.ingress.kubernetes.io/ssl-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06"
    alb.ingress.kubernetes.io/healthcheck-path: "/healthz"
    alb.ingress.kubernetes.io/healthcheck-protocol: "HTTP"
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "30"
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "5"
    alb.ingress.kubernetes.io/success-codes: "200-499"
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: eg-external-alb
                port:
                  name: https
---
apiVersion: v1
kind: Service
metadata:
  name: eg-external-alb
  namespace: {{ .Release.Namespace }}
  labels:
    app.kubernetes.io/name: envoy-gateway
    app.kubernetes.io/component: proxy
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: gateway-helm
    app.kubernetes.io/instance: {{ .Release.Name }}
  ports:
    - name: http
      port: 80
      targetPort: tohttps
      protocol: TCP
    - name: https
      port: 443
      targetPort: http
      protocol: TCP
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: corp-eg-proxy
  namespace: {{ .Release.Namespace }}
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: NodePort
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: corp-eg-gw-class
  namespace: {{ .Release.Namespace }}
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    name: corp-eg-proxy
    namespace: {{ .Release.Namespace }}
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: corp-eg-gw
  namespace: {{ .Release.Namespace }}
spec:
  gatewayClassName: corp-eg-gw-class
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: HTTPRouteFilter
metadata:
  name: httpbin-hc-dr
spec:
  directResponse:
    statusCode: 200
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin-hc
  namespace: {{ .Release.Namespace }}
spec:
  parentRefs:
    - name: corp-eg-gw
      namespace: {{ .Release.Namespace }}
  rules:
  - matches:
    - path:
        type: Exact
        value: /healthz
    filters:
    - type: ExtensionRef
      extensionRef:
       name: httpbin-hc-dr
       group: gateway.envoyproxy.io
       kind: HTTPRouteFilter

I'm positive the Service that I'm using is wrong and the HTTPRouteFilter with the HTTPRoute is unnecessary since I'm trying to receive a 200 request. There's not much documentation online on doing this so I would appreciate any help.

[UPDATE]
Looks like I resolved the issue with the following:

---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg-gw-class-external
  namespace: {{ .Release.Namespace }}
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    name: eg-proxy-external
    namespace: {{ .Release.Namespace }}
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-gw-external
  namespace: {{ .Release.Namespace }}
spec:
  gatewayClassName: eg-gw-class-external
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: envoy-gateway
            namespace: {{ .Release.Namespace }}
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg-proxy-external
  namespace: {{ .Release.Namespace }}
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: eg-alb-external
  namespace: {{ .Release.Namespace }}
  labels:
    app.kubernetes.io/name: envoy-gateway
    app.kubernetes.io/component: proxy
spec:
  type: NodePort
  selector:
    gateway.envoyproxy.io/owning-gateway-name: eg-gw-external
    gateway.envoyproxy.io/owning-gateway-namespace: {{ .Release.Namespace }}
  ports:
    - name: http
      port: 80
      targetPort: 10080
      protocol: TCP
    - name: https
      port: 443
      targetPort: 10443
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Values.environment }}-eg-alb-external
  namespace: {{ .Release.Namespace }}
  annotations:
    alb.ingress.kubernetes.io/scheme: "internet-facing"
    alb.ingress.kubernetes.io/load-balancer-name: "{{ .Values.environment }}-eg-alb-external"
    alb.ingress.kubernetes.io/target-type: "instance"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/certificate-arn: "{{ .Values.certArn }}"
    alb.ingress.kubernetes.io/group.name: "{{ .Values.environment }}-eg-alb-external-group"
    alb.ingress.kubernetes.io/ssl-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06"
    alb.ingress.kubernetes.io/healthcheck-port: "traffic-port"
    alb.ingress.kubernetes.io/healthcheck-path: "/healthz/ready"
    alb.ingress.kubernetes.io/healthcheck-protocol: "HTTP"
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "30"
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "5"
    alb.ingress.kubernetes.io/success-codes: "200-399"
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: eg-alb-external
                port:
                  number: 80

The only issue now the health check. The Targets in my Target Group of AWS ALB are giving: Unhealthy: Health checks failed with these codes: [404].

What healthcheck should we use to make this green?

[romelBen]

I went ahead and added 200 healthcheck via the following:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin-hc
  namespace: {{ .Release.Namespace }}
spec:
  parentRefs:
    - name: eg-gw-external
      namespace: {{ .Release.Namespace }}
  rules:
  - matches:
    - path:
        type: Exact
        value: /healthz
    filters:
    - type: ExtensionRef
      extensionRef:
       name: httpbin-hc-dr
       group: gateway.envoyproxy.io
       kind: HTTPRouteFilter
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: HTTPRouteFilter
metadata:
  name: httpbin-hc-dr
spec:
  directResponse:
    statusCode: 200

I'm wondering if this is the solution?

[sachinshaji]

@llegolas
By default envoy gateway provision NLB and we are in search of a solution to use ALB + Envoy gateway without ingress. Could you please provide the full manifest if possible; this will be really helpful for our team. Thanks in advance

[llegolas]

Here you go

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy-alb
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: ClusterIP            # no NLB/ALB here — ALB Gateway fronts it
        name: envoy-clients-svc    # fixed name (else autogenerated with a hash) so the bridge route can target it
      envoyDeployment:
        replicas: 2
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy-gateway-clients-test
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: envoy-proxy-alb
    namespace: envoy-gateway-system
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-gateway-clients-test
  namespace: envoy-gateway-system
  annotations:
    external-dns.alpha.kubernetes.io/target: internal-k8s-envoygat-albfront-XXXXXXX.eu-central-1.elb.amazonaws.com
spec:
  gatewayClassName: envoy-gateway-clients-test
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.k8s.aws/v1beta1
kind: LoadBalancerConfiguration
metadata:
  name: alb-front
  namespace: envoy-gateway-system
spec:
  scheme: internal                  # change to internet-facing if public
  ipAddressType: ipv4
  listenerConfigurations:
    - protocolPort: HTTPS:443
      defaultCertificate: arn:aws:acm:eu-central-1:XXXXX:certificate/YYYYYYYY
      sslPolicy: ELBSecurityPolicy-TLS13-1-2-2021-06
  # wafV2:
  #   webACL: <WAF_WEB_ACL_NAME>     # NOTE: name, not ARN (regional scope)
  # shieldConfiguration:
  #   enabled: true                # uncomment if you have Shield Advanced
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: aws-alb
spec:
  controllerName: gateway.k8s.aws/alb
---
# Bridge route
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: alb-front-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: aws-alb
  infrastructure:
    parametersRef:
      group: gateway.k8s.aws
      kind: LoadBalancerConfiguration
      name: alb-front
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: alb-to-envoy
  namespace: envoy-gateway-system
spec:
  parentRefs:
    - name: alb-front-gateway
      sectionName: https
  rules:
    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: envoy-clients-svc
          port: 80
---
apiVersion: gateway.k8s.aws/v1beta1
kind: TargetGroupConfiguration
metadata:
  name: envoy-tg-config
  namespace: envoy-gateway-system
spec:
  targetReference:
    name: envoy-clients-svc
    kind: Service
  defaultConfiguration:
    targetType: ip
    healthCheckConfig:
      healthCheckProtocol: HTTP
      healthCheckPort: "19003"
      healthCheckPath: /ready
      healthyThresholdCount: 2
      unhealthyThresholdCount: 2
      healthCheckInterval: 10
      healthCheckTimeout: 5

[sachinshaji]

Thanks a lot @llegolas. I will try the same and let you know.

[sachinshaji]

@llegolas Target group is not getting attached to ALB and I had to do this attachment manually. But yes the ALB is created successfully and able to route to Envoy Backend. Thanks a lot for your support.

[llegolas]

yeah, forgot to mention we did that in terrafrom...