Cilium Gateway API with K0s and hubble web UI obserevability

Edwardon

Install k0s on 1 cluster and 2 worker nodes

Prepare `k0sctl` on your local machine, I use Mac for example. Make sure your cluster and worker nodes are configured with Key login only.

Im using 3 VPS with have 1 Ipv4 associated with it.

k0s version

 k get nodes
NAME             STATUS   ROLES    AGE     VERSION
node1   Ready    <none>   4h26m   v1.31.3+k0s
node2   Ready    <none>   4h26m   v1.31.3+k0s

Cilium version

cilium-cli: v0.16.23 compiled with go1.23.4 on darwin/arm64
cilium image (default): v1.16.5
cilium image (stable): v1.16.5
cilium image (running): 1.16.5

Here is the example

export node1_IP=142.171.1.1
export node2_IP=148.135.1.1
export node3_IP=142.171.2.1
# The following command assumes the user has deployed 3 VMs
# with the default user "k0s" using the default ssh-key (without passphrase)

# Export the yaml
k0sctl init --k0s -n "k0s-cone-cluster" -u "root" -i "/Users/ssh_cert/id_ed25519_ssh_use" -C "1" "${node1_IP}" "${node2_IP}" "${node3_IP}" > k0s-cluster-config.yaml

# Install k0s
env -u SSH_AUTH_SOCK k0sctl apply --config k0s-cluster-config.yaml -d

# Export the kubeconfig
env -u SSH_AUTH_SOCK k0sctl kubeconfig --config k0s-cluster-config.yaml > ~/.kube/k0s-cluster-config.yaml

Install Gateway API Component

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.1/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.1/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.1/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.1/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.1/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml


kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.1/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

Install Cilium

cilium install \
    --set kubeProxyReplacement=true \
    --set gatewayAPI.enabled=true

# Check cilium status
export KUBECONFIG=~/.kube/k0s-cluster-config.yaml
cilium status --wait

If you want assigned multiple IP address to Cilium LoadBalancer, use this configuration.

apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "blue-pool"
spec:
  blocks:
  - start: "148.135.1.1"
  - start: "142.171.1.1"

Enable Cilium Gateway Class

---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: cilium
spec:
  controllerName: io.cilium/gateway-controller

Helm install cert manager

helm repo add jetstack https://charts.jetstack.io                                                                       
helm repo update

helm install cert-manager jetstack/cert-manager \                                                                           
    --namespace cert-manager \
    --create-namespace \
    --set crds.enabled=true

Setup your application

----
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: default
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - dns01:
        cloudflare:
          email: [email protected]
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token
      selector:
        dnsZones:
        - "xxx.com"
----
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: xxx-fyi-cert
  namespace: default
spec:
  secretName: xxx-fyi-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: xxx.fyi
  dnsNames:
    - xxx.fyi
  duration: 2160h # 90 days
  renewBefore: 360h # 15 days
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
----
apiVersion: v1
kind: ConfigMap
metadata:
  name: certmanager-config
data:
  domain: "xxx.fyi"
  acme_email: "[email protected]"
----
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflare-config
data:
  zone-id: "0ac69b3b"
  account-id: "d2988a"
----
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: default
type: Opaque
stringData:
  api-token: "VIipd"

Setup application deployment and config TLS termination with Gateway API

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xxx-fyi
spec:
  selector:
    matchLabels:
      app: xxx-fyi
  replicas: 2
  template:
    metadata:
      labels:
        app: xxx-fyi
    spec:
      containers:
      - name: xxx-fyi
        image: xxxheng/xxx-fyi:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 4000
----
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: xxx-gateway
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  labels:
    color: red

spec:
  gatewayClassName: cilium
  # https://github.com/cilium/cilium/issues/34099
  infrastructure:
    annotations:
      "lbipam.cilium.io/sharing-key": "1234"
      "lbipam.cilium.io/ips": "148.135.1.1,142.171.1.1"
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    hostname: "xxx.fyi"
    tls:
      certificateRefs:
      - kind: Secret
        name: xxx-fyi-tls
        namespace: default
    allowedRoutes:
      namespaces:
        from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: xxx-route
  namespace: default
spec:
  parentRefs:
  - name: xxx-gateway
    namespace: default
  hostnames:
  - "xxx.fyi"
  rules:
  - matches:
    - path:
        type: Exact
        value: /
    - path:
        type: PathPrefix
        value: /api/v1
    backendRefs:
    - name: xxx-fyi
      port: 4000
      kind: Service
status:
  parents: []

---
apiVersion: v1
kind: Service
metadata:
  name: xxx-fyi
  namespace: default
spec:
  selector:
    app: xxx-fyi
  ports:
    - name: http
      port: 4000
      targetPort: 4000
  type: ClusterIP

Enable Cilium Hubble UI

cilium hubble ui

On your browser you will see the traffic flow

Photo by Rod Long on Unsplash