gitea.kosmos.org/doc/ingress.md

HTTP(S) load balancing with Ingress

Resources

Features of GKE Ingress from the Google Cloud docs: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress

It does hostname-aware HTTP(S) load balancing, and is billed like a regular Load Balancer (https://cloud.google.com/compute/pricing#lb). The advantages are that we can use one set of firewall rules (ports 80 and 443) for multiple services, and easy Let's Encrypt certificates for services with no built-in support for it

This 3 part article was a good resource:

https://medium.com/google-cloud/global-kubernetes-in-3-steps-on-gcp-8a3585ec8547 https://medium.com/google-cloud/global-ingress-in-practice-on-google-container-engine-part-1-discussion-ccc1e5b27bd0 https://medium.com/google-cloud/global-ingress-in-practice-on-google-container-engine-part-2-demo-cf587765702

I couldn't find information about setting ingress.kubernetes.io/rewrite-target to / anywhere else, without it only / worked on an host, all other URLs would go to the default backend and return a 404.

cert-manager, for automated (among others) Let's Encrypt certificates: https://docs.cert-manager.io/en/release-0.8/

Create a global IP

Ephemeral IPs are only regional, and you lose them if you have to recreate the Ingress

gcloud compute addresses create ingress-ip --global

Create the ingress

A ClusterIP will not work, because it is allocating random ports. Explicitly create a NodePort to expose your service. On GKE, health checks are configured automatically

cat <<EOF > test-server-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: test-server-nodeport
spec:
  ports:
    - name: http
      port: 80
      targetPort: 3000
  type: NodePort
  selector:
    name: test-server
EOF
kubectl apply -f test-server-nodeport.yaml

Create the ingress resource

cat <<EOF > ingress-main.yaml
# A GCE Ingress that uses cert-manager to manage Let's Encrypt certificates
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-main
  annotations:
    # Required, otherwise only the / path works
    # https://medium.com/google-cloud/global-ingress-in-practice-on-google-container-engine-part-1-discussion-ccc1e5b27bd0
    ingress.kubernetes.io/rewrite-target: /
    certmanager.k8s.io/cluster-issuer: "letsencrypt-production"
    certmanager.k8s.io/acme-challenge-type: http01
    # Created using the following command
    #    gcloud compute addresses create ingress-ip --global
    kubernetes.io/ingress.global-static-ip-name: "ingress-ip"
spec:
  tls:
  - hosts:
    - test.kosmos.org
    secretName: test-kosmos-org-cert
    - test2.kosmos.org
    secretName: test2-kosmos-org-cert
  rules:
  - host: test.kosmos.org
    http:
      paths:
      - backend:
          serviceName: test-server-nodeport
          servicePort: 80
  - host: test2.kosmos.org
    http:
      paths:
      - backend:
          serviceName: test-server-nodeport
          servicePort: 80
EOF
kubectl apply -f ingress-main.yaml

cert-manager

Create the cert-manager resources

cert-manager provides a Let's Encrypt certificate issuer, and lets you mount it in an Ingress resource, making it possible to use HTTP ACME challenges

Get the reserved IP you created in the first step:

$ gcloud compute addresses list --global
NAME        ADDRESS/RANGE   TYPE      PURPOSE  NETWORK  REGION  SUBNET  STATUS
ingress-ip  35.244.164.133  EXTERNAL                                    IN_USE

Set the DNS record for the domain you want a Let's Encrypt cert for to this IP.

Now it's time to create the cert-manager

https://docs.cert-manager.io/en/release-0.8/getting-started/install/kubernetes.html

kubectl create namespace cert-manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.8.1/cert-manager.yaml --validate=false

I had to run the apply command twice for it to create all the resources. On the first run I got these errors. Running it a second time successfully created all the resources

unable to recognize "cert-manager.yaml": no matches for kind "Issuer" in version "certmanager.k8s.io/v1alpha1"
unable to recognize "cert-manager.yaml": no matches for kind "Certificate" in version "certmanager.k8s.io/v1alpha1"
unable to recognize "cert-manager.yaml": no matches for kind "Issuer" in version "certmanager.k8s.io/v1alpha1"
unable to recognize "cert-manager.yaml": no matches for kind "Certificate" in version "certmanager.k8s.io/v1alpha1"

We name the ingress explicitely so it only runs on one. Having only one IP to set on the DNS records makes the HTTP validation easier. Using the class would attach the validation endpoint to all Ingresses of that class (https://docs.cert-manager.io/en/latest/reference/api-docs/index.html#acmechallengesolverhttp01ingress-v1alpha1)

cat <<EOF > letsencrypt-staging.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: ops@kosmos.org
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: letsencrypt-staging-account-key
    solvers:
    - http01:
        ingress:
          name: ingress-main
EOF

cat <<EOF > letsencrypt-production.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: ops@kosmos.org
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: letsencrypt-production-account-key
    solvers:
    - http01:
        ingress:
          name: ingress-main

Add another service

To add another service behind the Ingress, you set the DNS entry for its domain to the Ingress IP, deploy your service, create a NodePort to expose it, and finally add its host to the Ingress config (both tls and rules, see example above)