WIP: Set up ingress with Let's Encrypt certificates using cert-manager

This is using haproxy-ingress to support forwarding SSH on port 22

Since we're using cert-manager with ingress to get Let's Encrypt certs,
we're not using the Let's Encrypt functionality that's part of Gitea. To
run this we need to change the config file, have Gitea run on port 3000
as HTTP and disable all the Let's Encrypt config keys. Currently the
gitea-ingress.yaml uses the letsencrypt-staging ClusterIssuer

This has been tested on a local Kubernetes cluster using Docker for Mac

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "vendor/ark"]
+	path = vendor/ark
+	url = git@github.com:heptio/ark.git

README.md

@@ -1,9 +1,27 @@
 # gitea.kosmos.org
 
 This repository contains configuration files and other assets, that are used to
-deploy and operate this Gitea instance. Feel free to [open
+deploy and operate this Gitea instance.
-issues](https://gitea.kosmos.org/kosmos/gitea.kosmos.org/issues) for questions,
-suggestions, bugs, to-do items, and whatever else you want to discuss or
-resolve.
 
-See `doc/` folder for some technical info.
+To create a new image containing the customizations:
+
+Edit `packer/custom.json` to increment the tag, then run this script (needs
+[Packer](https://www.packer.io/) in your path)
+
+```
+./script/build_customizations_image
+```
+
+Then edit `kubernetes/gitea-server.yaml` to use the new tag
+(`image: eu.gcr.io/fluted-magpie-218106/gitea_custom:$VERSION`) and apply the
+change:
+
+```
+cd kubernetes
+kubectl apply -f gitea-server.yaml
+```
+
+Feel free to [open issues] for questions, suggestions, bugs, to-do items, and
+whatever else you want to discuss or resolve.
+
+[open issues]: https://gitea.kosmos.org/kosmos/gitea.kosmos.org/issues

custom/options/label/Kosmos

@@ -1,5 +1,4 @@
 #db231d bug ; Something is not working
-#ead746 docs ; Documentation
 #76db1d enhancement ; Improving existing functionality
 #1d76db feature ; New functionality
 #db1d76 idea ; Something to consider

doc/backup-and-restore.md

@@ -1,28 +1,36 @@
 # Backups
 
-We're using [Velero][1] (formerly Ark) for backing up Kubernetes config and GKE
+We're using [Ark][1] for backing up Kubernetes config and GKE resources. It is
-resources. It is available as a compiled binary for your platform [on GitHub][2]
+available as a Git submodule in the `vendor/` folder (incl. the `ark`
+executable).
 
-The Velero service is running on the Sidamo cluster and was set up using the
+In order to initialize and update submodules in your local repo, run once:
-[official docs' GCP instructions][3]. There's a daily backup
+
+    git submodule update --init
+
+Then, to fetch/update the modules, run:
+
+    git submodule update
+
+The Ark service is running on the Sidamo cluster and was set up using the
+[official docs' GCP instructions and config files][4]. There's a daily backup
 schedule in effect for Gitea (using the label `app=gitea`).
 
-Please refer to Velero's [ Getting Started ][4] doc for all backup and restore
+Please refer to Ark's [ Getting Started ][5] doc for all backup and restore
 commands.
 
 ## Backup location
 
 Cluster configuration (including all live resources) is backed up to [a Google
-Cloud Storage container][5].
+Cloud Storage container][3].
 
 ## Persistent volumes
 
-Persistent volumes are just GCE disks. Thus, with the current config, Velero
+Persistent volumes are just GCE disks. Thus, with the current config, Ark
-creates volume snapshots as native [GCE disk snapshots][6].
+creates volume snapshots as native [GCE disk snapshots][2].
 
-[1]: https://velero.io/docs/v1.0.0
+[1]: https://heptio.github.io/ark/v0.10.0
-[2]: https://github.com/heptio/velero/releases/tag/v1.0.0
+[2]: https://console.cloud.google.com/compute/snapshots?organizationId=772167872692&project=fluted-magpie-218106&tab=snapshots&snapshotssize=50
-[3]: https://velero.io/docs/v1.0.0/gcp-config/
+[3]: https://console.cloud.google.com/storage/browser/sidamo-backups?project=fluted-magpie-218106&organizationId=772167872692
-[4]: https://velero.io/docs/v1.0.0/about/
+[4]: https://heptio.github.io/ark/v0.10.0/gcp-config
-[5]: https://console.cloud.google.com/storage/browser/sidamo-backups-new?project=fluted-magpie-218106&organizationId=772167872692
+[5]: https://heptio.github.io/ark/v0.10.0/get-started
-[6]: https://console.cloud.google.com/compute/snapshots?organizationId=772167872692&project=fluted-magpie-218106&tab=snapshots&snapshotssize=50

doc/customizations-image.md

@@ -1,20 +0,0 @@
-## Customizations image
-
-### Build
-
-To create a new Docker image containing our Gitea customizations (label sets,
-styles, page content, etc.):
-
-Edit `packer/custom.json` to increment the tag, then run this script (needs
-[Packer](https://www.packer.io/) in your path)
-
-    ./script/build_customizations_image
-
-### Deploy
-
-Edit `kubernetes/gitea-server.yaml` to use the new tag
-(`image: eu.gcr.io/fluted-magpie-218106/gitea_custom:$VERSION`) and apply the
-change:
-
-    cd kubernetes
-    kubectl apply -f gitea-server.yaml

kubernetes/gitea-ingress.yaml

@@ -0,0 +1,276 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: ingress-controller
+  namespace: default
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRole
+metadata:
+  name: ingress-controller
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - endpoints
+      - nodes
+      - pods
+      - secrets
+    verbs:
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+      - nodes
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - services
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses/status
+    verbs:
+      - update
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: Role
+metadata:
+  name: ingress-controller
+  namespace: default
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - pods
+      - secrets
+      - namespaces
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    verbs:
+      - create
+  - apiGroups:
+      - ""
+    resources:
+      - endpoints
+    verbs:
+      - get
+      - create
+      - update
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: ingress-controller
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: ingress-controller
+subjects:
+  - kind: ServiceAccount
+    name: ingress-controller
+    namespace: default
+  - apiGroup: rbac.authorization.k8s.io
+    kind: User
+    name: ingress-controller
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: RoleBinding
+metadata:
+  name: ingress-controller
+  namespace: default
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ingress-controller
+subjects:
+  - kind: ServiceAccount
+    name: ingress-controller
+    namespace: default
+  - apiGroup: rbac.authorization.k8s.io
+    kind: User
+    name: ingress-controller
+---
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  labels:
+    run: ingress-default-backend
+  name: ingress-default-backend
+  namespace: default
+spec:
+  selector:
+    matchLabels:
+      run: ingress-default-backend
+  template:
+    metadata:
+      labels:
+        run: ingress-default-backend
+    spec:
+      containers:
+      - name: ingress-default-backend
+        image: gcr.io/google_containers/defaultbackend:1.0
+        ports:
+        - containerPort: 8080
+        resources:
+          limits:
+            cpu: 10m
+            memory: 20Mi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ingress-default-backend
+  namespace: default
+spec:
+  ports:
+  - port: 8080
+  selector:
+    run: ingress-default-backend
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: haproxy-ingress
+  namespace: default
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: haproxy-ingress-tcp
+  namespace: default
+data:
+  "22": "default/gitea-server:22"
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  labels:
+    run: haproxy-ingress
+  name: haproxy-ingress
+  namespace: default
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  selector:
+    matchLabels:
+      run: haproxy-ingress
+  template:
+    metadata:
+      labels:
+        run: haproxy-ingress
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        role: ingress-controller
+      serviceAccountName: ingress-controller
+      containers:
+      - name: haproxy-ingress
+        image: quay.io/jcmoraisjr/haproxy-ingress
+        args:
+        - --default-backend-service=$(POD_NAMESPACE)/ingress-default-backend
+        - --configmap=$(POD_NAMESPACE)/haproxy-ingress
+        - --tcp-services-configmap=$(POD_NAMESPACE)/haproxy-ingress-tcp
+        - --sort-backends
+        ports:
+        - name: http
+          containerPort: 80
+        - name: https
+          containerPort: 443
+        - name: stat
+          containerPort: 1936
+        livenessProbe:
+          httpGet:
+            path: /healthz
+            port: 10253
+        env:
+        - name: POD_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: POD_NAMESPACE
+          value: default
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: gitea-server-nodeport
+  namespace: default
+  labels:
+    app: gitea
+    name: gitea-server
+  annotations:
+    # add an annotation indicating the issuer to use.
+    # TODO: Switch to production when we're ready
+    certmanager.k8s.io/cluster-issuer: letsencrypt-staging
+spec:
+  ports:
+    - name: http
+      port: 3000
+      targetPort: 3000
+    - name: ssh
+      port: 22
+      targetPort: 22
+      protocol: TCP
+  type: NodePort
+  selector:
+    name: gitea-server
+---
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: gitea-ingress
+  namespace: default
+  labels:
+    name: gitea-server
+    app: gitea
+  annotations:
+    kubernetes.io/ingress.class: "haproxy"
+spec:
+  tls:
+  - hosts:
+    - gitea.kosmos.org
+    secretName: gitea-kosmos-org-cert
+  rules:
+  - host: gitea.kosmos.org
+    http:
+      paths:
+      - path: /
+        backend:
+          serviceName: gitea-server-nodeport
+          servicePort: 3000

kubernetes/gitea-server-statefulset.yaml

@@ -1,103 +0,0 @@
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
-  name: gitea-server
-  labels:
-    app: gitea
-spec:
-  serviceName: gitea-server
-  updateStrategy:
-    type: RollingUpdate
-  replicas: 1
-  selector:
-    matchLabels:
-      app: gitea
-  template:
-    metadata:
-      labels:
-        name: gitea-server
-        app: gitea
-    spec:
-      initContainers:
-        - name: init-config
-          # This is a busybox image with our gitea customizations saved to
-          # /custom, built using ./script/build_customizations_image from the
-          # root of the repo
-          image: eu.gcr.io/fluted-magpie-218106/gitea_custom:0.1.2
-          command: [
-            'sh', '-c',
-            'mkdir -p /data/gitea/conf && mkdir -p /data/gitea/https && cp /root/conf/app.ini /data/gitea/conf/app.ini && chown 1000:1000 /data/gitea/conf/app.ini && chmod 660 /data/gitea/conf/app.ini && cp /root/conf/*.pem /data/gitea/https && chmod 600 /data/gitea/https/*.pem && cp -R /custom/* /data/gitea && chown -R 1000:1000 /data/gitea'
-          ]
-          volumeMounts:
-          - mountPath: /data
-            name: gitea-server-data
-          - mountPath: /root/conf
-            name: config
-      containers:
-      - name: gitea-server
-        image: gitea/gitea:1.10.1
-        ports:
-        - containerPort: 3000
-        - containerPort: 3001
-        - containerPort: 22
-        volumeMounts:
-        - mountPath: /data
-          name: gitea-server-data
-        resources:
-          requests:
-            cpu: 150m
-            memory: 256Mi
-          limits:
-            cpu: 250m
-            memory: 512Mi
-      restartPolicy: Always
-      volumes:
-      - name: config
-        secret:
-          secretName: gitea-config
-          items:
-            - key: app.ini
-              path: app.ini
-              mode: 256
-            - key: cert.pem
-              path: cert.pem
-              mode: 256
-            - key: key.pem
-              path: key.pem
-              mode: 256
-  volumeClaimTemplates:
-    - metadata:
-        name: gitea-server-data
-        labels:
-          app: gitea
-      spec:
-        accessModes:
-        - ReadWriteOnce
-        resources:
-          requests:
-            storage: 20Gi
-        storageClassName: gcePersistentDisk
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: gitea-server
-  labels:
-    name: gitea-server
-    app: gitea
-spec:
-  type: LoadBalancer
-  # preserves the client source IP
-  externalTrafficPolicy: Local
-  ports:
-    - name: "ssh"
-      port: 22
-      targetPort: 22
-    - name: "http"
-      port: 80
-      targetPort: 3001
-    - name: "https"
-      port: 443
-      targetPort: 3000
-  selector:
-    name: gitea-server

kubernetes/gitea-server.yaml

@@ -1,4 +1,4 @@
-apiVersion: apps/v1
+apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
   name: gitea-server
@@ -6,9 +6,6 @@ metadata:
     app: gitea
 spec:
   replicas: 1
-  selector:
-    matchLabels:
-      app: gitea
   template:
     metadata:
       labels:
@@ -20,7 +17,7 @@ spec:
           # This is a busybox image with our gitea customizations saved to
           # /custom, built using ./script/build_customizations_image from the
           # root of the repo
-          image: eu.gcr.io/fluted-magpie-218106/gitea_custom:0.1.2
+          image: eu.gcr.io/fluted-magpie-218106/gitea_custom:0.1
           command: [
             'sh', '-c',
             'mkdir -p /data/gitea/conf && mkdir -p /data/gitea/https && cp /root/conf/app.ini /data/gitea/conf/app.ini && chown 1000:1000 /data/gitea/conf/app.ini && chmod 660 /data/gitea/conf/app.ini && cp /root/conf/*.pem /data/gitea/https && chmod 600 /data/gitea/https/*.pem && cp -R /custom/* /data/gitea && chown -R 1000:1000 /data/gitea'
@@ -32,20 +29,33 @@ spec:
             name: config
       containers:
       - name: gitea-server
-        image: gitea/gitea:1.10.1
+        image: gitea/gitea:1.8.1
         ports:
         - containerPort: 3000
-        - containerPort: 3001
         - containerPort: 22
+        livenessProbe:
+          httpGet:
+            path: /
+            port: 3000
+            scheme: HTTP
+          initialDelaySeconds: 30
+          timeoutSeconds: 5
+        readinessProbe:
+          httpGet:
+            path: /
+            port: 3000
+            scheme: HTTP
+          initialDelaySeconds: 30
+          timeoutSeconds: 5
         volumeMounts:
         - mountPath: /data
           name: gitea-server-data
         resources:
           requests:
-            cpu: 150m
+            cpu: 250m
             memory: 256Mi
           limits:
-            cpu: 250m
+            cpu: 500m
             memory: 512Mi
       restartPolicy: Always
       volumes:
@@ -96,9 +106,6 @@ spec:
       targetPort: 22
     - name: "http"
       port: 80
-      targetPort: 3001
-    - name: "https"
-      port: 443
       targetPort: 3000
   selector:
     name: gitea-server

kubernetes/letsencrypt-production.yaml

@@ -0,0 +1,19 @@
+apiVersion: certmanager.k8s.io/v1alpha1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-production
+spec:
+  acme:
+    # You must replace this email address with your own.
+    # Let's Encrypt will use this to contact you about expiring
+    # certificates, and issues related to your account.
+    email: ops@kosmos.org
+    server: https://acme-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      # Secret resource used to store the account's private key.
+      name: letsencrypt-production-account-key
+    # Add a single challenge solver, HTTP01 using nginx
+    solvers:
+    - http01:
+        ingress:
+          class: nginx

kubernetes/letsencrypt-staging.yaml

@@ -0,0 +1,19 @@
+apiVersion: certmanager.k8s.io/v1alpha1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-staging
+spec:
+  acme:
+    # You must replace this email address with your own.
+    # Let's Encrypt will use this to contact you about expiring
+    # certificates, and issues related to your account.
+    email: ops@kosmos.org
+    server: https://acme-staging-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      # Secret resource used to store the account's private key.
+      name: letsencrypt-staging-account-key
+    # Add a single challenge solver, HTTP01 using nginx
+    solvers:
+    - http01:
+        ingress:
+          class: nginx

packer/custom.json

@@ -21,7 +21,7 @@
       {
         "type": "docker-tag",
         "repository": "eu.gcr.io/fluted-magpie-218106/gitea_custom",
-        "tag": "0.1.2"
+        "tag": "0.1"
       },
       "docker-push"
     ]