I have also deleted the heptio-ark namespace and the matching service account and IAM. Closing this one
I ended up switching to Velero 1.0 (Ark got renamed at version 0.11) using the new installation process that's part of the velero binary and using a new bucket. I could not figure out what was wrong with the current bucket. The same error was happening with velero as it did with our previous Ark setup.
You're right, I misunderstood, that would work with the cerbot systemd timer as it is (systemctl status --full certbot will contain the previous error). I'm going to close this and reopen another pull request that does it with a OnFailure section on the certbot service instead
Damnit, certbot is using stderr to output warnings, so we got an email when it ran at midnight:
I checked, no other change is needed for cron jobs so far, the backup gem is already successfully sending emails when warnings or errors occur (independently of using cron).
Update: SSH doesn't work with a DaemonSet in front of the haproxy-ingress controller, they already have an SSH daemon running on port 22 on the nodes.
Running a LoadBalancer in front of the ingress controller is one solution that I found works on GKE, I did not say it was the only way
I got haproxy-ingress to work without a LoadBalancer on GKE, using a DaemonSet for the Ingress controller. That way the haproxy Ingress controller runs on every Kubernetes node (4 in our case) and is accessible from the outside through each node's public IP
The GKE pricing is really confusing, I'm trying to find a definitive answer on the Ingress pricing. My previous understanding was that you can run one Ingress to replace multiple LoadBalancers too
A LoadBalancer is needed on GKE when using an Ingress controller that's not built-in. The built-in Ingress controller is for HTTP(S), and can serve traffic directly without a LoadBalancer. Here is a tutorial about deploying ingress-nginx on GKE, with a LoadBalancer in front of the ingress controller (https://cloud.google.com/community/tutorials/nginx-ingress-gke). And the docs about HTTP(S) load balancing with Ingress on GKE: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress
We're using systemd timers to run certbot, so now this issue is about both cron jobs and system timers. I'm going to work on that one