We have performed the migration, downtime was only a few seconds. We have also tried a running update for a non existing Docker image tag. The new pod kept running, a new one was created and failed to pull the image; I reverted the Docker image to the 1.9.0 tag and the failed pod was deleting, leaving the previous one.
It turns out we can get rid of one of the 3 small nodes after all. The master is managed by GKE (https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#master). The minimum of 3 nodes in a cluster is when you run your own cluster, because etcd needs at least 3 nodes for a production cluster; this is what the Kubernetes docs and blog posts are talking about, not the case of a managed service like Amazon EKS or GKE. This is confusing because they are still "nodes", but they don't have a role and only run pods, they do not provide the actual Kubernetes API (the master managed by GKE does)
I checked on barnard and andromeda, the only cron job is backup (already notifies on failures/warnings) and the certbot timer is the only one that I found that needs email notifications. Closing this one
I have managed to update mysql-server-5.7, after stopping /etc/systemd/system/mysql.service, moving it, running systemctl daemon-reload.
On andromeda there is a /etc/systemd/system/mysql.service file that was created by Chef, on a fresh VM the service is provided by the package and is located at /lib/systemd/system/mysql.service. So it looks like an issue caused by past versions of the mysql cookbook
The apt preferences files you created are being ignored because of their name (apt thinks .7 is the extension, so they would need to end with .pref). Here's the manpage