Ark backups stopped being performed
The last backups are from May 27
BackupStorageLocation is invalid according to the logs, but it does look valid.
An error occurred: some backup storage locations are invalid: backup store for location "default" is invalid (if upgrading from a pre-v0.10 version of Ark, please refer to https://heptio.github.io/ark/v0.10.0/storage-layout-reorg-v0.10 for instructions): Backup store contains invalid top-level directories: [stderr stdout]
I thought I had fixed it by recreating the BackupStorageLocation resource, but I had restarted the pod before it was recreated, so the pod ran successfully without any BackupStorageLocation
There isn't much in a BackupStorageLocation:
apiVersion: ark.heptio.com/v1 kind: BackupStorageLocation metadata: name: default namespace: heptio-ark spec: provider: gcp objectStorage: bucket: sidamo-backups
I am still working on it
I ended up switching to Velero 1.0 (Ark got renamed at version 0.11) using the new installation process that's part of the velero binary and using a new bucket. I could not figure out what was wrong with the current bucket. The same error was happening with velero as it did with our previous Ark setup.
I have executed a manual backup, now we need to wait until the next automated backup takes place to confirm everything works again before we can close this
Hooray, that did the trick:
$ velero get backups NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR drone-ci-daily-20190620000000 Completed 2019-06-20 02:00:00 +0200 CEST 29d default app=drone-ci gitea-daily-20190620000000 Completed 2019-06-20 02:00:05 +0200 CEST 29d default app=gitea
I have also deleted the heptio-ark namespace and the matching service account and IAM. Closing this one
Deleting a branch is permanent. It CANNOT be undone. Continue?