Ark backups stopped being performed #27

New Issue

greg · 2019-06-19T13:53:12Z

greg commented

2019-06-19 13:53:12 +00:00

The last backups are from May 27

The BackupStorageLocation is invalid according to the logs, but it does look valid.

An error occurred: some backup storage locations are invalid: backup store for location "default" is invalid (if upgrading from a pre-v0.10 version of Ark, please refer to https://heptio.github.io/ark/v0.10.0/storage-layout-reorg-v0.10 for instructions): Backup store contains invalid top-level directories: [stderr stdout]

I thought I had fixed it by recreating the BackupStorageLocation resource, but I had restarted the pod before it was recreated, so the pod ran successfully without any BackupStorageLocation

There isn't much in a BackupStorageLocation:

apiVersion: ark.heptio.com/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: heptio-ark
spec:
  provider: gcp
  objectStorage:
    bucket: sidamo-backups

I am still working on it

The last backups are from May 27 The `BackupStorageLocation` is invalid according to the logs, but it does look valid. ``` An error occurred: some backup storage locations are invalid: backup store for location "default" is invalid (if upgrading from a pre-v0.10 version of Ark, please refer to https://heptio.github.io/ark/v0.10.0/storage-layout-reorg-v0.10 for instructions): Backup store contains invalid top-level directories: [stderr stdout] ``` I thought I had fixed it by recreating the BackupStorageLocation resource, but I had restarted the pod before it was recreated, so the pod ran successfully without any BackupStorageLocation There isn't much in a BackupStorageLocation: ``` apiVersion: ark.heptio.com/v1 kind: BackupStorageLocation metadata: name: default namespace: heptio-ark spec: provider: gcp objectStorage: bucket: sidamo-backups ``` I am still working on it

greg self-assigned this 2019-06-19 13:53:12 +00:00

greg added the

bug

ops

labels 2019-06-19 13:53:12 +00:00

greg referenced this issue from a commit

2019-06-19 16:35:18 +00:00

Update the documentation, Ark is now Velero Refs #27

greg commented

2019-06-19 16:38:32 +00:00

I ended up switching to Velero 1.0 (Ark got renamed at version 0.11) using the new installation process that's part of the velero binary and using a new bucket. I could not figure out what was wrong with the current bucket. The same error was happening with velero as it did with our previous Ark setup.

I have executed a manual backup, now we need to wait until the next automated backup takes place to confirm everything works again before we can close this

I ended up switching to Velero 1.0 (Ark got renamed at version 0.11) using the new installation process that's part of the velero binary and using a new bucket. I could not figure out what was wrong with the current bucket. The same error was happening with velero as it did with our previous Ark setup. I have executed a manual backup, now we need to wait until the next automated backup takes place to confirm everything works again before we can close this

raucao commented

2019-06-19 16:50:27 +00:00

No description provided.

thumbs-up-asian-guy-gif.gif

480 KiB

greg added the

kredits-2

label 2019-06-20 09:13:46 +00:00

greg commented

2019-06-20 09:13:51 +00:00

Hooray, that did the trick:

$ velero get backups
NAME                            STATUS      CREATED                          EXPIRES   STORAGE LOCATION   SELECTOR
drone-ci-daily-20190620000000   Completed   2019-06-20 02:00:00 +0200 CEST   29d       default            app=drone-ci
gitea-daily-20190620000000      Completed   2019-06-20 02:00:05 +0200 CEST   29d       default            app=gitea

And the matching snapshots: https://console.cloud.google.com/compute/snapshots?organizationId=772167872692&project=fluted-magpie-218106&tab=snapshots&snapshotssize=50

Hooray, that did the trick: ``` $ velero get backups NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR drone-ci-daily-20190620000000 Completed 2019-06-20 02:00:00 +0200 CEST 29d default app=drone-ci gitea-daily-20190620000000 Completed 2019-06-20 02:00:05 +0200 CEST 29d default app=gitea ``` And the matching snapshots: https://console.cloud.google.com/compute/snapshots?organizationId=772167872692&project=fluted-magpie-218106&tab=snapshots&snapshotssize=50

greg commented

2019-06-20 09:16:18 +00:00

I have also deleted the heptio-ark namespace and the matching service account and IAM. Closing this one

greg closed this issue

2019-06-20 09:16:18 +00:00

This repo is archived. You cannot comment on issues.

2 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/gitea.kosmos.org#27