Ark backups stopped being performed #27

Closed
opened 2 years ago by greg · 4 comments
greg commented 2 years ago
Owner

The last backups are from May 27

The BackupStorageLocation is invalid according to the logs, but it does look valid.

An error occurred: some backup storage locations are invalid: backup store for location "default" is invalid (if upgrading from a pre-v0.10 version of Ark, please refer to https://heptio.github.io/ark/v0.10.0/storage-layout-reorg-v0.10 for instructions): Backup store contains invalid top-level directories: [stderr stdout]

I thought I had fixed it by recreating the BackupStorageLocation resource, but I had restarted the pod before it was recreated, so the pod ran successfully without any BackupStorageLocation

There isn't much in a BackupStorageLocation:

apiVersion: ark.heptio.com/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: heptio-ark
spec:
  provider: gcp
  objectStorage:
    bucket: sidamo-backups

I am still working on it

The last backups are from May 27 The `BackupStorageLocation` is invalid according to the logs, but it does look valid. ``` An error occurred: some backup storage locations are invalid: backup store for location "default" is invalid (if upgrading from a pre-v0.10 version of Ark, please refer to https://heptio.github.io/ark/v0.10.0/storage-layout-reorg-v0.10 for instructions): Backup store contains invalid top-level directories: [stderr stdout] ``` I thought I had fixed it by recreating the BackupStorageLocation resource, but I had restarted the pod before it was recreated, so the pod ran successfully without any BackupStorageLocation There isn't much in a BackupStorageLocation: ``` apiVersion: ark.heptio.com/v1 kind: BackupStorageLocation metadata: name: default namespace: heptio-ark spec: provider: gcp objectStorage: bucket: sidamo-backups ``` I am still working on it
greg self-assigned this 2 years ago
greg added the
bug
ops
labels 2 years ago
greg added
kredits-2
and removed
kredits-2
labels 2 years ago
Poster
Owner

I ended up switching to Velero 1.0 (Ark got renamed at version 0.11) using the new installation process that's part of the velero binary and using a new bucket. I could not figure out what was wrong with the current bucket. The same error was happening with velero as it did with our previous Ark setup.

I have executed a manual backup, now we need to wait until the next automated backup takes place to confirm everything works again before we can close this

I ended up switching to Velero 1.0 (Ark got renamed at version 0.11) using the new installation process that's part of the velero binary and using a new bucket. I could not figure out what was wrong with the current bucket. The same error was happening with velero as it did with our previous Ark setup. I have executed a manual backup, now we need to wait until the next automated backup takes place to confirm everything works again before we can close this
Owner
There is no content yet.
greg added the
kredits-2
label 2 years ago
Poster
Owner

Hooray, that did the trick:

$ velero get backups
NAME                            STATUS      CREATED                          EXPIRES   STORAGE LOCATION   SELECTOR
drone-ci-daily-20190620000000   Completed   2019-06-20 02:00:00 +0200 CEST   29d       default            app=drone-ci
gitea-daily-20190620000000      Completed   2019-06-20 02:00:05 +0200 CEST   29d       default            app=gitea

And the matching snapshots: https://console.cloud.google.com/compute/snapshots?organizationId=772167872692&project=fluted-magpie-218106&tab=snapshots&snapshotssize=50

Hooray, that did the trick: ``` $ velero get backups NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR drone-ci-daily-20190620000000 Completed 2019-06-20 02:00:00 +0200 CEST 29d default app=drone-ci gitea-daily-20190620000000 Completed 2019-06-20 02:00:05 +0200 CEST 29d default app=gitea ``` And the matching snapshots: https://console.cloud.google.com/compute/snapshots?organizationId=772167872692&project=fluted-magpie-218106&tab=snapshots&snapshotssize=50
Poster
Owner

I have also deleted the heptio-ark namespace and the matching service account and IAM. Closing this one

I have also deleted the heptio-ark namespace and the matching service account and IAM. Closing this one
greg closed this issue 2 years ago
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.