Initial kosmos_gitea cookbook #168

2020-05-18T17:47:33Z

greg commented

2020-05-18 17:47:33 +00:00

The default recipe deploys the gitea binary, generates a config file and our custom Kosmos label set. The service runs as a Systemd unit.

The pg_db recipe needs to run on the primary PostgreSQL (currently andromeda).

The backup recipe is empty for now

TODO:

Check if running the Update the '.ssh/authorized_keys' file with Gitea SSH keys. (Not needed for the built-in SSH server.) task on the admin screen is needed (answer: yes, this is needed, I have found a way to run it from the command line: sudo su - git -c "/usr/local/bin/gitea admin regenerate keys --config /etc/gitea/app.ini")
Write the backup recipe (for the gitea data)
Write docs (README)

Refs #147

The default recipe deploys the gitea binary, generates a config file and our custom Kosmos label set. The service runs as a Systemd unit. The pg_db recipe needs to run on the primary PostgreSQL (currently andromeda). The backup recipe is empty for now TODO: * [x] Check if running the `Update the '.ssh/authorized_keys' file with Gitea SSH keys. (Not needed for the built-in SSH server.)` task on the admin screen is needed (answer: yes, this is needed, I have found a way to run it from the command line: `sudo su - git -c "/usr/local/bin/gitea admin regenerate keys --config /etc/gitea/app.ini"`) * [x] Write the backup recipe (for the gitea data) * [x] Write docs (README) Refs #147

raucao commented

2020-05-18 19:49:08 +00:00

Nice!

I thought we said we'd build it from source, however, so that we can easily deploy our own fixes and customizations?

Nice! I thought we said we'd build it from source, however, so that we can easily deploy our own fixes and customizations?

greg commented

2020-05-21 08:59:19 +00:00

I think building from source is out of scope for this PR, but since this is simply deploying a binary it wouldn't be a big task like building a Docker image. I am already moving the URL of the binary to a cookbook attribute so it's easier to override

👍 1

greg marked the pull request as ready for review 2020-05-28 16:59:18 +00:00

greg commented

2020-05-28 17:06:05 +00:00

I think we can finally migrate away Gitea to Centaurus (with the DB on Andromeda as the master). Tomorrow I will check everything on Andromeda and Centaurus, then we can pick a time and date to do the switch. It will involve a DNS switch, in preparation for this I have just lowered the TTL on gitea.kosmos.org down to 300s, the lowest Gandi supports. It was previously set to 1800s.

Here are my notes for the dump/import:

Perform a dump

From https://discourse.gitea.io/t/migrate-gitea-db-from-mariadb-to-postgresql/2072/3

$ kubectl exec gitea-server-5d57fc877d-ghvps -n default -- /app/gitea/gitea dump -d postgres -c /data/gitea/conf/app.ini -f /data/gitea/gitea-dump.zip
$ kubectl cp gitea-server-5d57fc877d-ghvps:/data/gitea/gitea-dump.zip gitea-dump.zip
$ kubectl exec gitea-server-5d57fc877d-ghvps -n default -- /bin/rm /data/gitea/gitea-dump.zip

Import the dump

on Andromeda:

$ sudo su - postgres -c "psql gitea < gitea-db.sql"

on Centaurus:

SCP and unzip gitea-dump.zip

$ scp gitea-dump.zip centaurus.kosmos.org:
$ ssh centaurus.kosmos.org

$ sudo systemctl stop gitea
$ mkdir dump; mv gitea-dump.zip dump; cd dump; unzip gitea-dump.zip; sudo cp -R repositories /home/git/gitea-repositories
# Copy the content of `data`:
$ sudo cp -R dump/data/sessions /var/lib/gitea/
$ sudo cp -R dump/data/indexers /var/lib/gitea/data/
$ sudo cp -R dump/data/attachments /var/lib/gitea/data/
$ sudo cp -R dump/data/avatars /var/lib/gitea/data/
$ sudo cp -R dump/data/repo-avatars /var/lib/gitea/data/
$ sudo rm -rf /var/lib/gitea/data/queues; sudo cp -R dump/data/queues /var/lib/gitea/data/
$ sudo chown -R git:git /var/lib/gitea/data/
$ sudo systemctl start gitea

Log in, run Update the '.ssh/authorized_keys' file with Gitea SSH keys on
http://gitea.kosmos.org/admin so the users' public keys are copied to the /home/git/.ssh/authorized_key file, allowing them to access git@gitea.kosmos.org

I think we can finally migrate away Gitea to Centaurus (with the DB on Andromeda as the master). Tomorrow I will check everything on Andromeda and Centaurus, then we can pick a time and date to do the switch. It will involve a DNS switch, in preparation for this I have just lowered the TTL on gitea.kosmos.org down to 300s, the lowest Gandi supports. It was previously set to 1800s. Here are my notes for the dump/import: ## Perform a dump From https://discourse.gitea.io/t/migrate-gitea-db-from-mariadb-to-postgresql/2072/3 ``` $ kubectl exec gitea-server-5d57fc877d-ghvps -n default -- /app/gitea/gitea dump -d postgres -c /data/gitea/conf/app.ini -f /data/gitea/gitea-dump.zip $ kubectl cp gitea-server-5d57fc877d-ghvps:/data/gitea/gitea-dump.zip gitea-dump.zip $ kubectl exec gitea-server-5d57fc877d-ghvps -n default -- /bin/rm /data/gitea/gitea-dump.zip ``` ## Import the dump ### on Andromeda: ``` $ sudo su - postgres -c "psql gitea < gitea-db.sql" ``` ### on Centaurus: SCP and unzip gitea-dump.zip ``` $ scp gitea-dump.zip centaurus.kosmos.org: $ ssh centaurus.kosmos.org ``` ``` $ sudo systemctl stop gitea $ mkdir dump; mv gitea-dump.zip dump; cd dump; unzip gitea-dump.zip; sudo cp -R repositories /home/git/gitea-repositories # Copy the content of `data`: $ sudo cp -R dump/data/sessions /var/lib/gitea/ $ sudo cp -R dump/data/indexers /var/lib/gitea/data/ $ sudo cp -R dump/data/attachments /var/lib/gitea/data/ $ sudo cp -R dump/data/avatars /var/lib/gitea/data/ $ sudo cp -R dump/data/repo-avatars /var/lib/gitea/data/ $ sudo rm -rf /var/lib/gitea/data/queues; sudo cp -R dump/data/queues /var/lib/gitea/data/ $ sudo chown -R git:git /var/lib/gitea/data/ $ sudo systemctl start gitea ``` Log in, run `Update the '.ssh/authorized_keys' file with Gitea SSH keys` on http://gitea.kosmos.org/admin so the users' public keys are copied to the `/home/git/.ssh/authorized_key` file, allowing them to access git@gitea.kosmos.org

👍 1 🚀 1

raucao commented

2020-05-28 17:15:54 +00:00

Sounds good!

I think the only thing that is not clear to me is:

Log in, run Update the '.ssh/authorized_keys' file with Gitea SSH keys on http://gitea.kosmos.org/admin

Sounds good! I think the only thing that is not clear to me is: > Log in, run `Update the '.ssh/authorized_keys' file with Gitea SSH keys` on http://gitea.kosmos.org/admin

raucao commented

2020-05-28 17:16:43 +00:00

Also, are you sure the deployment should be part of the cookbook PR? We agreed that only master is being deployed to production, and I also don't see how the cookbook PR should include an entire production migration to begin with. The work done for that is different than for the cookbook code.

greg commented

2020-05-29 09:07:32 +00:00

I think the only thing that is not clear to me is:

Log in, run Update the '.ssh/authorized_keys' file with Gitea SSH keys on http://gitea.kosmos.org/admin

As we're importing a database dump, the /home/git/.ssh/authorized_keys file on centaurus will be empty at first, this admin task generates the content of the file with users' public key. It is then managed by Gitea, so new keys are automatically added, deleted keys deleted, etc

Also, are you sure the deployment should be part of the cookbook PR? We agreed that only master is being deployed to production, and I also don't see how the cookbook PR should include an entire production migration to begin with. The work done for that is different than for the cookbook code.

That's right, I will move this checklist to the issue instead

> I think the only thing that is not clear to me is: >> Log in, run Update the '.ssh/authorized_keys' file with Gitea SSH keys on http://gitea.kosmos.org/admin As we're importing a database dump, the `/home/git/.ssh/authorized_keys` file on centaurus will be empty at first, this admin task generates the content of the file with users' public key. It is then managed by Gitea, so new keys are automatically added, deleted keys deleted, etc > Also, are you sure the deployment should be part of the cookbook PR? We agreed that only master is being deployed to production, and I also don't see how the cookbook PR should include an entire production migration to begin with. The work done for that is different than for the cookbook code. That's right, I will move this checklist to the issue instead

raucao commented

2020-05-29 10:50:33 +00:00

As we're importing a database dump, the /home/git/.ssh/authorized_keys file on centaurus will be empty at first, this admin task generates the content of the file with users’ public key. It is then managed by Gitea, so new keys are automatically added, deleted keys deleted, etc

Yes, that's obvious from the original post. But what is this task? It is just English text in your post, but is it a script somewhere? How is it run?

> As we're importing a database dump, the /home/git/.ssh/authorized_keys file on centaurus will be empty at first, this admin task generates the content of the file with users’ public key. It is then managed by Gitea, so new keys are automatically added, deleted keys deleted, etc Yes, that's obvious from the original post. But what is this task? It is just English text in your post, but is it a script somewhere? How is it run?

raucao commented

2020-05-29 10:51:11 +00:00