Greg greg
  • Joined on 2018-11-05
greg commented on issue kosmos/chef#147 2020-05-29 09:08:59 +00:00
Prepare Gitea migration (from GKE to Kosmos server)

Copied from the PR:

I think we can finally migrate away Gitea to Centaurus (with the DB on Andromeda as the master). Tomorrow I will check everything on Andromeda and Centaurus, then we can pick a time and date to do the switch. It will involve a DNS switch, in preparation for this I have just lowered the TTL on gitea.kosmos.org down to 300s, the lowest Gandi supports. It was previously set to 1800s.

Here are my notes for the dump/import:

Perform a dump

From https://discourse.gitea.io/t/migrate-gitea-db-from-mariadb-to-postgresql/2072/3

$ kubectl exec gitea-server-5d57fc877d-ghvps -n default -- /app/gitea/gitea dump -d postgres -c /data/gitea/conf/app.ini -f /data/gitea/gitea-dump.zip
$ kubectl cp gitea-server-5d57fc877d-ghvps:/data/gitea/gitea-dump.zip gitea-dump.zip
$ kubectl exec gitea-server-5d57fc877d-ghvps -n default -- /bin/rm /data/gitea/gitea-dump.zip

Import the dump

on Andromeda:

$ sudo su - postgres -c "psql gitea < gitea-db.sql"

on Centaurus:

SCP and unzip gitea-dump.zip

$ scp gitea-dump.zip centaurus.kosmos.org:
$ ssh centaurus.kosmos.org
$ sudo systemctl stop gitea
$ mkdir dump; mv gitea-dump.zip dump; cd dump; unzip gitea-dump.zip; sudo cp -R repositories /home/git/gitea-repositories
# Copy the content of `data`:
$ sudo cp -R dump/data/sessions /var/lib/gitea/
$ sudo cp -R dump/data/indexers /var/lib/gitea/data/
$ sudo cp -R dump/data/attachments /var/lib/gitea/data/
$ sudo cp -R dump/data/avatars /var/lib/gitea/data/
$ sudo cp -R dump/data/repo-avatars /var/lib/gitea/data/
$ sudo rm -rf /var/lib/gitea/data/queues; sudo cp -R dump/data/queues /var/lib/gitea/data/
$ sudo chown -R git:git /var/lib/gitea/data/
$ sudo systemctl start gitea

Log in, run Update the '.ssh/authorized_keys' file with Gitea SSH keys on http://gitea.kosmos.org/admin so the users' public keys are copied to the /home/git/.ssh/authorized_key file

greg commented on pull request kosmos/chef#168 2020-05-29 09:07:32 +00:00
Initial kosmos_gitea cookbook

I think the only thing that is not clear to me is:

Log in, run Update the '.ssh/authorized_keys' file with Gitea SSH keys on http://gitea.kosmos.org/admin

As we're importing a database dump, the /home/git/.ssh/authorized_keys file on centaurus will be empty at first, this admin task generates the content of the file with users' public key. It is then managed by Gitea, so new keys are automatically added, deleted keys deleted, etc

Also, are you sure the deployment should be part of the cookbook PR? We agreed that only master is being deployed to production, and I also don't see how the cookbook PR should include an entire production migration to begin with. The work done for that is different than for the cookbook code.

That's right, I will move this checklist to the issue instead

greg commented on pull request kosmos/chef#168 2020-05-28 17:06:06 +00:00
Initial kosmos_gitea cookbook

I think we can finally migrate away Gitea to Centaurus (with the DB on Andromeda as the master). Tomorrow I will check everything on Andromeda and Centaurus, then we can pick a time and date to do the switch. It will involve a DNS switch, in preparation for this I have just lowered the TTL on gitea.kosmos.org down to 300s, the lowest Gandi supports. It was previously set to 1800s.

Here are my notes for the dump/import:

Perform a dump

From https://discourse.gitea.io/t/migrate-gitea-db-from-mariadb-to-postgresql/2072/3

$ kubectl exec gitea-server-5d57fc877d-ghvps -n default -- /app/gitea/gitea dump -d postgres -c /data/gitea/conf/app.ini -f /data/gitea/gitea-dump.zip
$ kubectl cp gitea-server-5d57fc877d-ghvps:/data/gitea/gitea-dump.zip gitea-dump.zip
$ kubectl exec gitea-server-5d57fc877d-ghvps -n default -- /bin/rm /data/gitea/gitea-dump.zip

Import the dump

on Andromeda:

$ sudo su - postgres -c "psql gitea < gitea-db.sql"

on Centaurus:

SCP and unzip gitea-dump.zip

$ scp gitea-dump.zip centaurus.kosmos.org:
$ ssh centaurus.kosmos.org
$ sudo systemctl stop gitea
$ mkdir dump; mv gitea-dump.zip dump; cd dump; unzip gitea-dump.zip; sudo cp -R repositories /home/git/gitea-repositories
# Copy the content of `data`:
$ sudo cp -R dump/data/sessions /var/lib/gitea/
$ sudo cp -R dump/data/indexers /var/lib/gitea/data/
$ sudo cp -R dump/data/attachments /var/lib/gitea/data/
$ sudo cp -R dump/data/avatars /var/lib/gitea/data/
$ sudo cp -R dump/data/repo-avatars /var/lib/gitea/data/
$ sudo rm -rf /var/lib/gitea/data/queues; sudo cp -R dump/data/queues /var/lib/gitea/data/
$ sudo chown -R git:git /var/lib/gitea/data/
$ sudo systemctl start gitea

Log in, run Update the '.ssh/authorized_keys' file with Gitea SSH keys on http://gitea.kosmos.org/admin so authorized SSH keys work for everyone

greg pushed to feature/147-gitea_cookbook at kosmos/chef 2020-05-28 16:45:36 +00:00
94330f2052 Comment out the COOKIE_SECURE config for now
baaae695af Merge branch 'master' into feature/147-gitea_cookbook
5b2d4f269d Merge branch 'bugfix/171-letsencrypt_resource' of kosmos/chef into master
210c76c479 Fix the name of the Let's Encrypt cert execute resource
6469d2286e Merge branch 'feature/zoom_options' of kosmos/chef into master
Compare 8 commits »
greg closed issue kosmos/chef#171 2020-05-26 15:25:31 +00:00
InvalidResourceSpecification: The string `letsencrypt cert for ldap.kosmos.org' is not valid for resource collection lookup
greg merged pull request kosmos/chef#172 2020-05-26 15:25:31 +00:00
Fix the name of the Let's Encrypt cert execute resource
greg pushed to master at kosmos/chef 2020-05-26 15:25:31 +00:00
5b2d4f269d Merge branch 'bugfix/171-letsencrypt_resource' of kosmos/chef into master
210c76c479 Fix the name of the Let's Encrypt cert execute resource
Compare 2 commits »
greg pushed to feature/147-gitea_cookbook at kosmos/chef 2020-05-26 13:22:28 +00:00
baa0739936 Add the backup recipe
3332a1b2e8 Write initial README
Compare 2 commits »
greg created pull request kosmos/chef#172 2020-05-26 12:13:16 +00:00
Fix the name of the Let's Encrypt cert execute resource
greg pushed to bugfix/171-letsencrypt_resource at kosmos/chef 2020-05-26 12:12:41 +00:00
210c76c479 Fix the name of the Let's Encrypt cert execute resource
greg pushed to feature/147-gitea_cookbook at kosmos/chef 2020-05-21 09:51:29 +00:00
1f0e2ccbdd Move the binary URL to an attribute
greg commented on pull request kosmos/chef#168 2020-05-21 08:59:20 +00:00
Initial kosmos_gitea cookbook

I think building from source is out of scope for this PR, but since this is simply deploying a binary it wouldn't be a big task like building a Docker image. I am already moving the URL of the binary to a cookbook attribute so it's easier to override

greg created pull request kosmos/chef#168 2020-05-18 17:47:34 +00:00
WIP: Initial kosmos_gitea cookbook
greg pushed to feature/147-gitea_cookbook at kosmos/chef 2020-05-18 17:44:04 +00:00
51d4d88568 Initial kosmos_gitea cookbook
20cbc678bc Add a method that returns the PostgreSQL service
greg deleted branch bugfix/160-cookbook_fixes from kosmos/chef 2020-05-16 08:53:36 +00:00
greg pushed to master at kosmos/chef 2020-05-16 08:53:29 +00:00
82f1e9863b Merge branch 'bugfix/160-cookbook_fixes' of kosmos/chef into master
d79cdf087b Move the PGPASS environment variable to the execute resource
31dc14e88c Fix the firewall rules for PostgreSQL
Compare 3 commits »
greg merged pull request kosmos/chef#167 2020-05-16 08:53:29 +00:00
Fix the firewall rules and move the password env variable
greg commented on pull request kosmos/chef#166 2020-05-16 08:00:45 +00:00
Encrypt PostgreSQL data directory

Another option is to not start the PostgreSQL service on boot and run a script on boot where you input the encrypted volume password and then it starts PostgreSQL

greg commented on issue kosmos/chef#160 2020-05-15 16:50:57 +00:00
Set up Postgres replication

Replication is now running on centaurus. I had to fix the firewall rules to make it work: #167. The data dir for PostgreSQL is also encrypted using encfs: #166. Both are ready to merge

greg created pull request kosmos/chef#167 2020-05-15 16:50:43 +00:00
Fix the firewall rules and move the password env variable