Move PostgreSQL to VMs and access via Zerotier #282

raucao · 2021-01-07T14:14:24Z

raucao commented

2021-01-07 14:14:24 +00:00

Update Jan 24:

@greg and I have finished the migration of our PostgreSQL cluster from the host systems of Andromeda, Centaurus, and Draco, to new VMs on Centaurus and Draco.

Instead of the previous encfs setup, we now use standard full-disk encryption (LUKS/LVM) for the VMs to encrypt data at rest. So the decryption password has to be provided during the booting of the VM now.

The new setup also uses our Zerotier private network exclusively for database connections, and thus no public connections or TLS anymore.

In order to make both this migration seamless, as well make the switching of the primary node to a hot standby node much easier and quicker in the future as well, we have changed all database clients to use a custom /etc/hosts entry (pg.kosmos.local).

Then we created a script to run all the steps for switching the primary at once, both triggering the promotion of a standby node to primary, as well as stopping the old one, and updating the hosts entries without full Chef runs. This way, all clients will immediately re-connect to the new primary after promotion, which worked exactly as intended when we did it for this migration.

refs #280

***Update Jan 24:*** @greg and I have finished the migration of our PostgreSQL cluster from the host systems of Andromeda, Centaurus, and Draco, to new VMs on Centaurus and Draco. Instead of the previous encfs setup, we now use standard full-disk encryption (LUKS/LVM) for the VMs to encrypt data at rest. So the decryption password has to be provided during the booting of the VM now. The new setup also uses our Zerotier private network exclusively for database connections, and thus no public connections or TLS anymore. In order to make both this migration seamless, as well make the switching of the primary node to a hot standby node much easier and quicker in the future as well, we have changed all database clients to use a custom `/etc/hosts` entry (`pg.kosmos.local`). Then we created [a script to run all the steps for switching the primary at once](https://gitea.kosmos.org/kosmos/chef/src/commit/b1fea4b09ffe87c48b5b1d8b524cb0968c13fabf/scripts/postgresql/switch_primary.sh), both triggering the promotion of a standby node to primary, as well as stopping the old one, and updating the hosts entries without full Chef runs. This way, all clients will immediately re-connect to the new primary after promotion, which worked exactly as intended when we did it for this migration. refs #280

raucao self-assigned this 2021-01-08 08:14:10 +00:00

greg was assigned by raucao

2021-01-08 08:14:13 +00:00

raucao commented

2021-01-09 09:51:39 +00:00

@greg I was thinking about the most simple, low-tech, and stable solution for updating the postgres master address, and my last idea was:

Configure clients to use a hostname instead of IP address for the master connection (the obvious first step)
Write a script (maybe just postgres client Chef recipe), which simply updates an /etc/hosts entry on all postgres client VMs

This way, we don't need our own DNS server at all, and there's no extra connection or caching to think about. WDYT?

@greg I was thinking about the most simple, low-tech, and stable solution for updating the postgres master address, and my last idea was: * Configure clients to use a hostname instead of IP address for the master connection (the obvious first step) * Write a script (maybe just postgres client Chef recipe), which simply updates an `/etc/hosts` entry on all postgres client VMs This way, we don't need our own DNS server at all, and there's no extra connection or caching to think about. WDYT?

greg commented

2021-01-09 12:54:50 +00:00

Yeah that seems like a good solution for now!

raucao added 1 commit 2021-01-23 16:06:29 +00:00

bd48dab1d8

Set up postgres-3 VM

greg added 9 commits 2021-01-23 16:12:01 +00:00

fdd70d1872 Remove postgres-1 server

It was created with an encfs volume inside a VM, we want full disk encryption
instead. I have deleted the VM from centaurus as well as its disk and
have also unauthorized its id from the zerotier controller

bb0e73d1b9 Switch ejabberd, mastodon and gitea to a hostname for Postgres

1e3a2b40d4 Add the postgresql_client role to the akkounts role

286cd2f2a3 Move the postgresql_client role to the top

We want the host file to be written before database connections happen

ee7b3626be Generate a host entry for the current PostgreSQL primary

39d0304ab4 Update the akkounts-1 node after Chef run

40fde8a861 Update the ejabberd nodes after Chef run

4e1bd458f3 Update the mastodon-1 node after Chef run

5cb390f340 Remove the postgresql_replica from centaurus, run Chef

I have also deleted the data directory (in /mnt/data)

greg added 1 commit 2021-01-23 16:35:27 +00:00

a4a35da0ff Replace the hardcoded primary pg IP with a hostname

greg added 1 commit 2021-01-23 18:09:13 +00:00

a7116b8fe5 Switch the TLS mode to disabled for Gitea

We connect through a Zerotier private IP

greg added 1 commit 2021-01-23 18:10:05 +00:00

7e61e9cb45 Promote a new PostgreSQL primary: postgres-2

greg added 2 commits 2021-01-23 18:15:14 +00:00

112eb903ec Add a script to switch the primary PostgreSQL server

Usage: scripts/postgresql/switch_primary.sh <old_primary_hostname> <new_primary_hostname>

7ce8b7d461 Enable PostgreSQL service

greg added 1 commit 2021-01-23 18:26:02 +00:00

b1fea4b09f Update the postgres-3 node after Chef run

raucao changed title from ~~WIP: Move PostgreSQL to VMs and access via Zerotier~~ to Move PostgreSQL to VMs and access via Zerotier

2021-01-23 18:30:06 +00:00

raucao added the

kredits-3

label 2021-01-23 18:34:06 +00:00

raucao requested review from Owners 2021-01-24 07:22:19 +00:00

greg added 1 commit 2021-01-24 09:11:53 +00:00

1ad7a6936c Generate the hosts entry for the PostgreSQL replica

slvrbckt commented

2021-01-24 15:52:21 +00:00

I had a look, and it looks good to me, but I know absolutely zero about the ruby code, let alone the infrastructure. So take my approval for a grain of salt :)

EDIT: Also, I couldn't add my name to the reviewrs list

I had a look, and it looks good to me, but I know absolutely zero about the ruby code, let alone the infrastructure. So take my approval for a grain of salt :) EDIT: Also, I couldn't add my name to the reviewrs list

raucao commented

2021-01-24 16:32:17 +00:00

EDIT: Also, I couldn't add my name to the reviewrs list

If you leave a comment using the review function on the changed-files tab (green button on the right), then you'll be added to the list, and also kredited by @galfert's new script.

> EDIT: Also, I couldn't add my name to the reviewrs list If you leave a comment using the review function on the changed-files tab (green button on the right), then you'll be added to the list, and also kredited by @galfert's new script.

slvrbckt approved these changes 2021-01-24 16:47:39 +00:00

slvrbckt left a comment

LGTM!

raucao merged commit ad271e55d4 into master

2021-01-25 10:56:42 +00:00

raucao referenced this issue from a commit

2021-01-25 10:56:42 +00:00

Merge pull request 'Move PostgreSQL to VMs and access via Zerotier' (#282) from feature/postgres_vms into master

raucao deleted branch feature/postgres_vms

2021-01-25 10:56:48 +00:00

Sign in to join this conversation.

No reviewers

No Milestone

No project

No Assignees

3 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/chef#282