Move PostgreSQL to VMs and access via Zerotier #282

2021-01-07T14:14:24Z

raucao commented

2021-01-07 14:14:24 +00:00

Update Jan 24:

@greg and I have finished the migration of our PostgreSQL cluster from the host systems of Andromeda, Centaurus, and Draco, to new VMs on Centaurus and Draco.

Instead of the previous encfs setup, we now use standard full-disk encryption (LUKS/LVM) for the VMs to encrypt data at rest. So the decryption password has to be provided during the booting of the VM now.

The new setup also uses our Zerotier private network exclusively for database connections, and thus no public connections or TLS anymore.

In order to make both this migration seamless, as well make the switching of the primary node to a hot standby node much easier and quicker in the future as well, we have changed all database clients to use a custom /etc/hosts entry (pg.kosmos.local).

Then we created a script to run all the steps for switching the primary at once, both triggering the promotion of a standby node to primary, as well as stopping the old one, and updating the hosts entries without full Chef runs. This way, all clients will immediately re-connect to the new primary after promotion, which worked exactly as intended when we did it for this migration.

refs #280

***Update Jan 24:*** @greg and I have finished the migration of our PostgreSQL cluster from the host systems of Andromeda, Centaurus, and Draco, to new VMs on Centaurus and Draco. Instead of the previous encfs setup, we now use standard full-disk encryption (LUKS/LVM) for the VMs to encrypt data at rest. So the decryption password has to be provided during the booting of the VM now. The new setup also uses our Zerotier private network exclusively for database connections, and thus no public connections or TLS anymore. In order to make both this migration seamless, as well make the switching of the primary node to a hot standby node much easier and quicker in the future as well, we have changed all database clients to use a custom `/etc/hosts` entry (`pg.kosmos.local`). Then we created [a script to run all the steps for switching the primary at once](https://gitea.kosmos.org/kosmos/chef/src/commit/b1fea4b09ffe87c48b5b1d8b524cb0968c13fabf/scripts/postgresql/switch_primary.sh), both triggering the promotion of a standby node to primary, as well as stopping the old one, and updating the hosts entries without full Chef runs. This way, all clients will immediately re-connect to the new primary after promotion, which worked exactly as intended when we did it for this migration. refs #280

raucao self-assigned this 2021-01-08 08:14:10 +00:00

greg was assigned by raucao

2021-01-08 08:14:13 +00:00

raucao commented

2021-01-09 09:51:39 +00:00

@greg I was thinking about the most simple, low-tech, and stable solution for updating the postgres master address, and my last idea was:

Configure clients to use a hostname instead of IP address for the master connection (the obvious first step)
Write a script (maybe just postgres client Chef recipe), which simply updates an /etc/hosts entry on all postgres client VMs

This way, we don't need our own DNS server at all, and there's no extra connection or caching to think about. WDYT?

@greg I was thinking about the most simple, low-tech, and stable solution for updating the postgres master address, and my last idea was: * Configure clients to use a hostname instead of IP address for the master connection (the obvious first step) * Write a script (maybe just postgres client Chef recipe), which simply updates an `/etc/hosts` entry on all postgres client VMs This way, we don't need our own DNS server at all, and there's no extra connection or caching to think about. WDYT?

greg commented

2021-01-09 12:54:50 +00:00

Yeah that seems like a good solution for now!

raucao added 1 commit 2021-01-23 16:06:29 +00:00

Set up postgres-3 VM bd48dab1d8

greg added 9 commits 2021-01-23 16:12:01 +00:00

Remove postgres-1 server fdd70d1872

It was created with an encfs volume inside a VM, we want full disk encryption
instead. I have deleted the VM from centaurus as well as its disk and
have also unauthorized its id from the zerotier controller

Switch ejabberd, mastodon and gitea to a hostname for Postgres bb0e73d1b9

Add the postgresql_client role to the akkounts role 1e3a2b40d4

Move the postgresql_client role to the top 286cd2f2a3

We want the host file to be written before database connections happen

Generate a host entry for the current PostgreSQL primary ee7b3626be

Update the akkounts-1 node after Chef run 39d0304ab4

Update the ejabberd nodes after Chef run 40fde8a861

Update the mastodon-1 node after Chef run 4e1bd458f3

Remove the postgresql_replica from centaurus, run Chef 5cb390f340

I have also deleted the data directory (in /mnt/data)

greg added 1 commit 2021-01-23 16:35:27 +00:00

Replace the hardcoded primary pg IP with a hostname a4a35da0ff

greg added 1 commit 2021-01-23 18:09:13 +00:00

Switch the TLS mode to disabled for Gitea a7116b8fe5

We connect through a Zerotier private IP

greg added 1 commit 2021-01-23 18:10:05 +00:00

Promote a new PostgreSQL primary: postgres-2 7e61e9cb45

greg added 2 commits 2021-01-23 18:15:14 +00:00

Add a script to switch the primary PostgreSQL server 112eb903ec

Usage: scripts/postgresql/switch_primary.sh <old_primary_hostname> <new_primary_hostname>

Enable PostgreSQL service 7ce8b7d461

greg added 1 commit 2021-01-23 18:26:02 +00:00

Update the postgres-3 node after Chef run b1fea4b09f

raucao marked the pull request as ready for review 2021-01-23 18:30:06 +00:00

raucao added the kredits-3 label 2021-01-23 18:34:06 +00:00

raucao requested review from Ghost 2021-01-24 07:22:19 +00:00

greg added 1 commit 2021-01-24 09:11:53 +00:00

Generate the hosts entry for the PostgreSQL replica 1ad7a6936c

slvrbckt commented

2021-01-24 15:52:21 +00:00

I had a look, and it looks good to me, but I know absolutely zero about the ruby code, let alone the infrastructure. So take my approval for a grain of salt :)

EDIT: Also, I couldn't add my name to the reviewrs list

I had a look, and it looks good to me, but I know absolutely zero about the ruby code, let alone the infrastructure. So take my approval for a grain of salt :) EDIT: Also, I couldn't add my name to the reviewrs list

raucao commented

2021-01-24 16:32:17 +00:00

EDIT: Also, I couldn't add my name to the reviewrs list

If you leave a comment using the review function on the changed-files tab (green button on the right), then you'll be added to the list, and also kredited by @galfert's new script.

> EDIT: Also, I couldn't add my name to the reviewrs list If you leave a comment using the review function on the changed-files tab (green button on the right), then you'll be added to the list, and also kredited by @galfert's new script.

slvrbckt approved these changes 2021-01-24 16:47:39 +00:00

slvrbckt left a comment

LGTM!

raucao merged commit ad271e55d4 into master

2021-01-25 10:56:42 +00:00

raucao referenced this issue from a commit

2021-01-25 10:56:42 +00:00

Merge pull request 'Move PostgreSQL to VMs and access via Zerotier' (#282) from feature/postgres_vms into master

raucao deleted branch feature/postgres_vms

2021-01-25 10:56:48 +00:00

Sign in to join this conversation.

3 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/chef#282