Move PostgreSQL to VMs and access via Zerotier #282

Merged
raucao merged 40 commits from feature/postgres_vms into master 2 years ago
raucao commented 2 years ago
Owner

Update Jan 24:

@greg and I have finished the migration of our PostgreSQL cluster from the host systems of Andromeda, Centaurus, and Draco, to new VMs on Centaurus and Draco.

Instead of the previous encfs setup, we now use standard full-disk encryption (LUKS/LVM) for the VMs to encrypt data at rest. So the decryption password has to be provided during the booting of the VM now.

The new setup also uses our Zerotier private network exclusively for database connections, and thus no public connections or TLS anymore.

In order to make both this migration seamless, as well make the switching of the primary node to a hot standby node much easier and quicker in the future as well, we have changed all database clients to use a custom /etc/hosts entry (pg.kosmos.local).

Then we created a script to run all the steps for switching the primary at once, both triggering the promotion of a standby node to primary, as well as stopping the old one, and updating the hosts entries without full Chef runs. This way, all clients will immediately re-connect to the new primary after promotion, which worked exactly as intended when we did it for this migration.

refs #280

***Update Jan 24:*** @greg and I have finished the migration of our PostgreSQL cluster from the host systems of Andromeda, Centaurus, and Draco, to new VMs on Centaurus and Draco. Instead of the previous encfs setup, we now use standard full-disk encryption (LUKS/LVM) for the VMs to encrypt data at rest. So the decryption password has to be provided during the booting of the VM now. The new setup also uses our Zerotier private network exclusively for database connections, and thus no public connections or TLS anymore. In order to make both this migration seamless, as well make the switching of the primary node to a hot standby node much easier and quicker in the future as well, we have changed all database clients to use a custom `/etc/hosts` entry (`pg.kosmos.local`). Then we created [a script to run all the steps for switching the primary at once](https://gitea.kosmos.org/kosmos/chef/src/commit/b1fea4b09ffe87c48b5b1d8b524cb0968c13fabf/scripts/postgresql/switch_primary.sh), both triggering the promotion of a standby node to primary, as well as stopping the old one, and updating the hosts entries without full Chef runs. This way, all clients will immediately re-connect to the new primary after promotion, which worked exactly as intended when we did it for this migration. refs #280
raucao self-assigned this 2 years ago
greg was assigned by raucao 2 years ago
Poster
Owner

@greg I was thinking about the most simple, low-tech, and stable solution for updating the postgres master address, and my last idea was:

  • Configure clients to use a hostname instead of IP address for the master connection (the obvious first step)
  • Write a script (maybe just postgres client Chef recipe), which simply updates an /etc/hosts entry on all postgres client VMs

This way, we don't need our own DNS server at all, and there's no extra connection or caching to think about. WDYT?

@greg I was thinking about the most simple, low-tech, and stable solution for updating the postgres master address, and my last idea was: * Configure clients to use a hostname instead of IP address for the master connection (the obvious first step) * Write a script (maybe just postgres client Chef recipe), which simply updates an `/etc/hosts` entry on all postgres client VMs This way, we don't need our own DNS server at all, and there's no extra connection or caching to think about. WDYT?
Owner

Yeah that seems like a good solution for now!

Yeah that seems like a good solution for now!
raucao added 1 commit 2 years ago
greg added 9 commits 2 years ago
greg added 1 commit 2 years ago
greg added 1 commit 2 years ago
a7116b8fe5 Switch the TLS mode to disabled for Gitea
greg added 1 commit 2 years ago
greg added 2 commits 2 years ago
112eb903ec Add a script to switch the primary PostgreSQL server
greg added 1 commit 2 years ago
raucao changed title from WIP: Move PostgreSQL to VMs and access via Zerotier to Move PostgreSQL to VMs and access via Zerotier 2 years ago
raucao added the
kredits-3
label 2 years ago
raucao requested review from Owners 2 years ago
greg added 1 commit 2 years ago
Owner

I had a look, and it looks good to me, but I know absolutely zero about the ruby code, let alone the infrastructure. So take my approval for a grain of salt :)

EDIT: Also, I couldn't add my name to the reviewrs list

I had a look, and it looks good to me, but I know absolutely zero about the ruby code, let alone the infrastructure. So take my approval for a grain of salt :) EDIT: Also, I couldn't add my name to the reviewrs list
Poster
Owner

EDIT: Also, I couldn't add my name to the reviewrs list

If you leave a comment using the review function on the changed-files tab (green button on the right), then you'll be added to the list, and also kredited by @galfert's new script.

> EDIT: Also, I couldn't add my name to the reviewrs list If you leave a comment using the review function on the changed-files tab (green button on the right), then you'll be added to the list, and also kredited by @galfert's new script.
slvrbckt approved these changes 2 years ago
slvrbckt left a comment
Owner

LGTM!

LGTM!
raucao merged commit ad271e55d4 into master 2 years ago
raucao deleted branch feature/postgres_vms 2 years ago

Reviewers

slvrbckt approved these changes 2 years ago
The pull request has been merged as ad271e55d4.
Sign in to join this conversation.
No reviewers
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/chef#282
Loading…
There is no content yet.