Move PostgreSQL to VMs and access via Zerotier #282

raucao · 2021-01-07T14:14:24Z

raucao 已留言

2021-01-07 14:14:24 +00:00

擁有者

Update Jan 24:

@greg and I have finished the migration of our PostgreSQL cluster from the host systems of Andromeda, Centaurus, and Draco, to new VMs on Centaurus and Draco.

Instead of the previous encfs setup, we now use standard full-disk encryption (LUKS/LVM) for the VMs to encrypt data at rest. So the decryption password has to be provided during the booting of the VM now.

The new setup also uses our Zerotier private network exclusively for database connections, and thus no public connections or TLS anymore.

In order to make both this migration seamless, as well make the switching of the primary node to a hot standby node much easier and quicker in the future as well, we have changed all database clients to use a custom /etc/hosts entry (pg.kosmos.local).

Then we created a script to run all the steps for switching the primary at once, both triggering the promotion of a standby node to primary, as well as stopping the old one, and updating the hosts entries without full Chef runs. This way, all clients will immediately re-connect to the new primary after promotion, which worked exactly as intended when we did it for this migration.

refs #280

***Update Jan 24:*** @greg and I have finished the migration of our PostgreSQL cluster from the host systems of Andromeda, Centaurus, and Draco, to new VMs on Centaurus and Draco. Instead of the previous encfs setup, we now use standard full-disk encryption (LUKS/LVM) for the VMs to encrypt data at rest. So the decryption password has to be provided during the booting of the VM now. The new setup also uses our Zerotier private network exclusively for database connections, and thus no public connections or TLS anymore. In order to make both this migration seamless, as well make the switching of the primary node to a hot standby node much easier and quicker in the future as well, we have changed all database clients to use a custom `/etc/hosts` entry (`pg.kosmos.local`). Then we created [a script to run all the steps for switching the primary at once](https://gitea.kosmos.org/kosmos/chef/src/commit/b1fea4b09ffe87c48b5b1d8b524cb0968c13fabf/scripts/postgresql/switch_primary.sh), both triggering the promotion of a standby node to primary, as well as stopping the old one, and updating the hosts entries without full Chef runs. This way, all clients will immediately re-connect to the new primary after promotion, which worked exactly as intended when we did it for this migration. refs #280

raucao 指派給自己 2021-01-08 08:14:10 +00:00

greg 被 raucao 指派 2021-01-08 08:14:13 +00:00

raucao 已留言

2021-01-09 09:51:39 +00:00

作者

擁有者

@greg I was thinking about the most simple, low-tech, and stable solution for updating the postgres master address, and my last idea was:

Configure clients to use a hostname instead of IP address for the master connection (the obvious first step)
Write a script (maybe just postgres client Chef recipe), which simply updates an /etc/hosts entry on all postgres client VMs

This way, we don't need our own DNS server at all, and there's no extra connection or caching to think about. WDYT?

@greg I was thinking about the most simple, low-tech, and stable solution for updating the postgres master address, and my last idea was: * Configure clients to use a hostname instead of IP address for the master connection (the obvious first step) * Write a script (maybe just postgres client Chef recipe), which simply updates an `/etc/hosts` entry on all postgres client VMs This way, we don't need our own DNS server at all, and there's no extra connection or caching to think about. WDYT?

greg 已留言

2021-01-09 12:54:50 +00:00

擁有者

Yeah that seems like a good solution for now!

raucao 加入了 1 個提交 2021-01-23 16:06:29 +00:00

Set up postgres-3 VM bd48dab1d8

greg 加入了 9 個提交 2021-01-23 16:12:01 +00:00

Remove postgres-1 server fdd70d1872

It was created with an encfs volume inside a VM, we want full disk encryption
instead. I have deleted the VM from centaurus as well as its disk and
have also unauthorized its id from the zerotier controller

Switch ejabberd, mastodon and gitea to a hostname for Postgres bb0e73d1b9

Add the postgresql_client role to the akkounts role 1e3a2b40d4

Move the postgresql_client role to the top 286cd2f2a3

We want the host file to be written before database connections happen

Generate a host entry for the current PostgreSQL primary ee7b3626be

Update the akkounts-1 node after Chef run 39d0304ab4

Update the ejabberd nodes after Chef run 40fde8a861

Update the mastodon-1 node after Chef run 4e1bd458f3

Remove the postgresql_replica from centaurus, run Chef 5cb390f340

I have also deleted the data directory (in /mnt/data)

greg 加入了 1 個提交 2021-01-23 16:35:27 +00:00

Replace the hardcoded primary pg IP with a hostname a4a35da0ff

greg 加入了 1 個提交 2021-01-23 18:09:13 +00:00

Switch the TLS mode to disabled for Gitea a7116b8fe5

We connect through a Zerotier private IP

greg 加入了 1 個提交 2021-01-23 18:10:05 +00:00

Promote a new PostgreSQL primary: postgres-2 7e61e9cb45

greg 加入了 2 個提交 2021-01-23 18:15:14 +00:00

Add a script to switch the primary PostgreSQL server 112eb903ec

Usage: scripts/postgresql/switch_primary.sh <old_primary_hostname> <new_primary_hostname>

Enable PostgreSQL service 7ce8b7d461

greg 加入了 1 個提交 2021-01-23 18:26:02 +00:00

Update the postgres-3 node after Chef run b1fea4b09f

raucao 將標題從 ~~WIP: Move PostgreSQL to VMs and access via Zerotier~~ 改為 Move PostgreSQL to VMs and access via Zerotier

2021-01-23 18:30:06 +00:00

raucao 加入了

kredits-3

標籤 2021-01-23 18:34:06 +00:00

raucao 請求了 Owners 來審核 2021-01-24 07:22:19 +00:00

greg 加入了 1 個提交 2021-01-24 09:11:53 +00:00

Generate the hosts entry for the PostgreSQL replica 1ad7a6936c

slvrbckt 已留言

2021-01-24 15:52:21 +00:00

擁有者

I had a look, and it looks good to me, but I know absolutely zero about the ruby code, let alone the infrastructure. So take my approval for a grain of salt :)

EDIT: Also, I couldn't add my name to the reviewrs list

I had a look, and it looks good to me, but I know absolutely zero about the ruby code, let alone the infrastructure. So take my approval for a grain of salt :) EDIT: Also, I couldn't add my name to the reviewrs list

raucao 已留言

2021-01-24 16:32:17 +00:00

作者

擁有者

EDIT: Also, I couldn't add my name to the reviewrs list

If you leave a comment using the review function on the changed-files tab (green button on the right), then you'll be added to the list, and also kredited by @galfert's new script.

> EDIT: Also, I couldn't add my name to the reviewrs list If you leave a comment using the review function on the changed-files tab (green button on the right), then you'll be added to the list, and also kredited by @galfert's new script.

slvrbckt 核可了這些變更 2021-01-24 16:47:39 +00:00

slvrbckt 留下了回應

擁有者

LGTM!

raucao merged commit ad271e55d4 into master

2021-01-25 10:56:42 +00:00

raucao 在提交中關聯了這個問題

2021-01-25 10:56:42 +00:00

Merge pull request 'Move PostgreSQL to VMs and access via Zerotier' (#282) from feature/postgres_vms into master

raucao 刪除分支 feature/postgres_vms

2021-01-25 10:56:48 +00:00

登入才能加入這對話。

3 參與者

通知

截止日期

未設定截止日期。

先決條件

未設定先決條件。

參考: kosmos/chef#282