# Migrating PostgreSQL cluster to a new major version

## Summary

1. Dump from a replica
2. Restore to fresh VM running new major version
3. Add logical replication for delta sync from current/old primary
4. Switch primary to new server
5. Remove logical replication on new server

## Runbook

* Primary host: `PRIMARY_HOST`
* Replica host: `REPLICA_HOST`
* New PG14 host: `NEW_HOST`
* PostgreSQL superuser: `postgres`
* Running locally on each machine via `sudo -u postgres`

Adjust hostnames/IPs/etc. where needed.

---

### 🟢 0. PRIMARY — Pre-checks

```bash
sudo -u postgres psql -c "SHOW wal_level;"
sudo -u postgres psql -c "SHOW max_replication_slots;"
```

If needed, edit config:

```bash
sudo -u postgres vi $PGDATA/postgresql.conf
```

Ensure:

```conf
wal_level = logical
max_replication_slots = 10
```

Restart if changed:

```bash
sudo systemctl restart postgresql
```

---

### 🔵🟡 3. Create keypair for syncing dump later

🔵 On NEW_HOST:

```bash
sudo mkdir -p /home/postgres/.ssh && \
sudo chown -R postgres:postgres /home/postgres && \
sudo chmod 700 /home/postgres/.ssh && \
sudo -u postgres bash -c 'ssh-keygen -t ecdsa -b 256 -f /home/postgres/.ssh/id_ecdsa -N "" -C "postgres@$(hostname)"' && \
sudo cat /home/postgres/.ssh/id_ecdsa.pub
```

Copy the public key from the above output

🟡 On replica:

```bash
sudo mkdir -p /home/postgres/.ssh && \
sudo chown -R postgres:postgres /home/postgres && \
sudo chmod 700 /home/postgres/.ssh && \
echo [public_key] | sudo tee /home/postgres/.ssh/authorized_keys > /dev/null && \
sudo chmod 700 /home/postgres/.ssh
```

---

### 🟢 1. PRIMARY — Create publication and replication slots

```bash
sudo -u postgres pg_create_replication_publications
```

or

```bash
sudo -u postgres pg_create_replication_publication [db_name]
```

Listing publications and slots:

```bash
sudo -u postgres pg_list_replication_publications
sudo -u postgres pg_list_replication_slots
```

---

### 🟡 3. REPLICA — Pause replication

```bash
sudo -u postgres psql -c "SELECT pg_wal_replay_pause();"
```

Verify:

```bash
sudo -u postgres psql -c "SELECT pg_is_wal_replay_paused();"
```

---

### 🟡 4. REPLICA — Run dump

```bash
sudo -u postgres pg_dump_all_databases
```

or

```bash
sudo -u postgres bash -c "pg_dumpall --globals-only > /tmp/globals.sql"
sudo -u postgres pg_dump_database [db_name]
```

---

### 🟡 5. REPLICA — Resume replication

```bash
sudo -u postgres psql -c "SELECT pg_wal_replay_resume();"
```

---

### 🔵 6. COPY dumps to NEW HOST

From NEW_HOST:

```bash
export REPLICA_HOST=[private_ip] && \
cd /tmp && \
sudo -u postgres scp "postgres@$REPLICA_HOST:/tmp/globals.sql" . && \
sudo -u postgres scp "postgres@$REPLICA_HOST:/tmp/dump_*.tar.zst" .
```

---

### 🔵 7. NEW HOST (PostgreSQL 14) — Restore

#### 7.1 Restore globals

```bash
sudo -u postgres psql -f /tmp/globals.sql
```

---

#### 7.2 Create databases

```bash
sudo -u postgres psql -Atqc "SELECT datname FROM pg_database WHERE datallowconn AND datname NOT IN ('template1')" | \
xargs -I{} sudo -u postgres createdb {}
```

or

```bash
sudo -u postgres createdb [db_name]
```

---

#### 7.3 Restore each database

```bash
sudo -u postgres pg_restore_all_databases
```

or

```bash
sudo -u postgres pg_restore_database [db_name]
```

---

### 🔵 8. NEW HOST — Create subscriptions

```bash
sudo -u postgres pg_create_replication_subscriptions
```

or

```bash
sudo -u postgres pg_create_replication_subscription [db_name]
```

---

### 🔵 9. NEW HOST — Monitor replication

```bash
sudo -u postgres pg_list_replication_subscriptions
```

---

### 🔴 11. CUTOVER

#### 11.1 Stop writes on old primary

Put app(s) in maintenance mode, stop the app/daemons.

---

#### 11.2 Wait for replication to catch up

TODO: not the best way to check, since WAL LSNs keep increasing

```bash
sudo -u postgres psql -d [db_name] -c "SELECT * FROM pg_stat_subscription;"
```

---

#### 11.3 Fix sequences

Run per DB:

```bash
sudo -u postgres pg_fix_sequences_in_all_databases
```

or

```bash
sudo -u postgres pg_fix_sequences [db_name]
```

---

#### 11.4 Point app to NEW_HOST

1. Update `pg.kosmos.local` in `/etc/hosts` on app server(s). For example:

   ```bash
   export NEW_PG_PRIMARY=[private_ip]
   bundle exec knife ssh roles:ejabberd -a knife_zero.host "sudo sed -r \"s/^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\s(pg.kosmos.local)/$NEW_PG_PRIMARY\t\1/\" -i /etc/hosts"
   ```

   Or override node attribute(s) if necessary and/or approporiate.

2. Start the app/daemons, and deactivate maintenance mode.

---

### 🧹 12. CLEANUP NEW_HOST

```bash
sudo -u postgres pg_drop_replication_subscriptions
```

---

### 🧹 13. CLEANUP PRIMARY

TODO: Looks like slots are dropped automatically, when subscriptions are dropped

```bash
sudo -u postgres pg_drop_replication_publications
```

---

### 🧹 13. CLEANUP Chef

Once all apps/databases are migrated, update the role in the node
config of the new primary to 'postgres_primary' and converge it.

Also delete the old primary node config from the Chef repo.

---

### ✅ DONE

---