60 Commits

Author SHA1 Message Date
raucao f2ebda4a1a Update node configs 2026-07-04 16:30:15 +02:00
raucao 67f62ebd6c Scrape garage metrics 2026-07-04 16:14:36 +02:00
raucao 7dc4895da3 Remove old garage nodes 2026-07-04 15:51:39 +02:00
raucao 153b1e77c5 Add all nodes with base role as node exporter targets 2026-07-04 15:46:41 +02:00
raucao ea69c7cec6 Add prometheus node exporter to base role 2026-07-04 15:45:57 +02:00
raucao 5813a45987 Use base role instead of recipe in all runlists 2026-07-04 15:45:23 +02:00
raucao 63534e1cf5 Remove upstream prometheus cookbook, migrate to our own 2026-07-04 15:27:18 +02:00
raucao 2d835335b5 Re-add global config, change values 2026-07-04 14:15:55 +02:00
raucao e21797b402 Apply changed configs to prometheus and alertmanager 2026-07-04 14:15:38 +02:00
raucao 7396af5ca4 WIP Add node exporter 2026-07-04 14:15:33 +02:00
raucao df8c8d1742 Remove obsolete CLI option
Was left over from trying to overwrite the defaults before
2026-07-04 13:31:20 +02:00
Greg Karekinian 765d0b080e WIP Initial kosmos_prometheus wrapper cookbook 2026-07-03 17:47:13 +02:00
Greg Karekinian 4cd6c41254 Add community prometheus cookbook 2026-07-03 17:46:18 +02:00
raucao ec73dd5b57 Set Chef environment for node 2026-07-03 15:52:46 +02:00
raucao 850db344b7 Add prometheus node 2026-07-03 15:50:15 +02:00
raucao 99e8259696 Merge pull request 'Update Gitea to 1.26.4, Gitea Runner to 2.0.0' (#631) from chore/upgrade_gitea_and_runner into master
Reviewed-on: #631
2026-07-01 12:30:57 +00:00
Greg Karekinian 7810f9f373 Update Gitea to 1.26.4, Gitea Runner to 2.0.0
Two avatar configs were moved to the admin settings.
2026-07-01 14:28:31 +02:00
Greg Karekinian c167c1861f Update knife-zero 2026-07-01 14:11:09 +02:00
raucao 96bab62af1 Merge pull request 'Set up Blossom server on blossom.kosmos.org' (#630) from feature/blossom_server into master
Reviewed-on: #630
Reviewed-by: Greg <greg@kosmos.org>
2026-05-26 12:49:54 +00:00
raucao 2169e7904c Add LDAP support for blossom-server
Only available in our feature branch atm
2026-05-19 16:32:03 +02:00
raucao 5a4905aa97 Upgrade deno 2026-05-14 07:52:45 +02:00
raucao 21e31440a7 Update node config 2026-05-14 07:52:12 +02:00
raucao 894ae3f77e Add new garage node 2026-05-14 07:51:53 +02:00
raucao 8afe2ad05d Update blossom whitelist 2026-05-14 07:51:23 +02:00
raucao ef6e4e3319 Upgrade blossom-server 2026-05-14 07:50:48 +02:00
raucao f8ce544452 Set up Blossom server on blossom.kosmos.org 2026-04-18 16:37:00 +04:00
greg 36e9ea8a01 Merge pull request 'Upgrade .NET and BTCPay Server' (#629) from chore/upgrade_btcpay into master
Reviewed-on: #629
Reviewed-by: Greg <greg@kosmos.org>
2026-04-17 15:02:40 +00:00
raucao 2c2780a9f0 Update node info 2026-04-17 18:57:32 +04:00
raucao 6bcdd3f4d6 Upgrade .NET and BTCPay 2026-04-17 18:56:54 +04:00
raucao abc3f7a0cd Update node info 2026-04-17 15:47:10 +04:00
raucao 6d35c0a415 Update node info 2026-04-14 10:36:48 +04:00
raucao be0d7105d3 Merge pull request 'Migrate PostgreSQL cluster to PG14' (#625) from feature/postgresql_migration into master
Reviewed-on: #625
Reviewed-by: Greg <greg@kosmos.org>
2026-04-12 14:16:22 +00:00
raucao b24a6107d2 Merge branch 'master' into feature/postgresql_migration 2026-04-12 18:15:01 +04:00
raucao 1f7a1d0909 Remove commented lines 2026-04-12 16:11:40 +04:00
greg ba361ad09f Merge pull request 'Create new VMs with Ubuntu 24.04' (#628) from feature/ubuntu_noble into master
Reviewed-on: #628
Reviewed-by: Greg <greg@kosmos.org>
2026-04-12 11:32:49 +00:00
greg 94be0a3543 Merge pull request 'Upgrade rskj to v9.0.1 and Ubuntu 24.04' (#627) from chore/upgrade_rskj into master
Reviewed-on: #627
Reviewed-by: Greg <greg@kosmos.org>
2026-04-12 11:31:26 +00:00
raucao 29fb3ae9c9 Automatically determine OS package name
Co-authored-by: Greg Karékinian <greg@karekinian.com>
2026-04-12 10:32:53 +00:00
raucao 3a1c3e20b8 Add new RSK testnet VM 2026-04-12 10:32:53 +00:00
raucao d7782ba41e Upgrade rskj to v9.0.1 and Ubuntu 24.04 2026-04-12 10:32:53 +00:00
raucao a3be57afbc Fix default apt keyring dir not existing on older Ubuntu
Recent Chef client versions use it
2026-04-12 09:10:58 +04:00
raucao 22d459b558 Create new VMs with Ubuntu 24.04 2026-04-12 08:54:47 +04:00
raucao 5ed5af6d50 Use hardware clock sync on Ubuntu 24.04+ VMs 2026-04-12 08:53:50 +04:00
greg 9bf21e8317 Merge pull request 'Slow down Gitea 404s to mess with scrapers/bots' (#626) from chore/gitea_scraping into master
Reviewed-on: #626
Reviewed-by: Greg <greg@kosmos.org>
2026-04-11 17:08:16 +00:00
raucao aaed9a56d1 Slow down Gitea 404s to mess with scrapers/bots
Seems to have helped quite a lot for dealing with AI scrapers using
up all available server resources
2026-04-11 15:37:38 +04:00
raucao f0314e0b99 Update vendored cookbooks 2026-04-11 15:34:44 +04:00
raucao ac4fb0c9ca Fix Java/Homebrew cookbook resource failing with recent Chef
Required by our Mastodon cookbook
2026-04-11 15:23:30 +04:00
raucao d5e3d62522 Add new postgres cluster, remove old one 2026-04-11 15:22:34 +04:00
raucao 061880536b Fix akkounts systemd unit stop command
Typo (not using pumactl), but we don't need to specify it to do the
right thing anyway. systemd can just send sigterm on its own.
2026-04-11 14:56:15 +04:00
raucao 9de37cde96 Update doc 2026-04-11 14:55:32 +04:00
raucao 64d5d34d85 Update postgres roles
Reset to normal/final
2026-04-11 14:55:02 +04:00
raucao db9177c9c6 Improve RAM usage allowance
Queries can spawn more processed and then use more RAM than
`shared_buffers`
2026-04-11 14:53:28 +04:00
raucao c92f9157a5 Fix method undefined in heredoc 2026-04-11 14:52:53 +04:00
raucao a89db454d0 Improve postgres management scripts 2026-04-11 14:51:51 +04:00
raucao fddcd4899e Ignore default db for migration/management 2026-04-10 12:57:51 +04:00
raucao 8e11df4544 Update PG migration runbook 2026-04-10 12:49:28 +04:00
raucao 0020677ab2 Drone: Make database details configurable 2026-04-10 08:37:06 +04:00
raucao 09412f69e8 Move doc 2026-04-10 08:36:54 +04:00
raucao bc3f291bd2 WIP Prepare postgres for migration by replication 2026-04-10 08:25:55 +04:00
raucao 6583cd7010 Upgrade WAL config for PG14 2026-04-08 15:41:13 +04:00
raucao 290af8177a Refactor postgres server recipes/resource 2026-04-08 15:41:10 +04:00
113 changed files with 2120 additions and 733 deletions
+3
View File
@@ -10,3 +10,6 @@
[submodule "site-cookbooks/deno"]
path = site-cookbooks/deno
url = git@gitea.kosmos.org:kosmos/deno-cookbook.git
[submodule "site-cookbooks/blossom"]
path = site-cookbooks/blossom
url = git@gitea.kosmos.org:kosmos/blossom-cookbook.git
+1
View File
@@ -24,6 +24,7 @@ cookbook 'composer', '~> 2.7.0'
cookbook 'fail2ban', '~> 7.0.4'
cookbook 'git', '~> 10.0.0'
cookbook 'golang', '~> 5.3.1'
cookbook 'homebrew', '>= 6.0.0'
cookbook 'hostname', '= 0.4.2'
cookbook 'hostsfile', '~> 3.0.1'
cookbook 'java', '~> 4.3.0'
+2 -1
View File
@@ -8,6 +8,7 @@ DEPENDENCIES
firewall (~> 6.2.16)
git (~> 10.0.0)
golang (~> 5.3.1)
homebrew (>= 6.0.0)
hostname (= 0.4.2)
hostsfile (~> 3.0.1)
ipfs
@@ -62,7 +63,7 @@ GRAPH
git (10.0.0)
golang (5.3.1)
ark (>= 6.0)
homebrew (5.4.1)
homebrew (6.0.2)
hostname (0.4.2)
hostsfile (>= 0.0.0)
hostsfile (3.0.1)
+1 -1
View File
@@ -1,3 +1,3 @@
source 'https://rubygems.org'
gem 'knife-zero', '>= 2.4.2'
gem 'knife-zero', '~> 2.6.0'
+306 -182
View File
@@ -1,264 +1,399 @@
GEM
remote: https://rubygems.org/
specs:
addressable (2.8.0)
public_suffix (>= 2.0.2, < 5.0)
aws-eventstream (1.2.0)
aws-partitions (1.551.0)
aws-sdk-core (3.125.6)
aws-eventstream (~> 1, >= 1.0.2)
aws-partitions (~> 1, >= 1.525.0)
aws-sigv4 (~> 1.1)
jmespath (~> 1.0)
aws-sdk-kms (1.53.0)
aws-sdk-core (~> 3, >= 3.125.0)
aws-sigv4 (~> 1.1)
aws-sdk-s3 (1.111.3)
aws-sdk-core (~> 3, >= 3.125.0)
abbrev (0.1.2)
addressable (2.9.0)
public_suffix (>= 2.0.2, < 8.0)
ast (2.4.3)
aws-eventstream (1.4.0)
aws-partitions (1.1263.0)
aws-sdk-core (3.252.0)
aws-eventstream (~> 1, >= 1.3.0)
aws-partitions (~> 1, >= 1.992.0)
aws-sigv4 (~> 1.9)
base64
bigdecimal
jmespath (~> 1, >= 1.6.1)
logger
aws-sdk-kms (1.129.0)
aws-sdk-core (~> 3, >= 3.248.0)
aws-sigv4 (~> 1.5)
aws-sdk-s3 (1.226.0)
aws-sdk-core (~> 3, >= 3.248.0)
aws-sdk-kms (~> 1)
aws-sigv4 (~> 1.4)
aws-sdk-secretsmanager (1.56.0)
aws-sdk-core (~> 3, >= 3.125.0)
aws-sigv4 (~> 1.1)
aws-sigv4 (1.4.0)
aws-sigv4 (~> 1.5)
aws-sdk-secretsmanager (1.133.0)
aws-sdk-core (~> 3, >= 3.248.0)
aws-sigv4 (~> 1.5)
aws-sigv4 (1.12.1)
aws-eventstream (~> 1, >= 1.0.2)
bcrypt_pbkdf (1.1.0)
builder (3.2.4)
chef (17.9.42)
base64 (0.3.0)
bcrypt_pbkdf (1.1.2)
bcrypt_pbkdf (1.1.2-arm64-darwin)
bcrypt_pbkdf (1.1.2-x86_64-darwin)
benchmark (0.5.0)
bigdecimal (4.1.2)
builder (3.3.0)
chef (19.3.15)
addressable
aws-sdk-s3 (~> 1.91)
aws-sdk-secretsmanager (~> 1.46)
chef-config (= 17.9.42)
chef-utils (= 17.9.42)
bcrypt_pbkdf (~> 1.0)
chef-config (= 19.3.15)
chef-licensing (~> 1.3)
chef-utils (= 19.3.15)
chef-vault
chef-zero (>= 14.0.11)
chef-zero (~> 15.1.0)
corefoundation (~> 0.3.4)
diff-lcs (>= 1.2.4, < 1.4.0)
csv (~> 3.3.5)
diff-lcs (~> 1.6.0)
ed25519 (~> 1.2)
erubis (~> 2.7)
ffi (>= 1.5.0)
ffi (>= 1.15.5, < 1.18.0)
ffi-libarchive (~> 1.0, >= 1.0.3)
ffi-yajl (~> 2.2)
ffi-yajl (>= 2.2, < 4.0)
iniparse (~> 1.4)
inspec-core (~> 4.23)
inspec-core (~> 7.0.107)
license-acceptance (>= 1.0.5, < 3)
mixlib-archive (>= 0.4, < 2.0)
mixlib-authentication (>= 2.1, < 4)
mixlib-cli (>= 2.1.1, < 3.0)
mixlib-log (>= 2.0.3, < 4.0)
mixlib-shellout (>= 3.1.1, < 4.0)
net-sftp (>= 2.1.2, < 4.0)
ohai (~> 17.0)
mixlib-shellout (>= 3.3.8, < 3.5.0)
net-ftp
net-sftp (>= 2.1.2, < 5.0)
ohai (~> 19.0)
plist (~> 3.2)
proxifier (~> 1.0)
proxifier2 (~> 1.1)
syslog
syslog-logger (~> 1.6)
train-core (~> 3.2, >= 3.2.28)
train-winrm (>= 0.2.5)
uuidtools (>= 2.1.5, < 3.0)
vault (~> 0.16)
chef-config (17.9.42)
train-core (~> 3.13, >= 3.13.4)
train-rest (>= 0.4.1)
train-winrm (>= 0.2.17)
unf_ext (~> 0.0.9.1)
uri (>= 1.0.4, < 1.2.0)
vault (>= 0.18.2, < 0.21.0)
chef-config (19.3.15)
addressable
chef-utils (= 17.9.42)
chef-utils (= 19.3.15)
fuzzyurl
mixlib-config (>= 2.2.12, < 4.0)
mixlib-shellout (>= 2.0, < 4.0)
tomlrb (~> 1.2)
racc
tomlrb (>= 1.2, < 3.0)
chef-gyoku (1.5.0)
builder (>= 2.1.2)
rexml (~> 3.4)
chef-licensing (1.4.1)
chef-config (>= 15)
faraday (>= 1, < 3)
faraday-http-cache
mixlib-log (~> 3.0)
ostruct (~> 0.6.0)
pstore (~> 0.1.1)
tty-prompt (~> 0.23)
tty-spinner (~> 0.9.3)
chef-telemetry (1.1.1)
chef-config
concurrent-ruby (~> 1.0)
chef-utils (17.9.42)
chef-utils (19.3.15)
concurrent-ruby
chef-vault (4.1.5)
chef-zero (15.0.11)
ffi-yajl (~> 2.2)
hashie (>= 2.0, < 5.0)
chef-vault (4.2.12)
syslog (~> 0.3)
chef-winrm (2.5.0)
builder (>= 2.1.2)
chef-gyoku (~> 1.5)
erubi (~> 1.8)
gssapi (~> 1.2)
httpclient (~> 2.2, >= 2.2.0.2)
logging (>= 1.6.1, < 3.0)
nori (~> 2.7)
rexml (>= 3.4.2, < 4.0)
rubyntlm (~> 0.6.0, >= 0.6.3)
chef-winrm-elevated (1.2.5)
chef-winrm (>= 2.3.11)
chef-winrm-fs (>= 1.3.7)
erubi (~> 1.8)
chef-winrm-fs (1.4.2)
benchmark (~> 0.5.0)
chef-winrm (~> 2.4)
csv (~> 3.3)
erubi (>= 1.7)
logging (>= 1.6.1, < 3.0)
rubyzip (~> 2.0)
chef-zero (15.1.11)
ffi-yajl (>= 2.2, < 4.0)
hashie (>= 2.0, < 6.0)
mixlib-log (>= 2.0, < 4.0)
rack (~> 2.0, >= 2.0.6)
uuidtools (~> 2.1)
rack (~> 3.2, >= 3.2.6)
rackup (~> 2.3, >= 2.3.1)
uuidtools (>= 2.1, < 4.0)
webrick
coderay (1.1.3)
concurrent-ruby (1.1.9)
corefoundation (0.3.10)
concurrent-ruby (1.3.7)
connection_pool (2.5.5)
cookstyle (8.7.6)
rubocop (= 1.86.1)
corefoundation (0.3.19)
ffi (>= 1.15.0)
diff-lcs (1.3)
erubi (1.10.0)
csv (3.3.5)
date (3.5.1)
diff-lcs (1.6.2)
domain_name (0.6.20240107)
ed25519 (1.4.0)
erubi (1.13.1)
erubis (2.7.0)
faraday (1.4.3)
faraday-em_http (~> 1.0)
faraday-em_synchrony (~> 1.0)
faraday-excon (~> 1.1)
faraday-net_http (~> 1.0)
faraday-net_http_persistent (~> 1.1)
multipart-post (>= 1.2, < 3)
ruby2_keywords (>= 0.0.4)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday_middleware (1.2.0)
faraday (~> 1.0)
ffi (1.15.5)
ffi-libarchive (1.1.3)
faraday (2.14.3)
faraday-net_http (>= 2.0, < 3.5)
json
logger
faraday-follow_redirects (0.5.0)
faraday (>= 1, < 3)
faraday-http-cache (2.5.1)
faraday (>= 0.8)
faraday-net_http (3.4.4)
net-http (~> 0.5)
ffi (1.17.4-arm64-darwin)
ffi (1.17.4-x86_64-darwin)
ffi (1.17.4-x86_64-linux-gnu)
ffi-libarchive (1.1.14)
ffi (~> 1.0)
ffi-yajl (2.4.0)
libyajl2 (>= 1.2)
ffi-yajl (2.7.11)
libyajl2 (>= 2.1)
fuzzyurl (0.9.0)
gssapi (1.3.1)
ffi (>= 1.0.1)
gyoku (1.3.1)
builder (>= 2.1.2)
hashie (4.1.0)
highline (2.0.3)
httpclient (2.8.3)
hashie (5.1.0)
logger
highline (3.1.2)
reline
http-accept (1.7.0)
http-cookie (1.1.6)
domain_name (~> 0.5)
httpclient (2.9.0)
mutex_m
iniparse (1.5.0)
inspec-core (4.52.9)
inspec-core (7.0.107)
addressable (~> 2.4)
chef-licensing (>= 1.2.0)
chef-telemetry (~> 1.0, >= 1.0.8)
faraday (>= 0.9.0, < 1.5)
faraday_middleware (~> 1.0)
hashie (>= 3.4, < 5.0)
cookstyle
csv (~> 3.0)
faraday (>= 1, < 3)
faraday-follow_redirects (~> 0.3)
hashie (>= 3.4, < 6.0)
license-acceptance (>= 0.2.13, < 3.0)
method_source (>= 0.8, < 2.0)
mixlib-log (~> 3.0)
multipart-post (~> 2.0)
ostruct (>= 0.1, < 0.7)
parallel (~> 1.9)
parslet (>= 1.5, < 2.0)
parslet (>= 1.5, < 3.0)
pry (~> 0.13)
rspec (>= 3.9, < 3.11)
rspec-its (~> 1.2)
rubyzip (>= 1.2.2, < 3.0)
rspec (>= 3.9, <= 3.14)
rspec-its (>= 1.2, < 3.0)
rubyzip (>= 1.2.2, < 4.0)
semverse (~> 3.0)
sslshake (~> 1.2)
thor (>= 0.20, < 2.0)
tomlrb (>= 1.2, < 2.1)
train-core (~> 3.0)
syslog (~> 0.1)
thor (>= 0.20, < 1.5.0)
tomlrb (>= 1.3, < 2.1)
train-core (~> 3.16, >= 3.16.1)
tty-prompt (~> 0.17)
tty-table (~> 0.10)
io-console (0.8.2)
ipaddress (0.8.3)
jmespath (1.5.0)
json (2.6.1)
knife (17.9.26)
jmespath (1.6.2)
json (2.20.0)
knife (19.0.134)
abbrev
bcrypt_pbkdf (~> 1.1)
chef (>= 17)
chef-config (>= 17)
chef-utils (>= 17)
chef-licensing (~> 1.2)
chef-vault
ed25519 (>= 1.2, < 2.0)
erubis (~> 2.7)
ffi (>= 1.15)
ffi-yajl (~> 2.2)
highline (>= 1.6.9, < 3)
ffi (>= 1.15, < 1.18.0)
ffi-yajl (>= 2.2, < 3.0)
highline (>= 1.6.9, < 4)
license-acceptance (>= 1.0.5, < 3)
mixlib-archive (>= 0.4, < 2.0)
mixlib-cli (>= 2.1.1, < 3.0)
net-ssh (>= 5.1, < 7)
net-ssh (>= 5.1, < 8)
net-ssh-multi (~> 1.2, >= 1.2.1)
ohai (~> 17.0)
pastel
train-core (~> 3.2, >= 3.2.28)
train-winrm (>= 0.2.5)
proxifier2 (~> 1.1)
train-core (~> 3.13, >= 3.13.4)
train-winrm (>= 0.2.17)
tty-prompt (~> 0.21)
tty-screen (~> 0.6)
tty-table (~> 0.11)
knife-zero (2.4.2)
chef (>= 15.0)
knife-zero (2.6.0)
chef (>= 16.6)
knife (>= 17.0)
language_server-protocol (3.17.0.5)
libyajl2 (2.1.0)
license-acceptance (2.1.13)
pastel (~> 0.7)
tomlrb (>= 1.2, < 3.0)
tty-box (~> 0.6)
tty-prompt (~> 0.20)
lint_roller (1.1.0)
little-plugger (1.1.4)
logging (2.3.0)
logger (1.7.0)
logging (2.4.0)
little-plugger (~> 1.1)
multi_json (~> 1.14)
method_source (1.0.0)
mixlib-archive (1.1.7)
method_source (1.1.0)
mime-types (3.7.0)
logger
mime-types-data (~> 3.2025, >= 3.2025.0507)
mime-types-data (3.2026.0701)
mixlib-archive (1.3.3)
mixlib-log
mixlib-authentication (3.0.10)
mixlib-cli (2.1.8)
mixlib-config (3.0.9)
mixlib-config (3.0.27)
tomlrb
mixlib-log (3.0.9)
mixlib-shellout (3.2.5)
mixlib-log (3.2.3)
ffi (>= 1.15.5)
mixlib-shellout (3.4.10)
chef-utils
multi_json (1.15.0)
multipart-post (2.1.1)
net-scp (3.0.0)
net-ssh (>= 2.6.5, < 7.0.0)
net-sftp (3.0.0)
net-ssh (>= 5.0.0, < 7.0.0)
net-ssh (6.1.0)
multi_json (1.19.1)
multipart-post (2.4.1)
mutex_m (0.3.0)
net-ftp (0.3.9)
net-protocol
time
net-http (0.9.1)
uri (>= 0.11.1)
net-http-persistent (4.0.8)
connection_pool (>= 2.2.4, < 4)
net-protocol (0.2.2)
timeout
net-scp (4.1.0)
net-ssh (>= 2.6.5, < 8.0.0)
net-sftp (4.0.0)
net-ssh (>= 5.0.0, < 8.0.0)
net-ssh (7.3.3)
net-ssh-gateway (2.0.0)
net-ssh (>= 4.0.0)
net-ssh-multi (1.2.1)
net-ssh (>= 2.6.5)
net-ssh-gateway (>= 1.2.0)
nori (2.6.0)
ohai (17.9.0)
chef-config (>= 14.12, < 18)
chef-utils (>= 16.0, < 18)
ffi (~> 1.9)
ffi-yajl (~> 2.2)
netrc (0.11.0)
nori (2.7.1)
bigdecimal
ohai (19.1.40)
base64
chef-config (>= 14.12, < 20)
chef-utils (>= 16.0, < 20)
ffi (>= 1.15.5)
ffi-yajl (>= 2.2, < 3.0)
ipaddress
mixlib-cli (>= 1.7.0)
mixlib-config (>= 2.0, < 4.0)
mixlib-log (>= 2.0.1, < 4.0)
mixlib-shellout (~> 3.2, >= 3.2.5)
mixlib-shellout (>= 3.3.6, < 3.5.0)
plist (~> 3.1)
train-core
wmi-lite (~> 1.0)
parallel (1.21.0)
parslet (1.8.2)
ostruct (0.6.3)
parallel (1.28.0)
parser (3.3.11.1)
ast (~> 2.4.1)
racc
parslet (2.0.0)
pastel (0.8.0)
tty-color (~> 0.5)
plist (3.6.0)
proxifier (1.0.3)
pry (0.14.1)
plist (3.7.2)
prism (1.9.0)
proxifier2 (1.1.0)
pry (0.16.0)
coderay (~> 1.1)
method_source (~> 1.0)
public_suffix (4.0.6)
rack (2.2.3)
rspec (3.10.0)
rspec-core (~> 3.10.0)
rspec-expectations (~> 3.10.0)
rspec-mocks (~> 3.10.0)
rspec-core (3.10.2)
rspec-support (~> 3.10.0)
rspec-expectations (3.10.2)
reline (>= 0.6.0)
pstore (0.1.4)
public_suffix (6.0.2)
racc (1.8.1)
rack (3.2.6)
rackup (2.3.1)
rack (>= 3)
rainbow (3.1.1)
regexp_parser (2.12.0)
reline (0.6.3)
io-console (~> 0.5)
rest-client (2.1.0)
http-accept (>= 1.7.0, < 2.0)
http-cookie (>= 1.0.2, < 2.0)
mime-types (>= 1.16, < 4.0)
netrc (~> 0.8)
rexml (3.4.4)
rspec (3.13.2)
rspec-core (~> 3.13.0)
rspec-expectations (~> 3.13.0)
rspec-mocks (~> 3.13.0)
rspec-core (3.13.6)
rspec-support (~> 3.13.0)
rspec-expectations (3.13.5)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.10.0)
rspec-its (1.3.0)
rspec-core (>= 3.0.0)
rspec-expectations (>= 3.0.0)
rspec-mocks (3.10.3)
rspec-support (~> 3.13.0)
rspec-its (2.0.0)
rspec-core (>= 3.13.0)
rspec-expectations (>= 3.13.0)
rspec-mocks (3.13.8)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.10.0)
rspec-support (3.10.3)
ruby2_keywords (0.0.5)
rubyntlm (0.6.3)
rubyzip (2.3.2)
semverse (3.0.0)
rspec-support (~> 3.13.0)
rspec-support (3.13.7)
rubocop (1.86.1)
json (~> 2.3)
language_server-protocol (~> 3.17.0.2)
lint_roller (~> 1.1.0)
parallel (>= 1.10)
parser (>= 3.3.0.2)
rainbow (>= 2.2.2, < 4.0)
regexp_parser (>= 2.9.3, < 3.0)
rubocop-ast (>= 1.49.0, < 2.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 2.4.0, < 4.0)
rubocop-ast (1.49.1)
parser (>= 3.3.7.2)
prism (~> 1.7)
ruby-progressbar (1.13.0)
rubyntlm (0.6.5)
base64
rubyzip (2.4.1)
semverse (3.0.2)
socksify (1.8.1)
sslshake (1.3.1)
strings (0.2.1)
strings-ansi (~> 0.2)
unicode-display_width (>= 1.5, < 3.0)
unicode_utils (~> 1.4)
strings-ansi (0.2.0)
syslog (0.4.0)
logger
syslog-logger (1.6.8)
thor (1.2.1)
tomlrb (1.3.0)
train-core (3.8.7)
thor (1.4.0)
time (0.4.2)
date
timeout (0.6.1)
tomlrb (2.0.4)
train-core (3.16.5)
addressable (~> 2.5)
ffi (!= 1.13.0)
json (>= 1.8, < 3.0)
ffi (>= 1.16.0, < 1.18)
json (>= 2.19.2, < 3.0)
mixlib-shellout (>= 2.0, < 4.0)
net-scp (>= 1.2, < 4.0)
net-ssh (>= 2.9, < 7.0)
train-winrm (0.2.12)
winrm (>= 2.3.6, < 3.0)
winrm-elevated (~> 1.2.2)
winrm-fs (~> 1.0)
net-scp (>= 1.2, < 5.0)
net-ssh (>= 2.9, < 8.0)
train-rest (0.5.0)
aws-sigv4 (~> 1.5)
rest-client (~> 2.1)
train-core (~> 3.0)
train-winrm (0.4.3)
chef-winrm (>= 2.4.4, < 3.0)
chef-winrm-elevated (>= 1.2.5, < 2.0)
chef-winrm-fs (>= 1.4.1, < 2.0)
socksify (~> 1.8)
tty-box (0.7.0)
pastel (~> 0.8)
strings (~> 0.2.0)
@@ -272,45 +407,34 @@ GEM
tty-cursor (~> 0.7)
tty-screen (~> 0.8)
wisper (~> 2.0)
tty-screen (0.8.1)
tty-screen (0.8.2)
tty-spinner (0.9.3)
tty-cursor (~> 0.7)
tty-table (0.12.0)
pastel (~> 0.8)
strings (~> 0.2.0)
tty-screen (~> 0.8)
unicode-display_width (2.1.0)
unf_ext (0.0.9.1)
unicode-display_width (2.6.0)
unicode_utils (1.4.0)
uuidtools (2.2.0)
vault (0.16.0)
uri (1.1.1)
uuidtools (3.0.0)
vault (0.20.1)
aws-sigv4
webrick (1.7.0)
winrm (2.3.6)
builder (>= 2.1.2)
erubi (~> 1.8)
gssapi (~> 1.2)
gyoku (~> 1.0)
httpclient (~> 2.2, >= 2.2.0.2)
logging (>= 1.6.1, < 3.0)
nori (~> 2.0)
rubyntlm (~> 0.6.0, >= 0.6.3)
winrm-elevated (1.2.3)
erubi (~> 1.8)
winrm (~> 2.0)
winrm-fs (~> 1.0)
winrm-fs (1.3.5)
erubi (~> 1.8)
logging (>= 1.6.1, < 3.0)
rubyzip (~> 2.0)
winrm (~> 2.0)
base64
net-http-persistent (~> 4.0, >= 4.0.2)
webrick (1.9.2)
wisper (2.0.1)
wmi-lite (1.0.5)
wmi-lite (1.0.7)
PLATFORMS
arm64-darwin-22
x86_64-darwin-18
x86_64-darwin-19
x86_64-linux
DEPENDENCIES
knife-zero (>= 2.4.2)
knife-zero (~> 2.6.0)
BUNDLED WITH
2.2.15
-4
View File
@@ -1,4 +0,0 @@
{
"name": "garage-10",
"public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAw2+3Wo+KkXVJCOX1SxT9\nSdwKXgPbCDM3EI9uwoxhMxQfRyN53dxIsBDsQUVOIe1Z8yqm4FenMQlNmeDR+QLE\nvNFf1fisinW+D9VVRm+CjcJy96i/Dyt786Z6YRrDlB860HxCbfTL2Zv5BRtbyIKg\nhz5gO+9PMEpPVR2ij9iue4K6jbM1AAL2ia/P6zDWLJqeIzUocCeHV5N0Z3jXH6qr\nf444v78x35MMJ+3tg5h95SU1/PDCpdSTct4uHEuKIosiN7p4DlYMoM5iSyvVoujr\nflRQPEpGzS9qEt3rDo/F4ltzYMx6bf1tB/0QaBKD+zwPZWTTwf61tSBo5/NkGvJc\nFQIDAQAB\n-----END PUBLIC KEY-----\n"
}
-4
View File
@@ -1,4 +0,0 @@
{
"name": "garage-12",
"public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA9GtHHi298BjiIqpZ3WkT\nkYAPfWD60hFe/8icYcq/F/6cHLYKZQ4chek9X/hDCMq4tHEN6Oh58T5x/nuNdPrK\nIAMGyVAGk6ekWlmD4jwdEf6TGb/J3ffJTRDvwX/I8xD/DW3wtXsN+X24T59ByGTm\nrnwRmmmwHF3otRx9wnCsIgDQ0AjiUujsfNNv1FcLXD/WJLys9lEeU5aJ4XtHTwDv\ntJM8YyVEFhEnuvgdKmzn5+F5k9VGdUwForlFOBfvzbCnTZMDMmDVeiUtAUv/7xWQ\nQl2mLUGCtgWuYJYXsQacAJ6pa3h+7cQyshC6w3dwUG+1fS9lNO0Yp1GGX1AGYKpp\nPQIDAQAB\n-----END PUBLIC KEY-----\n"
}
-4
View File
@@ -1,4 +0,0 @@
{
"name": "garage-13",
"public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvbqWc6OwRxgHfsQuTNL4\naxeVvNen5d9srYpZSHjuBB/k9NHB+9P6vU5qF37XHkw1lVUGeYbPHzhYsx3O0/kZ\nH5f4+4SMy/P9jc6SE7AJF4qtYKgJ88koZdqCww07c6K9g+BnEGFFZui/h3hUBxWj\nTfhBHEWPyQ2bl/lr9sIJwsEz+EN0isGn/eIXkmw9J6LdLJ5Q0LLks33K28FNOU7q\nfeAN4MiBVMUtgCGyT2Voe6WrOXwQLSDXQONOp3sfSfFExsIJ1s24xdd7AMD7/9a7\n4sFDZ4swhqAWgWmW2giR7Kb8wTvGQLO/O/uUbmKz3DZXgkOKXHdHCEB/PZx1mRNM\nEwIDAQAB\n-----END PUBLIC KEY-----\n"
}
+4
View File
@@ -0,0 +1,4 @@
{
"name": "garage-16",
"public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqYCUN4WNP05pjbxVJd1/\nvmDZU6XRrVVZS5piSSRzs+uxrcUQZew9oe8YwnB9Acf7nl1igS1Fa3e3TyVdrdrM\n+zKLIszJfZw8qH0BhUHYhIm5o+NXeUDR6zor5/4msq9yyXxMFM3FY6HSnz1IBY6P\n43GdArRqAhccGZTBzvowHZvVTkG553oYF1ETxlUWn+9l142YZufNK+B2JGUGSnf3\nS0A2vlQi7DGHBcUaPPqCLeaXB1cQ4Q6SqMvnExpi0xTdY2QXLlSIBJvJVowLtQKs\npS5qHxwCabMY/uHVoEKxgmMcGvjp4L0PoaXRcev5I2sDfbLuz1VxYfatjgI/1psg\ntwIDAQAB\n-----END PUBLIC KEY-----\n"
}
-4
View File
@@ -1,4 +0,0 @@
{
"name": "garage-9",
"public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnMHzKE8JBrsQkmRDeMjX\n71mBzvRzNM90cwA8xtvIkXesdTyGqohX9k/PJbCY5ySGK9PpMaYDPVAnwnUP8LFQ\n3G98aSbLxUjqU/PBzRsnWpihehr05uz9zYcNFzr4LTNvGQZsq47nN9Tk+LG3zHP7\nAZViv2mJ4ZRnukXf6KHlyoVvhuTu+tiBM8QzjTF97iP/aguNPzYHmrecy9Uf5bSA\nZrbNZT+ayxtgswC2OclhRucx7XLSuHXtpwFqsQzSAhiX1aQ3wwCyH9WJtVwpfUsE\nlxTjcQiSM9aPZ8iSC0shpBaKD1j3iF/2K2Jk+88++zMhJJPLermvaJxzsdePgvyk\nKQIDAQAB\n-----END PUBLIC KEY-----\n"
}
+4
View File
@@ -0,0 +1,4 @@
{
"name": "postgres-11",
"public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1foYpuubS2ovlg3uHO12\nQ/ROZ8MpG+LkCAM46uVfPaoWwfY0vdfMsBOanHDgm9DGUCEBJZ6LPrvCvGXbpPy6\n9GSswK75zVWODblNjvvV4ueGFq4bBFwRuZNjyMlqgyzeU+srZL0ivelu5XEuGuoD\nPYCBKWYqGMz85/eMC7/tinTJtKPyOtXe/G8meji+r7gh3j+ypj/EWeKfcRDa4aGe\n/DmMCurIjjPAXFLMAA6fIqPWVfcPw4APNPE60Z92yPGsTbPu7bL54M5f7udmmu7H\nOgk1HjMAmXCuLDzTkfaxqHP+57yELg/YpXR1E93VmBeQuIBsyOFEk6AmUmA1Ib6e\nnQIDAQAB\n-----END PUBLIC KEY-----\n"
}
+4
View File
@@ -0,0 +1,4 @@
{
"name": "postgres-12",
"public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1mYGrYB8keUKmXA8dhWc\ncCLzp50xR0ajSw+bWYydyRqD5wuEVKjiJu4+G9QmTVXkVgJ+AYI0Y9/WZYpDqVH6\nvLUo6BSNQaWx20q93qIdOGLy8YG3Qyznezk4l8T9u9vWZDyDpKw6gCxzikMkrXxb\n0cqOYtyud8+PtSEEMogSjOKhRURVHlVrlVH3SQO7Whke9rkiFcbXzubsK9yjkUtF\nxZafSoGorOlDsPvFTfYnkepVB+GHcgiribRYSrO+73GypC2kqMhCpWrb6a0VWsP/\nh53+q3JL3vBvdvjcv51Wpf4n6JdnXnQGn2/MdXEzw+NXgjU4/IdYtbORSbaI8F5t\nowIDAQAB\n-----END PUBLIC KEY-----\n"
}
+4
View File
@@ -0,0 +1,4 @@
{
"name": "prometheus-1",
"public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAp7T/OBo/TZm3YqnN4+ok\nHwcJ0kW9w2rl9UfrOlWUvoPHBd2LrqpEv3Az3a150IylQ1H/UozmQA7DtjIoTA7d\nV3oLY970vYrYiURcojOo8qAZBy8EH7dfAHxuZryUeELr+3vdcHF5WrrfSt2FdFVX\nPTY95ikafAnOO0Nt8jvnlPoDn7REV8TOE6KOiUzcHKa2xGlfaIe0oRC21LD86uQm\nR09xY1YaJkVgZfeN/opoRjZawkU3FFs3jlUEVBF8k153oOw9W3bgsFFjSOtRtRRg\nDwyQ7oDeMH83kXnaCdpkNZd59wjzPcpxYAL4LRN52ZXA4Btr4DTi+GxHz98Dr0kU\nUQIDAQAB\n-----END PUBLIC KEY-----\n"
}
@@ -3,3 +3,5 @@ config:
line-length: false # MD013
no-duplicate-heading: false # MD024
reference-links-images: false # MD052
ignores:
- .github/copilot-instructions.md
+42
View File
@@ -2,6 +2,48 @@
This file is used to list changes made in each version of the homebrew cookbook.
## 6.0.2 - *2025-09-04*
Standardise files with files in sous-chefs/repo-management
Standardise files with files in sous-chefs/repo-management
## 6.0.1 - *2025-03-24*
## 6.0.0 - *2025-03-17*
- Updated library call for new homebrew class name found in chef-client 18.6.2+ releases
## 5.4.9 - *2024-11-18*
Standardise files with files in sous-chefs/repo-management
Standardise files with files in sous-chefs/repo-management
Standardise files with files in sous-chefs/repo-management
Standardise files with files in sous-chefs/repo-management
Standardise files with files in sous-chefs/repo-management
## 5.4.8 - *2024-05-07*
## 5.4.7 - *2024-05-06*
- Explicitly include `Which` module from `Chef` which fixes runs on 18.x clients.
## 5.4.6 - *2024-05-06*
## 5.4.5 - *2023-11-01*
Standardise files with files in sous-chefs/repo-management
## 5.4.4 - *2023-09-28*
## 5.4.3 - *2023-09-04*
## 5.4.2 - *2023-07-10*
## 5.4.1 - *2023-06-01*
## 5.4.0 - *2023-04-24*
+8 -31
View File
@@ -20,8 +20,9 @@
#
class HomebrewUserWrapper
require 'chef/mixin/homebrew_user'
include Chef::Mixin::HomebrewUser
require 'chef/mixin/homebrew'
include Chef::Mixin::Homebrew
include Chef::Mixin::Which
end
module Homebrew
@@ -59,41 +60,17 @@ module Homebrew
def owner
@owner ||= begin
# once we only support 14.0 we can switch this to find_homebrew_username
require 'etc'
::Etc.getpwuid(HomebrewUserWrapper.new.find_homebrew_uid).name
rescue Chef::Exceptions::CannotDetermineHomebrewOwner
calculate_owner
HomebrewUserWrapper.new.find_homebrew_username
rescue
Chef::Exceptions::CannotDetermineHomebrewPath
end.tap do |owner|
Chef::Log.debug("Homebrew owner is #{owner}")
end
end
private
def calculate_owner
owner = homebrew_owner_attr || sudo_user || current_user
if owner == 'root'
raise Chef::Exceptions::User,
"Homebrew owner is 'root' which is not supported. " \
"To set an explicit owner, please set node['homebrew']['owner']."
end
owner
end
def homebrew_owner_attr
Chef.node['homebrew']['owner']
end
def sudo_user
ENV['SUDO_USER']
end
def current_user
ENV['USER']
end
end unless defined?(Homebrew)
class HomebrewWrapper
include Homebrew
end
Chef::Mixin::Homebrew.include(Homebrew)
+2 -2
View File
@@ -17,13 +17,13 @@
"recipes": {
},
"version": "5.4.1",
"version": "6.0.2",
"source_url": "https://github.com/sous-chefs/homebrew",
"issues_url": "https://github.com/sous-chefs/homebrew/issues",
"privacy": false,
"chef_versions": [
[
">= 15.3"
">= 18.6.2"
]
],
"ohai_versions": [
+2 -2
View File
@@ -3,9 +3,9 @@ maintainer 'Sous Chefs'
maintainer_email 'help@sous-chefs.org'
license 'Apache-2.0'
description 'Install Homebrew and includes resources for working with taps and casks'
version '5.4.1'
version '6.0.2'
supports 'mac_os_x'
source_url 'https://github.com/sous-chefs/homebrew'
issues_url 'https://github.com/sous-chefs/homebrew/issues'
chef_version '>= 15.3'
chef_version '>= 18.6.2'
+3 -2
View File
@@ -1,9 +1,10 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:base"],
"packageRules": [{
"packageRules": [
{
"groupName": "Actions",
"matchUpdateTypes": ["patch", "pin", "digest"],
"matchUpdateTypes": ["minor", "patch", "pin"],
"automerge": true,
"addLabels": ["Release: Patch", "Skip: Announcements"]
},
+1
View File
@@ -19,6 +19,7 @@
# limitations under the License.
#
unified_mode true
chef_version_for_provides '< 14.0' if respond_to?(:chef_version_for_provides)
property :cask_name, String, regex: %r{^[\w/-]+$}, name_property: true
+1
View File
@@ -19,6 +19,7 @@
# limitations under the License.
#
unified_mode true
chef_version_for_provides '< 14.0' if respond_to?(:chef_version_for_provides)
property :tap_name, String, name_property: true, regex: %r{^[\w-]+(?:\/[\w-]+)+$}
+24
View File
@@ -0,0 +1,24 @@
{
"id": "blossom",
"admin_password": {
"encrypted_data": "Gd6AzFmySL0p+xo1PnRn9p4Fwge1m3CQj+NRLIUD8P9u1C8=\n",
"iv": "l6KVzF9xEEBRRAmh\n",
"auth_tag": "P791KMh9TxuHiWJpDKxWQA==\n",
"version": 3,
"cipher": "aes-256-gcm"
},
"s3_access_key": {
"encrypted_data": "S8jB2LDQOxI/p5ugggW1Sk50TS9TJe9sLv04O/VD9/v22SSM7J6ETomTA+Hd\n",
"iv": "dUIIZbdAT9q72ioX\n",
"auth_tag": "+5fCNOuTE/+FqdV6rDNbkw==\n",
"version": 3,
"cipher": "aes-256-gcm"
},
"s3_secret_key": {
"encrypted_data": "soT63l2frBJDNmHetXmEPvNYBsTpvTyR95FA2rxuZXvVE7hMj21La8/0Amk7\nv+mHOBUMaGG9BTLN0tVFkL0+lGPXdZJTbtDHgluk5l6lLPyc8KY=\n",
"iv": "RuXs2pL9C/wpwJ/w\n",
"auth_tag": "nu7dE2udTkxaUZCR42h09w==\n",
"version": 3,
"cipher": "aes-256-gcm"
}
}
+287
View File
@@ -0,0 +1,287 @@
# Migrating PostgreSQL cluster to a new major version
## Summary
1. Dump from a replica
2. Restore to fresh VM running new major version
3. Add logical replication for delta sync from current/old primary
4. Switch primary to new server
5. Remove logical replication on new server
## Runbook
* Primary host: `PRIMARY_HOST`
* Replica host: `REPLICA_HOST`
* New PG14 host: `NEW_HOST`
* PostgreSQL superuser: `postgres`
* Running locally on each machine via `sudo -u postgres`
Adjust hostnames/IPs/etc. where needed.
---
### 🟢 0. PRIMARY — Pre-checks
```bash
sudo -u postgres psql -c "SHOW wal_level;"
sudo -u postgres psql -c "SHOW max_replication_slots;"
```
If needed, edit config:
```bash
sudo -u postgres vi $PGDATA/postgresql.conf
```
Ensure:
```conf
wal_level = logical
max_replication_slots = 10
```
Restart if changed:
```bash
sudo systemctl restart postgresql
```
---
### 🔵🟡 3. Create keypair for syncing dump later
🔵 On NEW_HOST:
```bash
sudo mkdir -p /home/postgres/.ssh && \
sudo chown -R postgres:postgres /home/postgres && \
sudo chmod 700 /home/postgres/.ssh && \
sudo -u postgres bash -c 'ssh-keygen -t ecdsa -b 256 -f /home/postgres/.ssh/id_ecdsa -N "" -C "postgres@$(hostname)"' && \
sudo cat /home/postgres/.ssh/id_ecdsa.pub
```
Copy the public key from the above output
🟡 On replica:
```bash
sudo mkdir -p /home/postgres/.ssh && \
sudo chown -R postgres:postgres /home/postgres && \
sudo chmod 700 /home/postgres/.ssh && \
echo [public_key] | sudo tee /home/postgres/.ssh/authorized_keys > /dev/null && \
sudo chmod 700 /home/postgres/.ssh
```
---
### 🟢 1. PRIMARY — Create publication and replication slots
```bash
sudo -u postgres pg_create_replication_publications
```
or
```bash
sudo -u postgres pg_create_replication_publication [db_name]
```
Listing publications and slots:
```bash
sudo -u postgres pg_list_replication_publications
sudo -u postgres pg_list_replication_slots
```
---
### 🟡 3. REPLICA — Pause replication
```bash
sudo -u postgres psql -c "SELECT pg_wal_replay_pause();"
```
Verify:
```bash
sudo -u postgres psql -c "SELECT pg_is_wal_replay_paused();"
```
---
### 🟡 4. REPLICA — Run dump
```bash
sudo -u postgres pg_dump_all_databases
```
or
```bash
sudo -u postgres bash -c "pg_dumpall --globals-only > /tmp/globals.sql"
sudo -u postgres pg_dump_database [db_name]
```
---
### 🟡 5. REPLICA — Resume replication
```bash
sudo -u postgres psql -c "SELECT pg_wal_replay_resume();"
```
---
### 🔵 6. COPY dumps to NEW HOST
From NEW_HOST:
```bash
export REPLICA_HOST=[private_ip] && \
cd /tmp && \
sudo -u postgres scp "postgres@$REPLICA_HOST:/tmp/globals.sql" . && \
sudo -u postgres scp "postgres@$REPLICA_HOST:/tmp/dump_*.tar.zst" .
```
---
### 🔵 7. NEW HOST (PostgreSQL 14) — Restore
#### 7.1 Restore globals
```bash
sudo -u postgres psql -f /tmp/globals.sql
```
---
#### 7.2 Create databases
```bash
sudo -u postgres psql -Atqc "SELECT datname FROM pg_database WHERE datallowconn AND datname NOT IN ('template1')" | \
xargs -I{} sudo -u postgres createdb {}
```
or
```bash
sudo -u postgres createdb [db_name]
```
---
#### 7.3 Restore each database
```bash
sudo -u postgres pg_restore_all_databases
```
or
```bash
sudo -u postgres pg_restore_database [db_name]
```
---
### 🔵 8. NEW HOST — Create subscriptions
```bash
sudo -u postgres pg_create_replication_subscriptions
```
or
```bash
sudo -u postgres pg_create_replication_subscription [db_name]
```
---
### 🔵 9. NEW HOST — Monitor replication
```bash
sudo -u postgres pg_list_replication_subscriptions
```
---
### 🔴 11. CUTOVER
#### 11.1 Stop writes on old primary
Put app(s) in maintenance mode, stop the app/daemons.
---
#### 11.2 Wait for replication to catch up
TODO: not the best way to check, since WAL LSNs keep increasing
```bash
sudo -u postgres psql -d [db_name] -c "SELECT * FROM pg_stat_subscription;"
```
---
#### 11.3 Fix sequences
Run per DB:
```bash
sudo -u postgres pg_fix_sequences_in_all_databases
```
or
```bash
sudo -u postgres pg_fix_sequences [db_name]
```
---
#### 11.4 Point app to NEW_HOST
1. Update `pg.kosmos.local` in `/etc/hosts` on app server(s). For example:
```bash
export NEW_PG_PRIMARY=[private_ip]
knife ssh roles:ejabberd -a knife_zero.host "sudo sed -r \"s/^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\s(pg.kosmos.local)/$NEW_PG_PRIMARY\t\1/\" -i /etc/hosts"
```
Or override node attribute(s) if necessary and/or approporiate.
2. Start the app/daemons, and deactivate maintenance mode.
---
### 🧹 12. CLEANUP NEW_HOST
```bash
sudo -u postgres pg_drop_replication_subscriptions
```
---
### 🧹 13. CLEANUP PRIMARY
TODO: Looks like slots are dropped automatically, when subscriptions are dropped
```bash
sudo -u postgres pg_drop_replication_publications
```
---
### 🧹 13. CLEANUP Chef
Once all apps/databases are migrated, update the role in the node
config of the new primary to 'postgres_primary' and converge it.
Also delete the old primary node config from the Chef repo.
---
### ✅ DONE
---
+10
View File
@@ -18,6 +18,16 @@
"relay_url": "wss://nostr.kosmos.org"
}
},
"blossom": {
"domain": "blossom.kosmos.org",
"storage": {
"s3": {
"endpoint": "s3.kosmos.org",
"region": "garage",
"bucket": "blossom"
}
}
},
"discourse": {
"domain": "community.kosmos.org"
},
+4 -4
View File
@@ -67,13 +67,13 @@
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.2.7",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib",
"version": "18.10.17",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.10.17/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.1.4",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.1.4/lib/ohai"
"version": "18.2.13",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.2.13/lib/ohai"
}
}
},
+1 -1
View File
@@ -8,7 +8,7 @@
"automatic": {
"fqdn": "bitcoin-2",
"os": "linux",
"os_version": "5.4.0-163-generic",
"os_version": "5.4.0-216-generic",
"hostname": "bitcoin-2",
"ipaddress": "192.168.122.148",
"roles": [
+1 -1
View File
@@ -61,7 +61,7 @@
}
},
"run_list": [
"recipe[kosmos-base]",
"role[base]",
"role[kvm_guest]",
"role[garage_gateway]",
"role[kosmos_discourse]"
+5 -4
View File
@@ -46,6 +46,7 @@
"kosmos_garage::default",
"kosmos_garage::firewall_rpc",
"kosmos_assets::nginx_site",
"kosmos_blossom::nginx",
"kosmos_discourse::nginx",
"kosmos_drone::nginx",
"kosmos_garage::nginx_web",
@@ -112,13 +113,13 @@
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.2.7",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib",
"version": "18.10.17",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.10.17/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.1.4",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.1.4/lib/ohai"
"version": "18.2.13",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.2.13/lib/ohai"
}
}
},
+1 -1
View File
@@ -55,7 +55,7 @@
}
},
"run_list": [
"recipe[kosmos-base]",
"role[base]",
"role[kvm_guest]",
"role[drone]"
]
+5 -4
View File
@@ -39,6 +39,7 @@
"kosmos_garage::default",
"kosmos_garage::firewall_rpc",
"kosmos_assets::nginx_site",
"kosmos_blossom::nginx",
"kosmos_discourse::nginx",
"kosmos_drone::nginx",
"kosmos_garage::nginx_web",
@@ -105,13 +106,13 @@
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.2.7",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib",
"version": "18.10.17",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.10.17/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.1.4",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.1.4/lib/ohai"
"version": "18.2.13",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.2.13/lib/ohai"
}
}
},
-64
View File
@@ -1,64 +0,0 @@
{
"name": "garage-10",
"chef_environment": "production",
"normal": {
"knife_zero": {
"host": "10.1.1.27"
}
},
"automatic": {
"fqdn": "garage-10",
"os": "linux",
"os_version": "5.4.0-1090-kvm",
"hostname": "garage-10",
"ipaddress": "192.168.122.70",
"roles": [
"base",
"kvm_guest",
"garage_node"
],
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_kvm::guest",
"kosmos_garage",
"kosmos_garage::default",
"kosmos_garage::firewall_rpc",
"kosmos_garage::firewall_apis",
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
"kosmos-base::firewall",
"kosmos-postfix::default",
"postfix::default",
"postfix::_common",
"postfix::_attributes",
"postfix::sasl_auth",
"hostname::default",
"firewall::default"
],
"platform": "ubuntu",
"platform_version": "20.04",
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.5.0",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.5.0/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.1.11",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.1.11/lib/ohai"
}
}
},
"run_list": [
"role[base]",
"role[kvm_guest]",
"role[garage_node]"
]
}
+2
View File
@@ -20,6 +20,7 @@
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_prometheus::node_exporter",
"kosmos_kvm::guest",
"kosmos_garage",
"kosmos_garage::default",
@@ -30,6 +31,7 @@
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::journald_conf",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
"kosmos-base::firewall",
+1
View File
@@ -20,6 +20,7 @@
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_prometheus::node_exporter",
"kosmos_kvm::guest",
"kosmos_garage",
"kosmos_garage::default",
+1
View File
@@ -20,6 +20,7 @@
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_prometheus::node_exporter",
"kosmos_kvm::guest",
"kosmos_garage",
"kosmos_garage::default",
+12 -13
View File
@@ -1,17 +1,17 @@
{
"name": "garage-12",
"name": "garage-16",
"chef_environment": "production",
"normal": {
"knife_zero": {
"host": "10.1.1.224"
"host": "10.1.1.153"
}
},
"automatic": {
"fqdn": "garage-12",
"fqdn": "garage-16",
"os": "linux",
"os_version": "5.15.0-1059-kvm",
"hostname": "garage-12",
"ipaddress": "192.168.122.173",
"os_version": "6.8.0-106-generic",
"hostname": "garage-16",
"ipaddress": "192.168.122.182",
"roles": [
"base",
"kvm_guest",
@@ -20,6 +20,7 @@
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_prometheus::node_exporter",
"kosmos_kvm::guest",
"kosmos_garage",
"kosmos_garage::default",
@@ -28,8 +29,6 @@
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::journald_conf",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
@@ -43,17 +42,17 @@
"firewall::default"
],
"platform": "ubuntu",
"platform_version": "22.04",
"platform_version": "24.04",
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.7.10",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.7.10/lib",
"version": "18.10.17",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.10.17/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.2.5",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.2.5/lib/ohai"
"version": "18.2.13",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.2.13/lib/ohai"
}
}
},
+2
View File
@@ -20,6 +20,7 @@
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_prometheus::node_exporter",
"kosmos_kvm::guest",
"kosmos_garage",
"kosmos_garage::default",
@@ -30,6 +31,7 @@
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::journald_conf",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
"kosmos-base::firewall",
+2
View File
@@ -20,6 +20,7 @@
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_prometheus::node_exporter",
"kosmos_kvm::guest",
"kosmos_garage",
"kosmos_garage::default",
@@ -30,6 +31,7 @@
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::journald_conf",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
"kosmos-base::firewall",
-64
View File
@@ -1,64 +0,0 @@
{
"name": "garage-9",
"chef_environment": "production",
"normal": {
"knife_zero": {
"host": "10.1.1.223"
}
},
"automatic": {
"fqdn": "garage-9",
"os": "linux",
"os_version": "5.4.0-1090-kvm",
"hostname": "garage-9",
"ipaddress": "192.168.122.21",
"roles": [
"base",
"kvm_guest",
"garage_node"
],
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_kvm::guest",
"kosmos_garage",
"kosmos_garage::default",
"kosmos_garage::firewall_rpc",
"kosmos_garage::firewall_apis",
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
"kosmos-base::firewall",
"kosmos-postfix::default",
"postfix::default",
"postfix::_common",
"postfix::_attributes",
"postfix::sasl_auth",
"hostname::default",
"firewall::default"
],
"platform": "ubuntu",
"platform_version": "20.04",
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.5.0",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.5.0/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.1.11",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.1.11/lib/ohai"
}
}
},
"run_list": [
"role[base]",
"role[kvm_guest]",
"role[garage_node]"
]
}
+1 -1
View File
@@ -33,7 +33,7 @@
"kosmos_gitea",
"kosmos_gitea::default",
"kosmos_gitea::backup",
"kosmos_gitea::act_runner",
"kosmos_gitea::runner",
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
+1 -1
View File
@@ -60,7 +60,7 @@
}
},
"run_list": [
"recipe[kosmos-base]",
"role[base]",
"role[kvm_guest]",
"role[ipfs_gateway]"
]
+1 -1
View File
@@ -57,7 +57,7 @@
}
},
"run_list": [
"recipe[kosmos-base]",
"role[base]",
"role[kvm_guest]",
"role[dirsrv_supplier]"
]
+1 -1
View File
@@ -8,7 +8,7 @@
"automatic": {
"fqdn": "leo",
"os": "linux",
"os_version": "5.15.0-164-generic",
"os_version": "5.15.0-173-generic",
"hostname": "leo",
"ipaddress": "5.9.81.116",
"roles": [
+1
View File
@@ -30,6 +30,7 @@
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::journald_conf",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
"kosmos-base::firewall",
+1 -1
View File
@@ -83,7 +83,7 @@
}
},
"run_list": [
"recipe[kosmos-base]",
"role[base]",
"role[kvm_guest]",
"role[ldap_client]",
"role[garage_gateway]",
@@ -1,16 +1,17 @@
{
"name": "postgres-6",
"name": "postgres-11",
"chef_environment": "production",
"normal": {
"knife_zero": {
"host": "10.1.1.196"
"host": "10.1.1.91"
}
},
"automatic": {
"fqdn": "postgres-6",
"fqdn": "postgres-11",
"os": "linux",
"os_version": "5.4.0-173-generic",
"hostname": "postgres-6",
"ipaddress": "192.168.122.60",
"os_version": "5.15.0-1095-kvm",
"hostname": "postgres-11",
"ipaddress": "192.168.122.142",
"roles": [
"base",
"kvm_guest",
@@ -21,18 +22,20 @@
"kosmos-base::default",
"kosmos_kvm::guest",
"kosmos_postgresql::primary",
"kosmos_postgresql::firewall",
"kosmos-akkounts::pg_db",
"kosmos-bitcoin::lndhub-go_pg_db",
"kosmos-bitcoin::nbxplorer_pg_db",
"kosmos_drone::pg_db",
"kosmos_gitea::pg_db",
"kosmos-mastodon::pg_db",
"kosmos_postgresql::firewall",
"kosmos_postgresql::management_scripts",
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::journald_conf",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
"kosmos-base::firewall",
@@ -44,17 +47,17 @@
"hostname::default"
],
"platform": "ubuntu",
"platform_version": "20.04",
"platform_version": "22.04",
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.4.2",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.4.2/lib",
"version": "18.10.17",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.10.17/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.1.11",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.1.11/lib/ohai"
"version": "18.2.13",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.2.13/lib/ohai"
}
}
},
@@ -1,17 +1,17 @@
{
"name": "postgres-10",
"name": "postgres-12",
"chef_environment": "production",
"normal": {
"knife_zero": {
"host": "10.1.1.176"
"host": "10.1.1.134"
}
},
"automatic": {
"fqdn": "postgres-10",
"fqdn": "postgres-12",
"os": "linux",
"os_version": "5.15.0-1095-kvm",
"hostname": "postgres-10",
"ipaddress": "192.168.122.41",
"os_version": "5.15.0-1096-kvm",
"hostname": "postgres-12",
"ipaddress": "192.168.122.139",
"roles": [
"base",
"kvm_guest",
@@ -24,6 +24,7 @@
"kosmos_postgresql::hostsfile",
"kosmos_postgresql::replica",
"kosmos_postgresql::firewall",
"kosmos_postgresql::management_scripts",
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
-62
View File
@@ -1,62 +0,0 @@
{
"name": "postgres-8",
"chef_environment": "production",
"normal": {
"knife_zero": {
"host": "10.1.1.99"
}
},
"automatic": {
"fqdn": "postgres-8",
"os": "linux",
"os_version": "5.15.0-1059-kvm",
"hostname": "postgres-8",
"ipaddress": "192.168.122.100",
"roles": [
"base",
"kvm_guest",
"postgresql_replica"
],
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_kvm::guest",
"kosmos_postgresql::hostsfile",
"kosmos_postgresql::replica",
"kosmos_postgresql::firewall",
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
"kosmos-base::firewall",
"kosmos-postfix::default",
"postfix::default",
"postfix::_common",
"postfix::_attributes",
"postfix::sasl_auth",
"hostname::default"
],
"platform": "ubuntu",
"platform_version": "22.04",
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.5.0",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.5.0/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.1.11",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.1.11/lib/ohai"
}
}
},
"run_list": [
"role[base]",
"role[kvm_guest]",
"role[postgresql_replica]"
]
}
-63
View File
@@ -1,63 +0,0 @@
{
"name": "postgres-9",
"chef_environment": "production",
"normal": {
"knife_zero": {
"host": "10.1.1.3"
}
},
"automatic": {
"fqdn": "postgres-9",
"os": "linux",
"os_version": "5.15.0-1059-kvm",
"hostname": "postgres-9",
"ipaddress": "192.168.122.64",
"roles": [
"base",
"kvm_guest",
"postgresql_replica"
],
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_kvm::guest",
"kosmos_postgresql::hostsfile",
"kosmos_postgresql::replica",
"kosmos_postgresql::firewall",
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::journald_conf",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
"kosmos-base::firewall",
"kosmos-postfix::default",
"postfix::default",
"postfix::_common",
"postfix::_attributes",
"postfix::sasl_auth",
"hostname::default"
],
"platform": "ubuntu",
"platform_version": "22.04",
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.8.54",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.8.54/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.2.8",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.2.8/lib/ohai"
}
}
},
"run_list": [
"role[base]",
"role[kvm_guest]",
"role[postgresql_replica]"
]
}
@@ -1,35 +1,32 @@
{
"name": "garage-13",
"name": "prometheus-1",
"chef_environment": "production",
"normal": {
"knife_zero": {
"host": "10.1.1.179"
"host": "10.1.1.146"
}
},
"automatic": {
"fqdn": "garage-13",
"fqdn": "prometheus-1",
"os": "linux",
"os_version": "5.15.0-1059-kvm",
"hostname": "garage-13",
"ipaddress": "192.168.122.27",
"os_version": "6.8.0-134-generic",
"hostname": "prometheus-1",
"ipaddress": "192.168.122.166",
"roles": [
"base",
"kvm_guest",
"garage_node"
"prometheus_server"
],
"recipes": [
"kosmos-base",
"kosmos-base::default",
"kosmos_prometheus::node_exporter",
"kosmos_kvm::guest",
"kosmos_garage",
"kosmos_garage::default",
"kosmos_garage::firewall_rpc",
"kosmos_garage::firewall_apis",
"kosmos_prometheus::server",
"kosmos_prometheus::alertmanager",
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
"ntp::default",
"ntp::apparmor",
"kosmos-base::journald_conf",
"kosmos-base::systemd_emails",
"apt::unattended-upgrades",
@@ -43,23 +40,23 @@
"firewall::default"
],
"platform": "ubuntu",
"platform_version": "22.04",
"platform_version": "24.04",
"cloud": null,
"chef_packages": {
"chef": {
"version": "18.7.10",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.7.10/lib",
"version": "18.10.17",
"chef_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/chef-18.10.17/lib",
"chef_effortless": null
},
"ohai": {
"version": "18.2.5",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.2.5/lib/ohai"
"version": "18.2.13",
"ohai_root": "/opt/chef/embedded/lib/ruby/gems/3.1.0/gems/ohai-18.2.13/lib/ohai"
}
}
},
"run_list": [
"role[base]",
"role[kvm_guest]",
"role[garage_node]"
"role[prometheus_server]"
]
}
+1 -1
View File
@@ -55,7 +55,7 @@
}
},
"run_list": [
"recipe[kosmos-base]",
"role[base]",
"role[kvm_guest]",
"role[remotestorage_discourse]"
]
+8 -3
View File
@@ -16,7 +16,8 @@
"base",
"kvm_guest",
"strfry",
"ldap_client"
"ldap_client",
"blossom"
],
"recipes": [
"kosmos-base",
@@ -28,6 +29,8 @@
"kosmos_strfry::policies",
"kosmos_strfry::firewall",
"kosmos_strfry::substr",
"kosmos_blossom",
"kosmos_blossom::default",
"apt::default",
"timezone_iii::default",
"timezone_iii::debian",
@@ -43,7 +46,8 @@
"postfix::_attributes",
"postfix::sasl_auth",
"hostname::default",
"deno::default"
"deno::default",
"blossom::default"
],
"platform": "ubuntu",
"platform_version": "22.04",
@@ -63,6 +67,7 @@
"run_list": [
"role[base]",
"role[kvm_guest]",
"role[strfry]"
"role[strfry]",
"role[blossom]"
]
}
+1 -1
View File
@@ -60,7 +60,7 @@
}
},
"run_list": [
"recipe[kosmos-base]",
"role[base]",
"role[kvm_guest]",
"recipe[kosmos-ejabberd::upload_service]"
]
+1
View File
@@ -2,4 +2,5 @@ name "base"
run_list %w(
kosmos-base::default
kosmos_prometheus::node_exporter
)
+16
View File
@@ -0,0 +1,16 @@
name "blossom"
override_attributes(
"blossom" => {
"allowed_pubkeys" => [
# "b3e1b7c0ef48294bd856203bfd460625de95d3afb894e5f09b14cd1f0e7097cf",
# "07e188a1ff87ce171d517b8ed2bb7a31b1d3453a0db3b15379ec07b724d232f3",
# "898a73f2c1f9a9f42d9ef4ac363622f92fdd4290c8f190340a0862d8e0f70046"
]
},
)
run_list %w(
role[ldap_client]
kosmos_blossom::default
)
+1 -1
View File
@@ -1,5 +1,5 @@
name "gitea_actions_runner"
run_list %w(
kosmos_gitea::act_runner
kosmos_gitea::runner
)
+1
View File
@@ -19,6 +19,7 @@ production_run_list = %w(
role[openresty]
role[garage_gateway]
kosmos_assets::nginx_site
kosmos_blossom::nginx
kosmos_discourse::nginx
kosmos_drone::nginx
kosmos_garage::nginx_web
+11 -10
View File
@@ -1,12 +1,13 @@
name "postgresql_primary"
run_list %w(
kosmos_postgresql::primary
kosmos_postgresql::firewall
kosmos-akkounts::pg_db
kosmos-bitcoin::lndhub-go_pg_db
kosmos-bitcoin::nbxplorer_pg_db
kosmos_drone::pg_db
kosmos_gitea::pg_db
kosmos-mastodon::pg_db
)
run_list [
"kosmos_postgresql::primary",
"kosmos-akkounts::pg_db",
"kosmos-bitcoin::lndhub-go_pg_db",
"kosmos-bitcoin::nbxplorer_pg_db",
"kosmos_drone::pg_db",
"kosmos_gitea::pg_db",
"kosmos-mastodon::pg_db",
"kosmos_postgresql::firewall",
"kosmos_postgresql::management_scripts"
]
+6 -5
View File
@@ -1,7 +1,8 @@
name "postgresql_replica"
run_list %w(
kosmos_postgresql::hostsfile
kosmos_postgresql::replica
kosmos_postgresql::firewall
)
run_list [
"kosmos_postgresql::hostsfile",
"kosmos_postgresql::replica",
"kosmos_postgresql::firewall",
"kosmos_postgresql::management_scripts"
]
+8
View File
@@ -0,0 +1,8 @@
name "postgresql_replica_logical"
run_list [
"kosmos_postgresql::hostsfile",
"kosmos_postgresql::replica_logical",
"kosmos_postgresql::firewall",
"kosmos_postgresql::management_scripts"
]
+12
View File
@@ -0,0 +1,12 @@
name "prometheus_server"
default_run_list = [
"kosmos_prometheus::server",
"kosmos_prometheus::alertmanager"
]
env_run_lists(
"_default" => default_run_list,
"development" => default_run_list,
"production" => default_run_list
)
@@ -230,7 +230,6 @@ systemd_unit "akkounts.service" do
WorkingDirectory: deploy_path,
Environment: "RAILS_ENV=#{rails_env} SOLID_QUEUE_IN_PUMA=true",
ExecStart: "#{bundle_path} exec puma -C config/puma.rb --pidfile #{deploy_path}/tmp/puma.pid",
ExecStop: "#{bundle_path} exec puma -C config/puma.rb --pidfile #{deploy_path}/tmp/puma.pid stop",
ExecReload: "#{bundle_path} exec pumactl -F config/puma.rb --pidfile #{deploy_path}/tmp/puma.pid phased-restart",
PIDFile: "#{deploy_path}/tmp/puma.pid",
TimeoutSec: "10",
+28 -22
View File
@@ -24,11 +24,17 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
include_recipe 'apt'
include_recipe 'timezone_iii'
include_recipe 'ntp'
include_recipe 'kosmos-base::journald_conf'
include_recipe 'kosmos-base::systemd_emails'
include_recipe "apt"
directory "/etc/apt/keyrings" do
mode "0755"
action :create
end
include_recipe "timezone_iii"
include_recipe "ntp" if node["platform"] == "ubuntu" && node["platform_version"].to_f < 24.04
include_recipe "kosmos-base::journald_conf"
include_recipe "kosmos-base::systemd_emails"
node.override["apt"]["unattended_upgrades"]["enable"] = true
node.override["apt"]["unattended_upgrades"]["mail_only_on_error"] = false
@@ -43,20 +49,20 @@ node.override["apt"]["unattended_upgrades"]["allowed_origins"] = [
]
node.override["apt"]["unattended_upgrades"]["mail"] = "ops@kosmos.org"
node.override["apt"]["unattended_upgrades"]["syslog_enable"] = true
include_recipe 'apt::unattended-upgrades'
include_recipe "apt::unattended-upgrades"
package 'mailutils'
package 'mosh'
package 'vim'
package "mailutils"
package "mosh"
package "vim"
# Don't create users and rewrite the sudo config in development environment.
# It breaks the vagrant user
unless node.chef_environment == "development"
# Searches data bag "users" for groups attribute "sysadmin".
# Places returned users in Unix group "sysadmin" with GID 2300.
users_manage 'sysadmin' do
users_manage "sysadmin" do
group_id 2300
action [:remove, :create]
action %i[remove create]
end
sudo "sysadmin" do
@@ -65,35 +71,35 @@ unless node.chef_environment == "development"
defaults [
# not default on Ubuntu, explicitely enable. Uses a minimal white list of
# environment variables
'env_reset',
"env_reset",
# Send emails on unauthorized attempts
'mail_badpass',
'secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"',
"mail_badpass",
'secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"'
]
end
include_recipe "kosmos-base::firewall"
include_recipe 'kosmos-postfix'
include_recipe "kosmos-postfix"
node.override['set_fqdn'] = '*'
include_recipe 'hostname'
node.override["set_fqdn"] = "*"
include_recipe "hostname"
package 'ca-certificates'
package "ca-certificates"
directory '/usr/local/share/ca-certificates/cacert' do
directory "/usr/local/share/ca-certificates/cacert" do
action :create
end
['http://www.cacert.org/certs/root.crt', 'http://www.cacert.org/certs/class3.crt'].each do |cert|
["http://www.cacert.org/certs/root.crt", "http://www.cacert.org/certs/class3.crt"].each do |cert|
remote_file "/usr/local/share/ca-certificates/cacert/#{File.basename(cert)}" do
source cert
action :create_if_missing
notifies :run, 'execute[update-ca-certificates]', :immediately
notifies :run, "execute[update-ca-certificates]", :immediately
end
end
execute 'update-ca-certificates' do
execute "update-ca-certificates" do
action :nothing
end
end
@@ -86,9 +86,6 @@ node.default['lndhub-go']['branding'] = {
'footer' => 'about=https://kosmos.org'
}
node.default['dotnet']['ms_packages_src_url'] = "https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb"
node.default['dotnet']['ms_packages_src_checksum'] = "4df5811c41fdded83eb9e2da9336a8dfa5594a79dc8a80133bd815f4f85b9991"
node.default['nbxplorer']['repo'] = 'https://github.com/dgarage/NBXplorer'
node.default['nbxplorer']['revision'] = 'v2.5.26'
node.default['nbxplorer']['source_dir'] = '/opt/nbxplorer'
@@ -98,7 +95,7 @@ node.default['nbxplorer']['postgres']['database'] = 'nbxplorer'
node.default['nbxplorer']['postgres']['user'] = 'nbxplorer'
node.default['btcpay']['repo'] = 'https://github.com/btcpayserver/btcpayserver'
node.default['btcpay']['revision'] = 'v2.1.1'
node.default['btcpay']['revision'] = 'v2.3.7'
node.default['btcpay']['source_dir'] = '/opt/btcpay'
node.default['btcpay']['config_path'] = "/home/#{node['bitcoin']['username']}/.btcpayserver/Main/settings.config"
node.default['btcpay']['log_path'] = "/home/#{node['bitcoin']['username']}/.btcpayserver/debug.log"
@@ -5,29 +5,16 @@
build_essential
apt_repository 'universe' do
uri 'http://archive.ubuntu.com/ubuntu/'
distribution 'focal'
components ['universe']
remote_file "/opt/dotnet-install.sh" do
source "https://dot.net/v1/dotnet-install.sh"
mode "0755"
end
apt_package 'apt-transport-https'
remote_file '/opt/packages-microsoft-prod.deb' do
source node['dotnet']['ms_packages_src_url']
checksum node['dotnet']['ms_packages_src_checksum']
action :create_if_missing
execute "install_dotnet_10" do
command "/opt/dotnet-install.sh -c 10.0 --install-dir /usr/share/dotnet"
not_if '/usr/share/dotnet/dotnet --version | grep -q "^10\."'
end
dpkg_package 'packages-microsoft-prod' do
source '/opt/packages-microsoft-prod.deb'
action :install
notifies :run, 'execute[apt_update]'
link "/usr/bin/dotnet" do
to "/usr/share/dotnet/dotnet"
end
execute 'apt_update' do
command 'apt update'
action :nothing
end
apt_package 'dotnet-sdk-8.0'
@@ -0,0 +1 @@
# No attributes here, use the blossom cookbook's attributes
@@ -0,0 +1,6 @@
name 'kosmos_blossom'
description 'Configures Blossom server for Kosmos infrastructure'
version '0.1.0'
depends 'blossom'
depends 'kosmos-base'
depends 'kosmos_openresty'
@@ -0,0 +1,38 @@
#
# Cookbook Name:: kosmos_blossom
# Recipe:: default
#
credentials = Chef::EncryptedDataBagItem.load('credentials', 'blossom')
ldap_credentials = Chef::EncryptedDataBagItem.load('credentials', 'dirsrv')
node.default['blossom']['repo_url'] = 'https://github.com/67P/blossom-server.git'
node.default['blossom']['revision'] = 'feature/ldap'
node.default['blossom']['storage']['backend'] = 's3'
node.default['blossom']['storage']['s3']['access_key'] = credentials['s3_access_key']
node.default['blossom']['storage']['s3']['secret_key'] = credentials['s3_secret_key']
node.default['blossom']['allow_anonymous_uploads'] = false
node.default['blossom']['ldap']['enabled'] = true
node.default['blossom']['ldap']['url'] = 'ldap://ldap.kosmos.local:389'
node.default['blossom']['ldap']['bind_dn'] = ldap_credentials["service_dn"]
node.default['blossom']['ldap']['password'] = ldap_credentials["service_password"]
node.default['blossom']['ldap']['search_dn'] = "ou=kosmos.org,cn=users,dc=kosmos,dc=org"
node.default['blossom']['ldap']['search_filter'] = "(nostrKey={pubkey})"
node.default['blossom']['dashboard']['enabled'] = true
node.default['blossom']['dashboard']['username'] = credentials['admin_username'] || 'admin'
node.default['blossom']['dashboard']['password'] = credentials['admin_password']
node.default['blossom']['landing']['title'] = 'Kosmos Blossom Server'
include_recipe 'blossom::default'
firewall_rule 'blossom' do
port node['blossom']['port']
source '10.1.1.0/24'
protocol :tcp
command :allow
end
@@ -0,0 +1,28 @@
#
# Cookbook Name:: kosmos_blossom
# Recipe:: nginx
#
domain = node['blossom']['domain']
blossom_node = search(:node, 'role:blossom').first
if blossom_node.nil?
Chef::Log.warn("No node found with 'blossom' role. Not configuring nginx site.")
return
end
tls_cert_for domain do
auth 'gandi_dns'
action :create
end
openresty_site domain do
template 'nginx_conf_blossom.erb'
variables domain: domain,
upstream_host: blossom_node['knife_zero']['host'],
upstream_port: node['blossom']['port'],
max_size_mb: node['blossom']['max_size'] / 1024 / 1024,
ssl_cert: "/etc/letsencrypt/live/#{domain}/fullchain.pem",
ssl_key: "/etc/letsencrypt/live/#{domain}/privkey.pem"
end
@@ -0,0 +1,26 @@
upstream _blossom {
server <%= @upstream_host %>:<%= @upstream_port %>;
}
server {
server_name <%= @domain %>;
listen <%= "#{node['openresty']['listen_ip']}:" if node['openresty']['listen_ip'] %>443 ssl http2;
listen <%= "[#{node['openresty']['listen_ipv6']}]" %>:443 ssl http2;
access_log "/var/log/nginx/<%= @domain %>.access.log";
error_log "/var/log/nginx/<%= @domain %>.error.log";
client_max_body_size <%= @max_size_mb %>M;
ssl_certificate <%= @ssl_cert %>;
ssl_certificate_key <%= @ssl_key %>;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://_blossom;
proxy_http_version 1.1;
}
}
@@ -1,2 +1,6 @@
node.default["kosmos_drone"]["domain"] = "drone.kosmos.org"
node.default["kosmos_drone"]["upstream_port"] = 80
node.default["kosmos_drone"]["pg_host"] = "pg.kosmos.local"
node.default["kosmos_drone"]["pg_port"] = 5432
node.default["kosmos_drone"]["pg_db"] = "drone"
node.default["kosmos_drone"]["pg_user"] = "drone"
@@ -9,11 +9,11 @@ credentials = data_bag_item("credentials", "drone")
drone_credentials = data_bag_item('credentials', 'drone')
postgres_config = {
username: "drone",
password: drone_credentials["postgresql_password"],
host: "pg.kosmos.local",
port: 5432,
database: "drone"
host: node["kosmos_drone"]["pg_host"],
port: node["kosmos_drone"]["pg_port"],
database: node["kosmos_drone"]["pg_db"],
username: node["kosmos_drone"]["pg_user"],
password: drone_credentials["postgresql_password"]
}
directory deploy_path do
@@ -1,5 +1,5 @@
node.default["gitea"]["version"] = "1.25.4"
node.default["gitea"]["checksum"] = "a3031853e67c53714728ef705642c9046a11fb0ea356aff592e23efe6114607d"
node.default["gitea"]["version"] = "1.26.4"
node.default["gitea"]["checksum"] = "0faa36d151918f8f7d6e0f3ae67597d1c338583d695add146ac393109d0fc44a"
node.default["gitea"]["repo"] = nil
node.default["gitea"]["revision"] = nil
node.default["gitea"]["working_directory"] = "/var/lib/gitea"
@@ -23,5 +23,5 @@ node.default["gitea"]["config"] = {
}
}
node.default["gitea"]["act_runner"]["version"] = "0.2.13"
node.default["gitea"]["act_runner"]["checksum"] = "3acac8b506ac8cadc88a55155b5d6378f0fab0b8f62d1e0c0450f4ccd69733e2"
node.default["gitea"]["runner"]["version"] = "2.0.0"
node.default["gitea"]["runner"]["checksum"] = "447156b33407ee045409f5552bd4a188a315cdd4085b4b498d8d4a9ad26c9f73"
@@ -1,10 +1,10 @@
#
# Cookbook:: kosmos_gitea
# Recipe:: act_runner
# Recipe:: runner
#
version = node["gitea"]["act_runner"]["version"]
download_url = "https://dl.gitea.com/act_runner/#{version}/act_runner-#{version}-linux-amd64"
version = node["gitea"]["runner"]["version"]
download_url = "https://dl.gitea.com/gitea-runner/#{version}/gitea-runner-#{version}-linux-amd64"
working_directory = node["gitea"]["working_directory"]
gitea_credentials = data_bag_item("credentials", "gitea")
runners = gitea_credentials["runners"]
@@ -25,9 +25,9 @@ end
package apt_pkg
end
remote_file "/usr/local/bin/act_runner" do
remote_file "/usr/local/bin/gitea_runner" do
source download_url
checksum node["gitea"]["act_runner"]["checksum"]
checksum node["gitea"]["runner"]["checksum"]
mode "0750"
end
@@ -46,7 +46,7 @@ runners.each do |runner|
bash "register_#{runner["org"]}_runner" do
cwd runner_dir
code <<-EOF
act_runner register \
gitea_runner register \
--no-interactive \
--instance #{gitea_host} \
--name #{runner_name} \
@@ -59,7 +59,7 @@ act_runner register \
content({
Unit: {
Description: "Gitea Actions Runner for '#{runner["org"]}' org",
Documentation: ["https://gitea.com/gitea/act_runner"],
Documentation: ["https://gitea.com/gitea/runner"],
Requires: "gitea.service",
After: "syslog.target network.target"
},
@@ -67,7 +67,7 @@ act_runner register \
Type: "simple",
WorkingDirectory: runner_dir,
Environment: "HOME=/root",
ExecStart: "/usr/local/bin/act_runner daemon",
ExecStart: "/usr/local/bin/gitea_runner daemon",
ExecStartPre: "/bin/sleep 3", # Wait for Gitea's API to be up when restarting at the same time
Restart: "always",
},
@@ -75,10 +75,6 @@ DEFAULT_ALLOW_CREATE_ORGANIZATION = false
DEFAULT_ENABLE_TIMETRACKING = false
NO_REPLY_ADDRESS = noreply.kosmos.org
[picture]
DISABLE_GRAVATAR = false
ENABLE_FEDERATED_AVATAR = true
[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false
@@ -18,6 +18,8 @@ server {
client_max_body_size 121M;
proxy_intercept_errors on;
location ~ ^/(avatars|repo-avatars)/.*$ {
proxy_buffers 1024 8k;
proxy_pass http://_gitea_web;
@@ -52,5 +54,18 @@ server {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
error_page 404 = @slow_404;
}
# Slow down 404 responses to make scraping random URLs less attractive
location @slow_404 {
internal;
default_type text/plain;
content_by_lua_block {
ngx.sleep(10)
ngx.status = 404
ngx.say("Not Found")
ngx.exit(ngx.HTTP_NOT_FOUND)
}
}
}
@@ -1,9 +1,9 @@
release = "20260320"
img_filename = "ubuntu-22.04-server-cloudimg-amd64-disk-kvm"
release = "20260321"
img_filename = "ubuntu-24.04-server-cloudimg-amd64"
node.default["kosmos_kvm"]["host"]["qemu_base_image"] = {
"url" => "https://cloud-images.ubuntu.com/releases/jammy/release-#{release}/#{img_filename}.img",
"checksum" => "f7173eb7137b4f0ebeaea8fffe68ecdab1e3c787bde1fd8dfdf27103554332b3",
"url" => "https://cloud-images.ubuntu.com/releases/noble/release-#{release}/#{img_filename}.img",
"checksum" => "5c3ddb00f60bc455dac0862fabe9d8bacec46c33ac1751143c5c3683404b110d",
"path" => "/var/lib/libvirt/images/base/#{img_filename}-#{release}.qcow2"
}
@@ -70,7 +70,7 @@ virt-install \
--vcpus "$CPUS" \
--cpu host \
--arch x86_64 \
--osinfo detect=on,name=ubuntujammy \
--osinfo detect=on,name=ubuntu24.04 \
--hvm \
--virt-type kvm \
--disk "$IMAGE_PATH" \
@@ -1,3 +1,8 @@
node.default['kosmos_postgresql']['postgresql_version'] = "14"
# This is set to false by default, and set to true in the server resource
# for replicas.
node.default['kosmos_postgresql']['ready_to_set_up_replica'] = false
# Address space from which clients are allowed to connect
node.default['kosmos_postgresql']['access_addr'] = "10.1.1.0/24"
@@ -0,0 +1,31 @@
#!/bin/bash
set -euo pipefail
DB_NAME="${1:?Usage: $0 <database_name>}"
echo "== Processing DB: $DB_NAME =="
# Create publication (idempotent)
psql -d "$DB_NAME" -v ON_ERROR_STOP=1 <<'SQL'
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_publication WHERE pubname = 'migrate_pub'
) THEN
CREATE PUBLICATION migrate_pub FOR ALL TABLES;
END IF;
END
$$;
SQL
# Create logical replication slot (idempotent-ish)
SLOT="migrate_slot_${DB_NAME}"
if ! psql -d "$DB_NAME" -Atqc "SELECT 1 FROM pg_replication_slots WHERE slot_name = '$SLOT'" | grep -q 1; then
echo " Creating slot: $SLOT"
psql -d "$DB_NAME" -c "SELECT pg_create_logical_replication_slot('$SLOT', 'pgoutput');"
else
echo " Slot already exists: $SLOT"
fi
echo "== Done =="
@@ -0,0 +1,34 @@
#!/bin/bash
set -e
echo "== Creating publication in each database =="
for db in $(psql -Atqc "SELECT datname FROM pg_database WHERE datallowconn AND datname NOT IN ('template1','postgres')"); do
echo "Processing DB: $db"
# Create publication (idempotent)
psql -d "$db" -v ON_ERROR_STOP=1 <<SQL
DO \$\$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_publication WHERE pubname = 'migrate_pub'
) THEN
CREATE PUBLICATION migrate_pub FOR ALL TABLES;
END IF;
END
\$\$;
SQL
# Create logical replication slot (idempotent-ish)
SLOT="migrate_slot_${db}"
if ! psql -d "$db" -Atqc "SELECT 1 FROM pg_replication_slots WHERE slot_name = '$SLOT'" | grep -q 1; then
echo " Creating slot: $SLOT"
psql -d "$db" -c "SELECT pg_create_logical_replication_slot('$SLOT', 'pgoutput');"
else
echo " Slot already exists: $SLOT"
fi
done
echo "== Done =="
@@ -0,0 +1,34 @@
#!/bin/bash
set -e
echo "== Dropping subscriptions slots and publications =="
for db in $(psql -Atqc "SELECT datname FROM pg_database WHERE datallowconn AND datname NOT IN ('template1','postgres')"); do
echo "Processing DB: $db"
SLOT="migrate_slot_${db}"
# Drop slot if exists
if psql -d "$db" -Atqc "SELECT 1 FROM pg_replication_slots WHERE slot_name = '$SLOT'" | grep -q 1; then
echo " Dropping slot: $SLOT"
psql -d "$db" -c "SELECT pg_drop_replication_slot('$SLOT');"
else
echo " Slot not found: $SLOT"
fi
# Drop publication if exists
psql -d "$db" -v ON_ERROR_STOP=1 <<SQL
DO \$\$
BEGIN
IF EXISTS (
SELECT 1 FROM pg_publication WHERE pubname = 'migrate_pub'
) THEN
DROP PUBLICATION migrate_pub;
END IF;
END
\$\$;
SQL
done
echo "== Done =="
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
set -e
echo "== Dropping subscriptions =="
for db in $(psql -Atqc "SELECT datname FROM pg_database WHERE datallowconn AND datname NOT IN ('template1','postgres')"); do
echo "Processing DB: $db"
SUB="migrate_sub_${db}"
# Check if subscription exists
EXISTS=$(psql -d "$db" -Atqc "SELECT 1 FROM pg_subscription WHERE subname = '$SUB'")
if [ "$EXISTS" = "1" ]; then
echo " Found subscription: $SUB"
# Disable first (good practice)
psql -d "$db" -c "ALTER SUBSCRIPTION $SUB DISABLE;"
# Drop it (must be top-level)
psql -d "$db" -c "DROP SUBSCRIPTION $SUB;"
else
echo " No subscription: $SUB"
fi
done
echo "== Done =="
@@ -0,0 +1,9 @@
#!/bin/bash
cd /tmp && \
(pg_dumpall --globals-only > globals.sql) && \
psql -Atqc "SELECT datname FROM pg_database WHERE datallowconn AND datname NOT IN (''template1'',''postgres'')" | \
xargs -I{} -P4 sh -c "
pg_dump -Fd -j 4 -d \"{}\" -f dump_{} &&
tar -cf - dump_{} | zstd -19 -T0 > dump_{}.tar.zst &&
rm -rf dump_{}
"
@@ -0,0 +1,10 @@
#!/bin/bash
set -euo pipefail
DB_NAME="${1:?Usage: $0 <database_name>}"
cd /tmp
pg_dump -Fd -j 4 -d "$DB_NAME" -f "dump_${DB_NAME}"
tar -cf - "dump_${DB_NAME}" | zstd -19 -T0 > "dump_${DB_NAME}.tar.zst"
rm -rf "dump_${DB_NAME}"
@@ -0,0 +1,35 @@
#!/bin/bash
set -e
DB="$1"
if [ -z "$DB" ]; then
echo "Usage: $0 <database>"
exit 1
fi
echo "== Fixing sequences in database: $DB =="
SQL=$(psql -d "$DB" -Atqc "
SELECT
'SELECT setval(' ||
quote_literal(pg_get_serial_sequence(quote_ident(n.nspname)||'.'||quote_ident(c.relname), a.attname)) ||
', COALESCE(MAX(' || quote_ident(a.attname) || '), 0) + 1, false) FROM ' ||
quote_ident(n.nspname)||'.'||quote_ident(c.relname) || ';'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
WHERE c.relkind = 'r'
AND a.attnum > 0
AND NOT a.attisdropped
AND pg_get_serial_sequence(quote_ident(n.nspname)||'.'||quote_ident(c.relname), a.attname) IS NOT NULL;
")
if [ -z "$SQL" ]; then
echo "No sequences found in $DB"
exit 0
fi
echo "$SQL" | psql -d "$DB"
echo "== Done =="
@@ -0,0 +1,38 @@
#!/bin/bash
set -e
echo "== Fixing sequences across all databases =="
for db in $(psql -Atqc "SELECT datname FROM pg_database WHERE datallowconn AND datname NOT IN ('template1','postgres')"); do
echo "---- DB: $db ----"
# Generate fix statements
SQL=$(psql -d "$db" -Atqc "
SELECT
'SELECT setval(' ||
quote_literal(pg_get_serial_sequence(quote_ident(n.nspname)||'.'||quote_ident(c.relname), a.attname)) ||
', COALESCE(MAX(' || quote_ident(a.attname) || '), 0) + 1, false) FROM ' ||
quote_ident(n.nspname)||'.'||quote_ident(c.relname) || ';'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
WHERE c.relkind = 'r'
AND a.attnum > 0
AND NOT a.attisdropped
AND pg_get_serial_sequence(quote_ident(n.nspname)||'.'||quote_ident(c.relname), a.attname) IS NOT NULL;
")
if [ -z "$SQL" ]; then
echo "No sequences found in $db"
continue
fi
echo "Fixing sequences in $db..."
# Execute generated statements
echo "$SQL" | psql -d "$db"
done
echo "== Done fixing sequences =="
@@ -0,0 +1,5 @@
#!/bin/bash
for db in $(psql -Atqc "SELECT datname FROM pg_database WHERE datallowconn AND datname NOT IN ('template1','postgres')"); do
echo "DB: $db"
psql -d "$db" -Atqc "SELECT pubname FROM pg_publication;"
done
@@ -0,0 +1,5 @@
#!/bin/bash
psql -c "
SELECT slot_name,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))
FROM pg_replication_slots;"
@@ -0,0 +1,16 @@
#!/bin/bash
set -euo pipefail
psql -Atqc "
SELECT datname
FROM pg_database
WHERE datallowconn
AND datname NOT IN ('template1','postgres')
" | while read -r db; do
result=$(psql -X -At -d "$db" -c "SELECT * FROM pg_stat_subscription;" 2>/dev/null || true)
if [[ -n "$result" ]]; then
echo "==== DB: $db ===="
echo "$result"
fi
done
@@ -0,0 +1,12 @@
#!/bin/bash
set -euo pipefail
cd /tmp
for f in dump_*.tar.zst; do
db=$(echo $f | sed "s/dump_\(.*\)\.tar\.zst/\1/")
echo "Restoring $db"
zstd -d "$f" -c | tar -xf -
pg_restore -j 4 -d "$db" dump_$db
rm -rf "dump_$db"
done
@@ -0,0 +1,14 @@
#!/bin/bash
set -euo pipefail
DB_NAME="${1:?Usage: $0 <database_name>}"
cd /tmp
FILE="dump_${DB_NAME}.tar.zst"
DIR="dump_${DB_NAME}"
echo "Restoring $DB_NAME"
zstd -d "$FILE" -c | tar -xf -
pg_restore -j 4 -d "$DB_NAME" "$DIR"
rm -rf "$DIR"
@@ -36,10 +36,8 @@ class Chef
end
end
def postgresql_service_name
postgresql_version = "12"
"postgresql@#{postgresql_version}-main"
def postgresql_version
node['kosmos_postgresql']['postgresql_version']
end
end
end
@@ -0,0 +1,121 @@
#
# Cookbook:: kosmos_postgresql
# Recipe:: management_scripts
#
credentials = data_bag_item('credentials', 'postgresql')
cookbook_file "/usr/local/bin/pg_dump_all_databases" do
source "dump_all_databases.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_dump_database" do
source "dump_database.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_restore_all_databases" do
source "restore_all_databases.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_restore_database" do
source "restore_database.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_create_replication_publications" do
source "create_publications.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_create_replication_publication" do
source "create_publication.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_drop_replication_publications" do
source "drop_publications.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_list_replication_publications" do
source "list_publications.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_list_replication_slots" do
source "list_replication_slots.sh"
user "postgres"
group "postgres"
mode "0744"
end
template "/usr/local/bin/pg_create_replication_subscriptions" do
source "create_subscriptions.sh.erb"
user "postgres"
group "postgres"
mode "0740"
variables pg_host: "pg.kosmos.local",
pg_port: 5432,
pg_user: "replication",
pg_pass: credentials["replication_password"]
sensitive true
end
template "/usr/local/bin/pg_create_replication_subscription" do
source "create_subscription.sh.erb"
user "postgres"
group "postgres"
mode "0740"
variables pg_host: "pg.kosmos.local",
pg_port: 5432,
pg_user: "replication",
pg_pass: credentials["replication_password"]
sensitive true
end
cookbook_file "/usr/local/bin/pg_drop_replication_subscriptions" do
source "drop_subscriptions.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_list_replication_subscriptions" do
source "list_subscriptions.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_fix_sequences_in_all_databases" do
source "fix_sequences.sh"
user "postgres"
group "postgres"
mode "0744"
end
cookbook_file "/usr/local/bin/pg_fix_sequences" do
source "fix_sequences.sh"
user "postgres"
group "postgres"
mode "0744"
end
@@ -3,31 +3,6 @@
# Recipe:: primary
#
postgresql_version = "12"
postgresql_service = "postgresql@#{postgresql_version}-main"
service postgresql_service do
supports restart: true, status: true, reload: true
end
postgresql_custom_server postgresql_version do
role "primary"
end
postgresql_access "zerotier members" do
access_type "host"
access_db "all"
access_user "all"
access_addr "10.1.1.0/24"
access_method "md5"
notifies :reload, "service[#{postgresql_service}]", :immediately
end
postgresql_access "zerotier members replication" do
access_type "host"
access_db "replication"
access_user "replication"
access_addr "10.1.1.0/24"
access_method "md5"
notifies :reload, "service[#{postgresql_service}]", :immediately
end
@@ -3,54 +3,34 @@
# Recipe:: replica
#
postgresql_version = "12"
postgresql_service = "postgresql@#{postgresql_version}-main"
postgresql_custom_server postgresql_version do
role "replica"
end
service postgresql_service do
supports restart: true, status: true, reload: true
end
postgresql_data_bag_item = data_bag_item('credentials', 'postgresql')
primary = postgresql_primary
unless primary.nil?
# TODO
if primary.nil?
Chef::Log.warn("No PostgreSQL primary node found. Skipping replication setup.")
return
end
postgresql_service_name = "postgresql@#{postgresql_version}-main"
postgresql_data_dir = "/var/lib/postgresql/#{postgresql_version}/main"
# FIXME get zerotier IP
# TODO Replace pg.kosmos.local with private IP once available
# via proper node attribute
# https://gitea.kosmos.org/kosmos/chef/issues/263
execute "set up replication" do
command <<-EOF
systemctl stop #{postgresql_service}
systemctl stop #{postgresql_service_name}
mv #{postgresql_data_dir} #{postgresql_data_dir}.old
pg_basebackup -h pg.kosmos.local -U replication -D #{postgresql_data_dir} -R
chown -R postgres:postgres #{postgresql_data_dir}
systemctl start #{postgresql_service}
systemctl start #{postgresql_service_name}
EOF
environment 'PGPASSWORD' => postgresql_data_bag_item['replication_password']
sensitive true
not_if { ::File.exist? "#{postgresql_data_dir}/standby.signal" }
end
postgresql_access "zerotier members" do
access_type "host"
access_db "all"
access_user "all"
access_addr "10.1.1.0/24"
access_method "md5"
notifies :reload, "service[#{postgresql_service}]", :immediately
end
postgresql_access "zerotier members replication" do
access_type "host"
access_db "replication"
access_user "replication"
access_addr "10.1.1.0/24"
access_method "md5"
notifies :reload, "service[#{postgresql_service}]", :immediately
end
end
@@ -0,0 +1,8 @@
#
# Cookbook:: kosmos_postgresql
# Recipe:: replica_logical
#
postgresql_custom_server postgresql_version do
role "replica_logical"
end
@@ -44,25 +44,28 @@ action :create do
shared_buffers = if node['memory']['total'].to_i / 1024 < 1024 # < 1GB RAM
"128MB"
else # >= 1GB RAM, use 50% of total RAM
"#{node['memory']['total'].to_i / 1024 / 2}MB"
else # >= 1GB RAM, use 25% of total RAM
"#{node['memory']['total'].to_i / 1024 / 4}MB"
end
additional_config = {
max_connections: 200, # default
shared_buffers: shared_buffers,
work_mem: "4MB",
unix_socket_directories: "/var/run/postgresql",
dynamic_shared_memory_type: "posix",
timezone: "UTC", # default is GMT
listen_addresses: "0.0.0.0",
promote_trigger_file: "#{postgresql_data_dir}/failover.trigger",
wal_keep_segments: 256
wal_level: "logical",
wal_keep_size: 4096, # 256 segments, 16MB each
max_replication_slots: 16
}
postgresql_server_conf "main" do
version postgresql_version
additional_config additional_config
notifies :reload, "service[#{postgresql_service}]", :delayed
notifies :restart, "service[#{postgresql_service}]", :delayed
end
postgresql_user "replication" do
@@ -70,6 +73,24 @@ action :create do
replication true
password postgresql_credentials['replication_password']
end
postgresql_access "all members" do
access_type "host"
access_db "all"
access_user "all"
access_addr node['kosmos_postgresql']['access_addr']
access_method "md5"
notifies :reload, "service[#{postgresql_service}]", :immediately
end
postgresql_access "replication members" do
access_type "host"
access_db "replication"
access_user "replication"
access_addr node['kosmos_postgresql']['access_addr']
access_method "md5"
notifies :reload, "service[#{postgresql_service}]", :immediately
end
end
action_class do
@@ -0,0 +1,31 @@
#!/bin/bash
set -euo pipefail
DB_NAME="${1:?Usage: $0 <database_name>}"
echo "== Processing DB: $DB_NAME =="
SLOT="migrate_slot_${DB_NAME}"
SUB="migrate_sub_${DB_NAME}"
psql -d "$DB_NAME" -v ON_ERROR_STOP=1 <<SQL
DO \$\$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_subscription WHERE subname = '$SUB'
) THEN
CREATE SUBSCRIPTION $SUB
CONNECTION 'host=<%= @pg_host %> port=<%= @pg_port %> dbname=$DB_NAME user=<%= @pg_user %> password=<%= @pg_pass %>'
PUBLICATION migrate_pub
WITH (
slot_name = '$SLOT',
create_slot = false,
copy_data = false,
enabled = true
);
END IF;
END
\$\$;
SQL
echo "== Done =="

Some files were not shown because too many files have changed in this diff Show More