Commit Graph
2328 Commits
Author SHA1 Message Date
raucao 10c95886a1 Update nodes 2026-11-04 16:41:44 +04:00
raucao 79a3b47f1d Upgrade Liquor Cabinet to 2.0.0-rc.2 and enable hashed access tokens 2026-11-04 16:41:44 +04:00
raucao b00e766ee7 Rotate the Liquor Cabinet Garage S3 key for the rs-kosmos bucket 2026-11-04 16:28:58 +04:00
raucao 52e84dc86c Merge pull request 'Migrate the Chef repo to Cinc Workstation' (#692) from chore/cinc into master
Reviewed-on: #692
2026-11-04 10:00:07 +00:00
raucao d256f7c865 Merge remote-tracking branch 'origin/master' into chore/cinc
# Conflicts:
#	AGENTS.md
#	site-cookbooks/kosmos_gitea/CHANGELOG.md
2026-11-04 13:55:42 +04:00
raucao 11afecdff8 Merge pull request 'Use an isolated network per Gitea Actions job' (#694) from fix/gitea-actions-network into chore/cinc
Reviewed-on: #694
2026-11-03 16:48:04 +00:00
raucao 8a60ef2aea Use an isolated network per Gitea Actions job
Job containers were pinned to a single shared `gitea-actions` bridge
network, added with the Actions cache to give service containers DNS and a
fixed gateway for the built-in cache server. Because every job of every run
attached to that one network, concurrent jobs' service containers collided
on hostnames: two simultaneous jobs that both expose a `redis` service meant
either could resolve the other's container, causing intermittent cross-job
CI failures that vanished when runs did not overlap.

Set `container.network` back to empty so the runner auto-creates an isolated
user-defined network per job. Embedded DNS still works on each per-job
network, so service containers remain reachable by name, but jobs can no
longer resolve each other's containers.

`cache.host` is left empty for auto-detection; the runner here is a host
binary (systemd), not containerized, so the detected address is reachable
from job containers. The cache port stays fixed per runner. The cache
firewall rule now allows Docker's default address pool (172.16.0.0/12)
because per-job subnets are dynamic, and the now-unused `gitea-actions`
network is removed on converge once idle.
2026-11-03 20:46:50 +04:00
raucao 3883fbc3e5 Update nodes 2026-11-02 12:05:40 +04:00
raucao a96a49c633 Cinc Client: install the current stable release, not a floating pin
The migration script installed Cinc with no version, which floated and
triggered omnitruck's "You are installing a package without a version pin"
warning. Pass -v latest so omnitruck resolves the current stable build for
the node's platform (20.04 -> 19.2.12, 22.04+/24.04/26.04 -> 19.4.46) and
the warning is suppressed. Document the behavior in the README.

Drop the stale chef_version: 18.2.7 pins from the Test Kitchen suites:
dokken defaults chef_version to "latest", so passing -v is what actually
tracks the current stable client.

Add the missing provisioner.product_name: cinc to the deno and strfry
submodule suites so they run Cinc instead of the license-gated Chef Infra
Client.

Submodule pointers:
  site-cookbooks/deno   3795c9e -> c78bc66
  site-cookbooks/strfry 9c04b39 -> 8107b1e
2026-11-02 11:38:19 +04:00
raucao 8241995fd7 Add sockethub-testing node 2026-11-02 11:33:16 +04:00
raucao 292c1e9296 Update nodes 2026-11-02 11:32:16 +04:00
raucao ee922186eb Merge pull request 'Upgrade Gitea to 28.3.0, Actions runner to 5.2.4' (#693) from chore/upgrade_gitea into master
Reviewed-on: #693
2026-11-02 07:29:46 +00:00
raucao 27dd10d2ee Upgrade Gitea to 28.3.0, Actions runner to 5.2.4
Gitea 28.1.0 -> 28.3.0 (skipping 28.2.0). Both are maintenance releases
with no breaking changes and no app.ini changes. They carry a large set of
security fixes: token scope enforcement across many API and web routes,
OAuth2 ACCOUNT_LINKING=auto account takeover, review-triggered Actions
workflows running with repository secrets, stored XSS via Actions artifact
Content-Encoding, SSH certificate principal confusion, OpenID association
CSRF, and several private-data leaks in feeds, profiles, project boards and
issue timelines.

Behavior changes to be aware of, none requiring a cookbook change:
git maintenance now defaults to the gc strategy; profile pages treat token
requests as anonymous and user/org RSS/Atom feeds require read:user or
read:organization; the API rejects changes to issues, labels, milestones
and releases of archived repositories; review-triggered workflows run on
the pull request head; the gitea-vscode OAuth2 application is
pre-registered; email validation is relaxed to what Go's mail.ParseAddress
accepts.

Gitea Actions runner 5.2.2 -> 5.2.4. No config.yaml changes were required.
5.2.4 fixes the v4 artifact API in exec mode so upload-artifact v4 and
later work; 5.2.3 fixes the action header group, dry runs touching the
Docker daemon, and damaged shared action cache trees.

Checksums verified against the published binaries.
2026-11-02 11:23:26 +04:00
raucao 3c9e58f324 Update AGENTS.md 2026-11-02 11:03:43 +04:00
raucao 7de46b6115 nodes: track mail.kosmos.org after the Cinc Client migration 2026-11-01 23:15:22 +04:00
raucao 5f4c902e95 nodes: track garage-11 after the Cinc Client migration
Points the submodule at the re-converged garage-11 node object (Cinc
Client 19.4.46). Normal attributes (knife_zero, vm_host) are unchanged.
2026-11-01 18:42:19 +04:00
raucao 87d4305c0d scripts: add migrate_node_to_cinc.sh
One-shot, idempotent helper that prepares an existing (Chef) node for
Cinc Workstation management: installs Cinc Client, points /etc/cinc at
the existing /etc/chef config, drops the deprecated attribute lists, and
pre-seeds the allowed_* attribute lists so the first converge does not
filter node attributes (e.g. vm_host). The node can then be converged as
usual. Verified against drone-1.
2026-11-01 18:35:44 +04:00
raucao f2b4834112 nodes: track drone-1 after the Cinc Client migration
Points the submodule at the re-converged drone-1 node object (Cinc
Client 19.2.12). The node's normal attributes (knife_zero, vm_host)
are unchanged.
2026-11-01 18:28:56 +04:00
raucao fd21e09342 test-kitchen: fix Berksfiles and the rskj pin; document open tasks
- kosmos_email: add the missing Berksfile so its declared dependencies
  (hostname/unbound/postfix) resolve
- kosmos_garage: add local path deps for the internal cookbooks so
  Berkshelf stops fetching them from Supermarket
- kosmos_rsk: bump rskj to 9.0.4~<codename> (current ppa:rsksmart/rskj
  build) and update the matching InSpec assertion; suite now passes 9/9
- document the remaining suite work in doc/test-kitchen-open-tasks.md
2026-11-01 18:28:54 +04:00
raucao f6b9d55c58 test-kitchen: provision Cinc Client instead of Chef Infra Client
Set `product_name: cinc` on the dokken provisioner so kitchen-dokken
uses cincproject/cinc and skips the Chef license prompt, and pin
chef_version where it was missing.
2026-11-01 18:28:50 +04:00
raucao b9e16876f9 Remove Vagrant and the vagrant-based Test Kitchen suites
The root Vagrantfile used the license-gated chef_zero provisioner, and
Vagrant's built-in chef provisioner cannot provision Cinc. The
kosmos-bitcoin suite converged nothing (no run_list) and tor-full
targeted long-EOL platforms, so drop both rather than port them.
2026-11-01 18:28:46 +04:00
raucao 470dcb82ee Migrate the Chef repo to Cinc Workstation
Cinc Workstation replaces Chef Workstation for driving knife-zero and
Test Kitchen; nodes run (or are moving to) Cinc Client.

- Pin .ruby-version to cinc-workstation
- Move knife config from .chef/ to .cinc/ (Cinc's USER_CONF_DIR) and
  update .gitignore, README, AGENTS and docs accordingly. On-node config
  now lives in /etc/cinc
- Drop the Gemfile/Gemfile.lock; install knife-zero into Cinc
  Workstation's user gem dir with `cinc gem install knife-zero`
- README: rewrite setup/usage for Cinc Workstation
- scripts/postgresql/switch_primary.sh: drop `bundle exec`
2026-11-01 18:28:44 +04:00
raucao 96b394f48a Track the postgres-13 replica in nodes 2026-11-01 09:14:23 +04:00
greg 8904bb2e67 Merge pull request 'Add interim mitigations for CVE-2026-98432' (#691) from chore/mptcp into master
Reviewed-on: #691
Reviewed-by: Greg <greg@kosmos.org>
2026-10-31 10:49:13 +00:00
raucao 1e018ac6b4 Update nodes
Pick up the refreshed node objects from the CVE-2026-98432 converge.
2026-10-31 14:33:13 +04:00
raucao b299910dc6 Add interim mitigations for CVE-2026-98432
CVE-2026-98432 is a use-after-free in the Linux IPv4 TCP option
handling (inet_opt/ireq_opt are freed with a plain kfree()) that is
reachable by sending a TCP SYN carrying IP options. No fixed kernel has
been released for any supported Ubuntu release yet, so mitigate in
role[base] until one is:

* Drop inbound IPv4 TCP segments that carry IP options (no legitimate
  traffic uses them) and log the drops, rate limited, so probing is
  visible. The rules are added to /etc/ufw/before.rules so they survive
  ufw reloads, and are idempotent.

* Disable Multipath TCP where the kernel supports it (5.6+). Go
  programs built with Go >= 1.24 open MPTCP listeners by default, which
  is the precondition of the published exploit; nothing here needs
  MPTCP. Hosts on older kernels (5.4, no MPTCP) skip this.

The sysctl drop-in is written directly rather than via the core sysctl
resource, whose reload runs `sysctl -p` against /etc/sysctl.conf - a
file that no longer exists on Ubuntu 26.04.

Opt node_exporter (Go >= 1.24, so it opened an MPTCP listener) out of
MPTCP with GODEBUG=multipathtcp=0, and restart it when the unit
changes so the change takes effect without a separate restart.
2026-10-31 14:33:13 +04:00
raucao c8f15c63b4 Merge pull request 'Upgrade Mastodon to 4.7.4' (#689) from chore/upgrade_mastodon into master
Reviewed-on: #689
2026-10-29 21:04:13 +00:00
raucao 695603f37d Upgrade Mastodon to 4.7.4
Bump the runtime Ruby to 4.0.9 (and the ruby-build definitions to
v20261023, the first release that defines it), matching upstream's
updated default Ruby for the 4.7.4 security release.
2026-10-30 00:53:50 +04:00
raucao b0bc630551 Merge pull request 'kredits-github: fix webhook path and restart on unit changes' (#688) from fix/kredits-github-webhook-path into master
Reviewed-on: #688
2026-10-29 15:13:43 +00:00
raucao 3bf51f44d5 kredits-github: fix webhook path and restart on unit changes
Probot 14 changed its default webhook path from "/" (Probot 7) to
"/api/github/webhooks", but the GitHub App still POSTs to
https://kredits-github.kosmos.org/. Every delivery returned 404, so the
app never processed pull_request events and never created the Kredits
check run. Serve "/" again via WEBHOOK_PATH.

Chef's systemd_unit :create only runs daemon-reload and never restarts a
running unit, so the env change above would not have taken effect until
the next repo change or reboot. Manage the unit with :create/:enable and
restart it from a service resource that subscribes to unit changes.
Apply the same fix to the kosmos-hubot bot resource.
2026-10-29 18:51:34 +04:00
raucao 96aa2e0007 Merge pull request 'eMudhra CA follow-up: patched Node.js releases, Cinc, CAcert; migrating off nodejs-4' (#687) from chore/emudhra into master
Reviewed-on: #687
Reviewed-by: Greg <greg@kosmos.org>
2026-10-28 15:19:21 +00:00
raucao 0405459db0 Document remaining items for hubot migration 2026-10-28 18:22:24 +04:00
raucao 34db4dee8f Retire nodejs-4; add role[hubot] to nodejs-5
Drop the nodejs-4 node object (host is powered off) and give nodejs-5
role[hubot], so hubot runs on Node 22 there.
2026-10-28 18:22:24 +04:00
raucao 54e838d77d credentials: consolidate the hubot secrets into one data bag
Read the hal8000 webhook token from credentials/hubot in the mediawiki and
alertmanager recipes, and drop the now-unused botka_irc-libera-chat,
hal8000_xmpp and hal8000_freenode bags.
2026-10-28 18:22:23 +04:00
raucao 69d0f72f87 kosmos_redis: relax the chef_version constraint
Use ">= 18.0" instead of "~> 18.0" so newer Chef/Cinc releases converge.
2026-10-28 18:22:23 +04:00
raucao e53df92b57 kosmos-hubot: scope the nginx upstream to the production environment
search(:node, 'role:hubot') also matched retired nodes outside production
(e.g. nodejs-4 in _default), so the vhosts could proxy to a dead host and
return 502. Restrict the search to chef_environment:production and pick
deterministically.
2026-10-28 18:22:23 +04:00
raucao 086b31a387 kosmos-hubot: declare bot firewall rules at the recipe top level
The per-bot firewall_rule declared inside the kosmos_hubot_bot custom
resource was not collected by the ufw provider when it rebuilt the rules
file, so the bots' HTTP ports were never opened. Declare the rules in the
default recipe instead, matching the pattern used by the other cookbooks,
and drop the now-unused firewall property from the resource.
2026-10-28 18:22:23 +04:00
raucao d3062d701f hubot: drop the unused hubot-auth script
hubot-auth is CoffeeScript-only (needs porting) and nothing in the bots calls
robot.auth; remove it from hal8000's external scripts and drop HUBOT_AUTH_ADMIN
from both bots.
2026-10-28 18:22:22 +04:00
raucao cc9b324a18 Refactor kosmos-hubot to a data-driven model
Replace the per-bot and per-vhost recipes with a single data-driven setup.

- resources/bot.rb (kosmos_hubot_bot): git checkout, npm install, generated
  external-scripts.json, extra files, and one hardened systemd unit for every
  hubot (Type=simple, Restart=always, RestartSec=2, TimeoutSec=10, PrivateTmp,
  ProtectSystem=full, NoNewPrivileges, PrivateDevices)
- recipes/default.rb: shared setup + loop over node['kosmos-hubot']['bots'];
  recipes/nginx.rb: loop over node['kosmos-hubot']['vhosts'];
  recipes/hal8000_extras.rb: augments the hal8000 bot (searched IPFS/RSK hosts,
  kredits .env, wallet) and declares no resources itself
- bot config and vhosts now live in attributes; secrets come from the encrypted
  credentials/hubot bag via secret_env
- Node 22.23.4; kosmos_redis replaces redisio; REDIS_URL uses ?<prefix>; the
  adapter is passed as -a <module>; botka gains hubot-incoming-webhook
- roles/hubot, nginx_proxy, openresty_proxy and the chef-nodes pointer updated
2026-10-28 18:22:22 +04:00
raucao b04e2e153a Remove the dormant wormhole bot
wormhole has no run-list entry on any node; drop its recipe and data bag.
2026-10-28 18:22:22 +04:00
raucao 93fd33a3ef Remove host config for xmpp bots 2026-10-28 18:22:22 +04:00
raucao 552427a10b Track nodejs-5 and the kredits-github migration
Update the nodes submodule (nodejs-5 added, kredits-github moved off nodejs-4) and add the nodejs-5 client.
2026-10-28 18:22:21 +04:00
raucao 44f86d3f07 Serve kredits-github via openresty, on Node 22
Move kredits-github onto Node 22.23.4 and behind the openresty reverse proxy
on the proxy hosts, replacing its own nginx + the custom haproxy entry.

- pin Node 22.23.4; run the app with HOST=0.0.0.0 / NODE_ENV=production and
  open port 3000 to 10.1.1.0/24 so the proxy can reach it over ZeroTier
- create /opt/kredits-github owned by the deploy user before cloning
- add kredits-github::nginx_proxy (tls_cert_for + openresty_site, upstream
  from role:kredits_github) and register it in the openresty_proxy run list
- drop the app-node nginx recipe/template and its role entry
2026-10-28 18:22:21 +04:00
raucao bd4a6501ab Update Ubuntu base image to release-20261017
Refresh the noble cloud image so newly created VMs start with fewer pending
package upgrades. Old base images are left in place.
2026-10-28 18:22:21 +04:00
raucao fc55b0e062 Upgrade RTL to v0.15.15
Pin v0.15.15 and set blockExplorerUrl (config-only since 0.15.14). Discard
local checkout changes before the pinned checkout, and use
`npm ci --omit=dev --legacy-peer-deps` per upstream guidance.
2026-10-28 18:22:21 +04:00
raucao 5bb1499371 Move Node services onto patched releases for the eMudhra CA incident
Bump Node to patched releases (mastodon/sockethub 24.22.1, RTL 22.23.4).
Force the pinned major's latest patch in kosmos-nodejs: nodejs_install only
selects the repo major and a bare `package nodejs` never upgrades an already
installed package.

For hosts stuck on Node 16 (no patched 16.x), run the services with
NODE_OPTIONS=--use-openssl-ca so they use the patched system trust store:
ipfs-pinner, kredits-github, hubot botka/hal8000, sockethub.
2026-10-28 18:22:20 +04:00
raucao c76921da78 Vendor CAcert roots instead of fetching from cacert.org
cacert.org is intermittently unreachable, which fails bootstraps at the
remote_file step. Ship both roots in the cookbook and install them with
cookbook_file; contents match what was already installed fleet-wide.
2026-10-28 18:22:20 +04:00
raucao 6a9ff1eea0 Support Cinc in kosmos-base
Resolve the client config dir at runtime (Chef::Config[:config_file] /
:client_d_dir) instead of hardcoding /etc/chef, and create client.d when
missing so fresh nodes work.

Reload only the attributes file we just wrote, not the whole client.rb:
reloading client.rb reset chef_server_url and dropped knife-zero's -S (TCP)
override, so later server calls fell back to socketless chefzero and failed
with No socketless Cinc Zero server.
2026-10-28 18:22:20 +04:00
raucao 1c96ef42c6 Use Cinc by default for knife-zero bootstraps 2026-10-28 18:22:20 +04:00
raucao b3c9e1e139 Merge pull request 'Upgrade Gitea Runner to 5.2.2' (#686) from chore/upgrade_gitea-runner into master
Reviewed-on: #686
2026-10-25 09:27:43 +00:00