eMudhra CA follow-up: patched Node.js releases, Cinc, CAcert; migrating off nodejs-4 #687

2026-10-28T14:22:55Z

raucao commented

2026-10-28 14:22:55 +00:00

Follow-up to the eMudhra CA incident, plus the Node.js-side migrations that went with it.

eMudhra CA / Node.js releases

Move Node.js services onto patched releases (mastodon/sockethub 24.22.1, RTL 22.23.4); force the pinned major's latest patch in kosmos-nodejs (a bare package nodejs never upgrades an already-installed package).
For hosts stuck on Node.js 16, run the services with NODE_OPTIONS=--use-openssl-ca so they use the patched system trust store (ipfs-pinner, kredits-github, hubot botka/hal8000, sockethub).
Vendor CAcert roots in the cookbook instead of fetching from cacert.org (intermittently unreachable, failing bootstraps).
Support Cinc in kosmos-base (resolve the client config dir at runtime) and use it by default for knife-zero bootstraps.
Upgrade RTL to v0.15.15; refresh the Ubuntu base image.

Hubot migration (off `nodejs-4`)

Data-driven kosmos-hubot: replaces the per-bot/per-vhost recipes with resources/bot.rb (git checkout, npm install, generated external-scripts.json, extra files, one hardened systemd unit per bot) plus recipes/default.rb / recipes/nginx.rb looping over attributes. recipes/hal8000_extras.rb augments hal8000 (IPFS/RSK hosts, kredits .env, wallet).
Bots: botka → @hubot-friends/hubot-irc, hal8000 → hubot-xmpp; Node.js 22.23.4; kosmos_redis replaces redisio; REDIS_URL uses ?<prefix>; the adapter is passed as -a <module>. Drop hubot-auth, hubot-shipit, hubot-schedule, hubot-tell, hubot-seen, hubot-web-push-notifications, hubot-scripts, coffee-script, request, and the dormant wormhole bot.
Networking: bot HTTP ports opened via top-level firewall_rules; vhost upstream scoped to chef_environment:production.
Secrets consolidated into the encrypted credentials/hubot bag; per-bot bags dropped.
Retire nodejs-4 (node object removed; host powered off).

kredits-github (off `nodejs-4`)

Move onto Node.js 22.23.4 and behind the openresty reverse proxy on the proxy hosts, replacing its own nginx + the custom haproxy entry.
Run with HOST=0.0.0.0 / NODE_ENV=production, open port 3000 to 10.1.1.0/24; add kredits-github::nginx_proxy and register it in the openresty_proxy run list; drop the app-node nginx recipe/template and role entry.
Track nodejs-5 in the nodes submodule and add its client.

Related repos

kosmos/hubot-xmpp (Gitea): new hubot 14 XMPP adapter built on @xmpp/client 0.14.0, replacing the old node-xmpp-client fork (host resolved from the JID domain via DNS SRV / well-known; XEP-0368 SNI workaround).
67P/hubot-remotestorage-logger (GitHub): modernized for hubot 14 / ESM and published v2.0.0 and v2.0.1 (npm + GitHub releases); 2.0.1 logs via receiveMiddleware/responseMiddleware, so bot commands and their replies stay out of the archives.
67P/hubot-incoming-webhook (GitHub): modernized for hubot 14 / Node.js 22 and published v2.0.0.
kredits/hubot-kredits (Gitea): target Node.js 22 and use robot.logger.warn (#72); release-drafter added (draft v4.1.1 pending).
kosmos/botka, kosmos/hal8000 (Gitea): modernized to hubot 14 / ESM; hal8000 picks up the new hubot-xmpp.

Verification

Deployed and verified on nodejs-5: botka in IRC rooms + remoteStorage logging; hal8000 online in XMPP MUCs with kredits; kredits-github and the bot webhooks reachable through openresty on draco and fornax.

Waiting on

See doc/hubot-migration.md (upstream @xmpp/stream-management fix, etc.).

Follow-up to the eMudhra CA incident, plus the Node.js-side migrations that went with it. ## eMudhra CA / Node.js releases - Move Node.js services onto patched releases (mastodon/sockethub 24.22.1, RTL 22.23.4); force the pinned major's latest patch in `kosmos-nodejs` (a bare `package nodejs` never upgrades an already-installed package). - For hosts stuck on Node.js 16, run the services with `NODE_OPTIONS=--use-openssl-ca` so they use the patched system trust store (ipfs-pinner, kredits-github, hubot botka/hal8000, sockethub). - Vendor CAcert roots in the cookbook instead of fetching from cacert.org (intermittently unreachable, failing bootstraps). - Support [Cinc](https://cinc.sh/) in `kosmos-base` (resolve the client config dir at runtime) and use it by default for knife-zero bootstraps. - Upgrade RTL to v0.15.15; refresh the Ubuntu base image. ## Hubot migration (off `nodejs-4`) - Data-driven `kosmos-hubot`: replaces the per-bot/per-vhost recipes with `resources/bot.rb` (git checkout, `npm install`, generated `external-scripts.json`, extra files, one hardened systemd unit per bot) plus `recipes/default.rb` / `recipes/nginx.rb` looping over attributes. `recipes/hal8000_extras.rb` augments hal8000 (IPFS/RSK hosts, kredits `.env`, wallet). - Bots: botka → `@hubot-friends/hubot-irc`, hal8000 → `hubot-xmpp`; Node.js 22.23.4; `kosmos_redis` replaces `redisio`; `REDIS_URL` uses `?<prefix>`; the adapter is passed as `-a <module>`. Drop `hubot-auth`, `hubot-shipit`, `hubot-schedule`, `hubot-tell`, `hubot-seen`, `hubot-web-push-notifications`, `hubot-scripts`, `coffee-script`, `request`, and the dormant wormhole bot. - Networking: bot HTTP ports opened via top-level `firewall_rule`s; vhost upstream scoped to `chef_environment:production`. - Secrets consolidated into the encrypted `credentials/hubot` bag; per-bot bags dropped. - Retire `nodejs-4` (node object removed; host powered off). ## kredits-github (off `nodejs-4`) - Move onto Node.js 22.23.4 and behind the openresty reverse proxy on the proxy hosts, replacing its own nginx + the custom haproxy entry. - Run with `HOST=0.0.0.0` / `NODE_ENV=production`, open port 3000 to `10.1.1.0/24`; add `kredits-github::nginx_proxy` and register it in the `openresty_proxy` run list; drop the app-node nginx recipe/template and role entry. - Track `nodejs-5` in the nodes submodule and add its client. ## Related repos - **`kosmos/hubot-xmpp`** (Gitea): new hubot 14 XMPP adapter built on `@xmpp/client` 0.14.0, replacing the old `node-xmpp-client` fork (host resolved from the JID domain via DNS SRV / well-known; XEP-0368 SNI workaround). - **`67P/hubot-remotestorage-logger`** (GitHub): modernized for hubot 14 / ESM and published **v2.0.0** and **v2.0.1** (npm + GitHub releases); 2.0.1 logs via `receiveMiddleware`/`responseMiddleware`, so bot commands and their replies stay out of the archives. - **`67P/hubot-incoming-webhook`** (GitHub): modernized for hubot 14 / Node.js 22 and published **v2.0.0**. - **`kredits/hubot-kredits`** (Gitea): target Node.js 22 and use `robot.logger.warn` (#72); release-drafter added (draft **v4.1.1** pending). - **`kosmos/botka`**, **`kosmos/hal8000`** (Gitea): modernized to hubot 14 / ESM; hal8000 picks up the new `hubot-xmpp`. ## Verification Deployed and verified on `nodejs-5`: botka in IRC rooms + remoteStorage logging; hal8000 online in XMPP MUCs with kredits; kredits-github and the bot webhooks reachable through openresty on draco and fornax. ## Waiting on See `doc/hubot-migration.md` (upstream `@xmpp/stream-management` fix, etc.).

raucao added 18 commits 2026-10-28 14:22:56 +00:00

Use Cinc by default for knife-zero bootstraps 1c96ef42c6

Support Cinc in kosmos-base 6a9ff1eea0

Resolve the client config dir at runtime (Chef::Config[:config_file] /
:client_d_dir) instead of hardcoding /etc/chef, and create client.d when
missing so fresh nodes work.

Reload only the attributes file we just wrote, not the whole client.rb:
reloading client.rb reset chef_server_url and dropped knife-zero's -S (TCP)
override, so later server calls fell back to socketless chefzero and failed
with No socketless Cinc Zero server.

Vendor CAcert roots instead of fetching from cacert.org c76921da78

cacert.org is intermittently unreachable, which fails bootstraps at the
remote_file step. Ship both roots in the cookbook and install them with
cookbook_file; contents match what was already installed fleet-wide.

Move Node services onto patched releases for the eMudhra CA incident 5bb1499371

Bump Node to patched releases (mastodon/sockethub 24.22.1, RTL 22.23.4).
Force the pinned major's latest patch in kosmos-nodejs: nodejs_install only
selects the repo major and a bare `package nodejs` never upgrades an already
installed package.

For hosts stuck on Node 16 (no patched 16.x), run the services with
NODE_OPTIONS=--use-openssl-ca so they use the patched system trust store:
ipfs-pinner, kredits-github, hubot botka/hal8000, sockethub.

Upgrade RTL to v0.15.15 fc55b0e062

Pin v0.15.15 and set blockExplorerUrl (config-only since 0.15.14). Discard
local checkout changes before the pinned checkout, and use
`npm ci --omit=dev --legacy-peer-deps` per upstream guidance.

Update Ubuntu base image to release-20261017 bd4a6501ab

Refresh the noble cloud image so newly created VMs start with fewer pending
package upgrades. Old base images are left in place.

Serve kredits-github via openresty, on Node 22 44f86d3f07

Move kredits-github onto Node 22.23.4 and behind the openresty reverse proxy
on the proxy hosts, replacing its own nginx + the custom haproxy entry.

- pin Node 22.23.4; run the app with HOST=0.0.0.0 / NODE_ENV=production and
  open port 3000 to 10.1.1.0/24 so the proxy can reach it over ZeroTier
- create /opt/kredits-github owned by the deploy user before cloning
- add kredits-github::nginx_proxy (tls_cert_for + openresty_site, upstream
  from role:kredits_github) and register it in the openresty_proxy run list
- drop the app-node nginx recipe/template and its role entry

Track nodejs-5 and the kredits-github migration 552427a10b

Update the nodes submodule (nodejs-5 added, kredits-github moved off nodejs-4) and add the nodejs-5 client.

Remove host config for xmpp bots 93fd33a3ef

Remove the dormant wormhole bot b04e2e153a

wormhole has no run-list entry on any node; drop its recipe and data bag.

Refactor kosmos-hubot to a data-driven model cc9b324a18

Replace the per-bot and per-vhost recipes with a single data-driven setup.

- resources/bot.rb (kosmos_hubot_bot): git checkout, npm install, generated
  external-scripts.json, extra files, and one hardened systemd unit for every
  hubot (Type=simple, Restart=always, RestartSec=2, TimeoutSec=10, PrivateTmp,
  ProtectSystem=full, NoNewPrivileges, PrivateDevices)
- recipes/default.rb: shared setup + loop over node['kosmos-hubot']['bots'];
  recipes/nginx.rb: loop over node['kosmos-hubot']['vhosts'];
  recipes/hal8000_extras.rb: augments the hal8000 bot (searched IPFS/RSK hosts,
  kredits .env, wallet) and declares no resources itself
- bot config and vhosts now live in attributes; secrets come from the encrypted
  credentials/hubot bag via secret_env
- Node 22.23.4; kosmos_redis replaces redisio; REDIS_URL uses ?<prefix>; the
  adapter is passed as -a <module>; botka gains hubot-incoming-webhook
- roles/hubot, nginx_proxy, openresty_proxy and the chef-nodes pointer updated

hubot: drop the unused hubot-auth script d3062d701f

hubot-auth is CoffeeScript-only (needs porting) and nothing in the bots calls
robot.auth; remove it from hal8000's external scripts and drop HUBOT_AUTH_ADMIN
from both bots.

kosmos-hubot: declare bot firewall rules at the recipe top level 086b31a387

The per-bot firewall_rule declared inside the kosmos_hubot_bot custom
resource was not collected by the ufw provider when it rebuilt the rules
file, so the bots' HTTP ports were never opened. Declare the rules in the
default recipe instead, matching the pattern used by the other cookbooks,
and drop the now-unused firewall property from the resource.

kosmos-hubot: scope the nginx upstream to the production environment e53df92b57

search(:node, 'role:hubot') also matched retired nodes outside production
(e.g. nodejs-4 in _default), so the vhosts could proxy to a dead host and
return 502. Restrict the search to chef_environment:production and pick
deterministically.

kosmos_redis: relax the chef_version constraint 69d0f72f87

Use ">= 18.0" instead of "~> 18.0" so newer Chef/Cinc releases converge.

credentials: consolidate the hubot secrets into one data bag 54e838d77d

Read the hal8000 webhook token from credentials/hubot in the mediawiki and
alertmanager recipes, and drop the now-unused botka_irc-libera-chat,
hal8000_xmpp and hal8000_freenode bags.

Retire nodejs-4; add role[hubot] to nodejs-5 34db4dee8f

Drop the nodejs-4 node object (host is powered off) and give nodejs-5
role[hubot], so hubot runs on Node 22 there.

Document remaining items for hubot migration 0405459db0

raucao changed title from ~~eMudhra CA follow-up: patched Node.js releases, Cinc, CAcert; hubot + kredits-github off nodejs-4~~ to eMudhra CA follow-up: patched Node.js releases, Cinc, CAcert; migrating off nodejs-4

2026-10-28 14:23:43 +00:00

raucao added the kredits-3 dev environment ops security enhancement labels 2026-10-28 14:25:10 +00:00

raucao requested review from greg 2026-10-28 14:25:41 +00:00

greg approved these changes

2026-10-28 15:01:20 +00:00

greg left a comment

👍

raucao merged commit 96aa2e0007 into master

2026-10-28 15:19:22 +00:00

raucao deleted branch chore/emudhra

2026-10-28 15:19:22 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/chef#687