Upgrade Mastodon from 4.3 to 4.7, deploy on new Ubuntu 24.04 VM #681

2026-10-08T12:34:23Z

raucao commented

2026-10-08 12:34:23 +00:00

This took us way longer than expected, because in order to solve all the little upgrade tasks and issues, we refactored the cookbook entirely to make upgrades as well as VM migrations much easier in the future.

The cookbook now allows to install dependencies separately from deploying Mastodon, and you can also pre-build the new version (app dependencies, asset precompilation, etc.) before doing an actual deployment (useful for new VMs). It will also automatically run pre- and post-deploy migrations by itself now. And builds and deployments are now tied to the git revision and can be run again if something fails on the way (without having to reset the repo and having everything tied to the git sync).

Our production Mastodon branch also had to be reworked, because both theme integration as well as the custom URLs for account management have changed upstream. And we added a deploy key to the setup, since the Mastodon repo is now set to private (because it was overrun by AI scrapers before).

Also upgrades libretranslate to the latest stable version.

Tested/running on mastodon-4. Search is still in the process of indexing posts, because we opted for a full backround re-index instead of moving ES files between machines and ES versions.

closes #646

This took us way longer than expected, because in order to solve all the little upgrade tasks and issues, we refactored the cookbook entirely to make upgrades as well as VM migrations much easier in the future. The cookbook now allows to install dependencies separately from deploying Mastodon, and you can also pre-build the new version (app dependencies, asset precompilation, etc.) before doing an actual deployment (useful for new VMs). It will also automatically run pre- and post-deploy migrations by itself now. And builds and deployments are now tied to the git revision and can be run again if something fails on the way (without having to reset the repo and having everything tied to the git sync). Our production Mastodon branch also had to be reworked, because both theme integration as well as the custom URLs for account management have changed upstream. And we added a deploy key to the setup, since the Mastodon repo is now set to private (because it was overrun by AI scrapers before). Also upgrades libretranslate to the latest stable version. Tested/running on `mastodon-4`. Search is still in the process of indexing posts, because we opted for a full backround re-index instead of moving ES files between machines and ES versions. closes #646

raucao added the

service

mastodon

kredits-2 ops labels 2026-10-08 12:34:23 +00:00

raucao self-assigned this 2026-10-08 12:34:23 +00:00

greg was assigned by raucao

2026-10-08 12:34:23 +00:00

raucao added 21 commits 2026-10-08 12:34:23 +00:00

Upgrade Mastodon to 4.7 b60ca687ec

Bump the production branch to 4.7 and adjust for the changes between 4.3 and 4.7:

- Node 24.21.0 and Ruby 4.0.7 (via ruby-build v20260924)
- Redis 7.4.11 (Mastodon 4.5 requires >= 7.0)
- Replace ImageMagick with libvips (required since 4.6) and libidn11 with libidn
- Split database migrations into pre-/post-deployment phases and rebuild
  the Elasticsearch accounts index mappings (required since 4.4)
- Remove the OTP_SECRET environment variable (removed in 4.4)
- Disable email subscriptions (new optional feature in 4.6) and force the
  default locale (DEFAULT_LOCALE no longer overrides it since 4.4)
- Add the new fasp Sidekiq queue
- Enable corepack for yarn instead of the removed no-arg 'corepack prepare'

Add mastodon-4 node and client config 735145f5a9

Split Mastodon runtime deps from deployment, use Redis cluster 591d6dc4ec

Add a kosmos-mastodon::dependencies recipe with everything needed to run Mastodon (Node, Ruby, libvips, Elasticsearch, packages) but without the app deployment. The default recipe includes it and only runs the deployment when node['kosmos-mastodon']['deploy'] is true, so a VM can be prepared ahead of a maintenance window.

Stop using the local redisio instance and connect to the external Redis cluster (redis_server role, db 2, password from the credentials data bag) instead. Remove the redisio service dependencies from the systemd units and drop the redisio cookbook dependency.

Update mastodon-4 node to dependency-only deployment fef57f1ee1

Move Mastodon deploy flag to the mastodon role 9fe5def5f9

Set kosmos-mastodon.deploy to false as a role default, making it overridable per node, and drop the node-level attribute.

Create search indices during deploy, import in background 8a11cd88f7

Mastodon 4.4+ can create/upgrade the Elasticsearch indices and mappings without importing data. Do that synchronously during the deployment so search does not fail on a missing index, then populate the (potentially very long) import via a systemd unit started with --no-block so the maintenance window is not extended.

Drop the java cookbook dependency b09808b12e

The java cookbook's openjdk recipe unconditionally adds the dead openjdk-r PPA on Ubuntu and cannot be told not to, which breaks on Ubuntu 24.04. Elasticsearch 7.x ships a bundled JDK and no JAVA_HOME is configured, so no system Java is needed.

Install LibreTranslate in a virtualenv ec2645c5da

Ubuntu 24.04's system Python is externally managed (PEP 668) and refuses pip installs. Also bump LibreTranslate from 1.3.8 to 1.9.6, since the former pulls ctranslate2 2.24.0 which has no Python 3.12 wheels.

Remove the backup gem from Mastodon 9dfdf6bc9b

The shared backup cookbook hardcodes postgresql-client-12, which does not exist on Ubuntu 24.04. Drop the Mastodon backup recipe and its role inclusion; the PostgreSQL standby is the safety net. The backup cookbook itself is still used by other services.

Add a build phase, make deployment revision-driven fb0c7b2b1c

Split the deployment into kosmos-mastodon::build (checkout + bundle/yarn/assets, no database) and kosmos-mastodon::deploy (migrations + services), dispatched by the new 'build' attribute. Both phases are idempotent per checked-out revision via stamps, and the migration/restart/search-index chain is now triggered by the deployed revision instead of the git resource, so it still runs when the build was pre-staged.

Pre-stage the Mastodon 4.7 build on mastodon-4 d05f00dae2

Clone the private Mastodon repo with an SSH deploy key abfd654ae5

The repository is private, so use git@gitea.kosmos.org with a read-only deploy key stored in credentials/mastodon (repo_deploy_key). The pinned gitea host key is an attribute.

Create the mastodon home directory in the dependencies recipe a3877e620e

The user resource does not manage the home directory, so removing /opt/mastodon broke the build phase when it tried to create /opt/mastodon/.ssh.

Add repo deploy key aeeb900f14

Ensure the SSH deploy key ends with a newline b0718a5dac

OpenSSH's PEM parser rejects a private key file without a trailing newline ('error in libcrypto').

Replace the removed bundler --deployment flag 0696d03374

Bundler 4 (shipped with Ruby 4) removed --deployment. Use 'bundle config set --local deployment true' (frozen + vendor/bundle) and the 'without' setting instead, in the Mastodon, akkounts and liquor-cabinet cookbooks.

Give the mastodon user a /home/mastodon home 1b564c285d

The mastodon user's home was /opt/mastodon, i.e. the clone destination, so writing the SSH deploy key into ~/.ssh made the directory non-empty and Chef's git resource silently skipped cloning (it only clones into an empty directory). Use a dedicated /home/mastodon home for the user and keep the clone destination separate. Also clear a non-git (partial) clone destination before cloning.

Record revision stamps as the mastodon user af4983d747

The stamp files' lazy git call ran as root, which git rejects on the mastodon-owned repository (dubious ownership), silently writing an empty stamp. Use execute resources with the mastodon user instead so git works and the revision is recorded.

Declare the search deploy unit before the migration chain 8ac0b965b2

The migration chain starts mastodon-search-deploy.service immediately, but the systemd_unit resource was declared later in the recipe, so the unit did not exist yet and systemctl failed with 'Unit not found'.

Deploy Mastodon from now on 2bb5b582ec

The mastodon role now always deploys (deploy=true), so a plain converge rebuilds and restarts on new production-4.7 commits.

Update Mastodon data bag, nodes 63e4401715

raucao added kredits-3 and removed kredits-2 labels 2026-10-08 12:36:44 +00:00

raucao requested review from greg 2026-10-08 12:39:56 +00:00

greg approved these changes

2026-10-08 12:57:46 +00:00

greg left a comment

👍

raucao merged commit 17285bc2c0 into master

2026-10-08 13:02:46 +00:00

raucao deleted branch chore/upgrade_mastodon

2026-10-08 13:02:47 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/chef#681