liquor_cabinet: preload jemalloc into the Rainbows workers #704

2026-11-14T06:25:01Z

raucao commented

2026-11-14 06:25:01 +00:00

Preloads jemalloc (libjemalloc2) into the Rainbows master and workers via LD_PRELOAD on liquor-cabinet.service, so the long-lived workers stop retaining and fragmenting heap the way glibc malloc does.

Also fixes a latent issue: systemd_unit only reloads the daemon and never restarts a running unit, so an env-only change would not have taken effect until the next deploy or reboot. The service now subscribes to the unit and restarts whenever it changes.

Deployed to lq-1 and lq-2 and verified: LD_PRELOAD present, jemalloc mapped into both workers, still 2 workers, service active, storage.kosmos.org returning 401. Cold-start ΣPSS is 50 MB (down from ~250 MB on 10-day-old glibc workers); re-measuring after 24 h to confirm it stays flat.

Cookbook version 0.3.0 → 0.4.0. Includes a routine nodes submodule update.

Preloads jemalloc (`libjemalloc2`) into the Rainbows master and workers via `LD_PRELOAD` on `liquor-cabinet.service`, so the long-lived workers stop retaining and fragmenting heap the way glibc malloc does. Also fixes a latent issue: `systemd_unit` only reloads the daemon and never restarts a running unit, so an env-only change would not have taken effect until the next deploy or reboot. The service now subscribes to the unit and restarts whenever it changes. Deployed to `lq-1` and `lq-2` and verified: `LD_PRELOAD` present, jemalloc mapped into both workers, still 2 workers, service active, `storage.kosmos.org` returning 401. Cold-start ΣPSS is 50 MB (down from ~250 MB on 10-day-old glibc workers); re-measuring after 24 h to confirm it stays flat. Cookbook version 0.3.0 → 0.4.0. Includes a routine `nodes` submodule update.

raucao added 2 commits 2026-11-14 06:25:01 +00:00

Update nodes 88ef468015

liquor_cabinet: preload jemalloc into the Rainbows workers afc0453166

The Rainbows master and workers run on glibc malloc, which retains and
fragments heap in long-lived workers. Preload jemalloc (libjemalloc2) so the
aged per-worker RSS stays flat instead of growing.

Also restart liquor-cabinet.service when its systemd unit changes:
systemd_unit only reloads the daemon, so an env-only change would not take
effect until the next deploy or reboot.

Cookbook version 0.3.0 -> 0.4.0.

raucao added the kredits-1 ops enhancement labels 2026-11-14 06:26:12 +00:00

raucao merged commit b5d3f35596 into master

2026-11-14 06:26:36 +00:00

raucao deleted branch chore/liquor_cabinet

2026-11-14 06:26:36 +00:00

raucao added the

service

remotestorage

label 2026-11-14 06:26:50 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/chef#704