liquor_cabinet: preload jemalloc into the Rainbows workers

The Rainbows master and workers run on glibc malloc, which retains and
fragments heap in long-lived workers. Preload jemalloc (libjemalloc2) so the
aged per-worker RSS stays flat instead of growing.

Also restart liquor-cabinet.service when its systemd unit changes:
systemd_unit only reloads the daemon, so an env-only change would not take
effect until the next deploy or reboot.

Cookbook version 0.3.0 -> 0.4.0.

site-cookbooks/liquor_cabinet/CHANGELOG.md

@@ -2,6 +2,13 @@
 
 This file is used to list changes made in each version of the liquor_cabinet cookbook.
 
+## 0.4.0
+
+- Preload jemalloc into the Rainbows master and workers (`LD_PRELOAD` on the
+  `liquor-cabinet.service` unit) to reduce heap fragmentation and growth.
+- Restart the service when the systemd unit changes, so the new environment
+  actually takes effect (`systemd_unit` only reloads the daemon).
+
 ## 0.3.0
 
 - Pin Liquor Cabinet to the 2.0.0-rc.2 release.

site-cookbooks/liquor_cabinet/metadata.rb

@@ -3,7 +3,7 @@ maintainer 'Kosmos Developers'
 maintainer_email 'ops@kosmos.org'
 license 'MIT'
 description 'Installs/configures the Liquor Cabinet remoteStorage API server'
-version '0.3.0'
+version '0.4.0'
 chef_version '>= 18.2'
 issues_url 'https://gitea.kosmos.org/kosmos/chef/issues'
 # source_url 'https://gitea.kosmos.org/kosmos/chef'

site-cookbooks/liquor_cabinet/recipes/default.rb

@@ -104,6 +104,9 @@ template "/etc/rainbows/#{app_name}.rb" do
   notifies :restart, "service[#{app_name}]", :delayed
 end
 
+# jemalloc for the Rainbows workers (referenced by LD_PRELOAD in the unit below)
+package 'libjemalloc2'
+
 systemd_unit "#{app_name}.service" do
   content({
     Unit: {
@@ -115,7 +118,12 @@ systemd_unit "#{app_name}.service" do
       Type: "simple",
       User: deploy_user,
       WorkingDirectory: deploy_path,
-      Environment: "RACK_ENV=#{rack_env}",
+      Environment: [
+        "RACK_ENV=#{rack_env}",
+        # jemalloc keeps the long-lived Rainbows workers from fragmenting/
+        # retaining heap the way glibc malloc does.
+        "LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"
+      ],
       ExecStart: "#{bundle_path} exec rainbows -c /etc/rainbows/#{app_name}.rb -E #{rack_env}",
       PIDFile: "#{deploy_path}/tmp/rainbows.pid",
       TimeoutSec: "10",
@@ -130,8 +138,12 @@ systemd_unit "#{app_name}.service" do
   action [:create, :enable]
 end
 
+# `systemd_unit` only reloads the daemon and never restarts a running unit, so
+# unit/env changes would not take effect until the next repo change or reboot.
+# Restart whenever the unit file changes (e.g. to pick up LD_PRELOAD).
 service app_name do
   action [:enable, :start]
+  subscribes :restart, "systemd_unit[#{app_name}.service]", :delayed
 end
 
 if node[app_name]['ufw_source_allowed']