I have solved the last remaining issue, by moving the upload config to the respective vhosts config files and configuring a subdirectory (just hash of respective user address domain).
Here's my successful production test: https://uploads.kosmos.chat/8af2c77/791c7ed148e453f934ef56e1a4acb79a30845f0f/iMNpkgBBCLOpmQoHX5TqhQKUbZG4gHHDlPm0cva2/japan-map-in-english.jpeg
Ugh, that's dumb: ejabberd accepted my config with reload_config and then just failed when I removed the second cluster node, as well as when actually restarting it. Saying it cannot configure http uploads without @host variable when using multiple vhosts. :/
There's an option to configure subdirectory levels in upload.pm, but I've been at it for almost 12 hours straight now, so that'll have to wait until tomorrow.
I tried it with the new cluster node (after fixing iptables being weird on draco after the recent reboot), and when I got it working I manually added the config on andromeda, which we cannot provision with chef anymore.
All uploads now go to uploads-1 on draco! Here's my first production test:
I commented that entry in haproxy.cfg for now, meaning audio/video calls via XMPP signaling are more likely to fail until this is fixed.
Actually, we're not using the node on draco in production yet, so there's currently no problem. It only becomes an issue when we roll out the cluster.