Merge branch 'master' into live

Prevent future mistakes

AGENTS.md

@@ -7,16 +7,13 @@ ejabberd, Discourse, Mastodon, remoteStorage, Nostr, LNDHub, and BTCPay.
 ## Development environment
 
 Development runs in Docker Compose — run all commands against the `web` container.
-Start services: `docker compose up` (web, ldap, redis, garage, liquor-cabinet, strfry).
+Start services: `docker compose up` (web, ldap, redis, minio, liquor-cabinet, strfry).
 The `web` service runs `bin/dev` (foreman: Puma + Tailwind CSS watcher) and embeds
 Solid Queue workers (`SOLID_QUEUE_IN_PUMA=true`).
 
-First-time LDAP and database setup is automated: the `ldap-init` service
+First-time LDAP setup (after creating the 389ds backend once):
-creates the 389ds back-end, then the `web` entrypoint seeds LDAP and the
+`docker compose exec ldap dsconf localhost backend create --suffix="dc=kosmos,dc=org" --be_name="dev"`
-databases on first start and runs `db:prepare` on every boot. Manual
+then `docker compose run web bin/rails ldap:setup`.
-equivalents: `docker compose exec ldap dsconf localhost backend create
---suffix="dc=kosmos,dc=org" --be-name="dev"` and `docker compose run --rm web
-bin/rails ldap:setup`.
 
 ## Common commands (prefix with `docker compose exec web`)
 

README.md

@@ -14,14 +14,13 @@ so:
 
 1. Make sure [Docker Compose is installed][1] and Docker is running (included in
    Docker Desktop)
-2. Run `docker compose up --build` and wait until all services have started
+3. Run `docker compose up --build` and wait until all services have started
    (389ds might take an extra minute to be ready). This will take a while when
    running for the first time, so you might want to do something else in the
    meantime.
-
+4. `docker-compose exec ldap dsconf localhost backend create --suffix="dc=kosmos,dc=org" --be-name="dev"`
-On the first start, the `ldap-init` service creates the 389ds back-end, and the
+5. `docker compose run web rails ldap:setup`
-`web` container then seeds the LDAP directory and the databases automatically.
+6. `docker compose run web rails db:setup`
-On every start, `web` also applies any pending database migrations.
 
 After these steps, you should have a working Rails app with a handful of test
 users running on [http://localhost:3000](http://localhost:3000).
@@ -72,12 +71,15 @@ containers you want to run to the `up` command, like so:
 
 #### LDAP server
 
-On first start, the `ldap-init` service creates the dirsrv back-end
+After creating the Docker container for the first time (or after deleting it),
-automatically, and the `web` container then seeds it with development entries.
+you need to run the following command once, in order to create the dirsrv
-To do either step manually (for example, after changing the setup), run:
+back-end:
 
-    docker compose exec ldap dsconf localhost backend create --suffix="dc=kosmos,dc=org" --be-name="dev"
+    docker-compose exec ldap dsconf localhost backend create --suffix="dc=kosmos,dc=org" --be-name="dev"
-    docker compose run --rm web bin/rails ldap:setup
+
+Now you can seed the back-end with data using this Rails task:
+
+    bundle exec rails ldap:setup
 
 The setup task will first delete any existing entries in the directory tree
 ("dc=kosmos,dc=org"), and then create our development entries.
@@ -86,42 +88,20 @@ Note that all 389ds data is stored in the `389ds-data` volume. So if you want
 to start over with a fresh installation, delete both that volume as well as the
 container.
 
-To reset the development environment completely, remove all volumes plus the
+#### Minio / remoteStorage
-generated database files and the first-run marker, then start over:
 
-    docker compose down -v
+If you want to run remoteStorage accounts locally, you will have to create the
-    rm -f db/*.sqlite3 tmp/.setup-complete
+respective bucket first. With the `minio` container running (run by default
-    docker compose up --build
+when using Docker Compose), follow these steps:
 
-#### Garage / remoteStorage
+* `docker compose up web redis minio liquor-cabinet`
-
+* Head to http://localhost:9001 and log in with user `minioadmin`, password
-remoteStorage accounts use the `garage` S3-compatible object store. On first
+  `minioadmin`
-start, Garage automatically configures a single-node cluster and creates the
+* Create a new bucket called `remotestorage` (or whatever you
-`remotestorage` bucket together with a `dev-key1` access key (secret
+  change the `S3_BUCKET` config to)
-`1234567890123456`), so no manual setup is required.
+* Create a new key with ID "dev-key" and secret "123456789" (or whatever you
-
+  change `S3_ACCESS_KEY` and `S3_SECRET_KEY` to). Leave the policy field empty,
-If you want to run remoteStorage accounts locally, the `garage` container is
+  as it will automatically allow access to the bucket you created.
-started by default when using Docker Compose. To run just the remoteStorage
-stack:
-
-* `docker compose up web redis garage liquor-cabinet`
-
-The S3 API is available at http://localhost:3900 (region `garage`). If you want
-to start over with a fresh storage, delete the `garage-data` volume as well as
-the container.
-
-#### Accessing remoteStorage from another machine
-
-remoteStorage clients force HTTPS for any host except `localhost`, and browsers
-block plain-HTTP requests to a LAN IP as mixed content. To connect to the dev
-remoteStorage from a browser on another machine (including production apps such
-as Inspektor), forward the ports over SSH and connect as `localhost`:
-
-    ssh -N -L 3000:localhost:3000 -L 4567:localhost:4567 <user>@<dev-host>
-
-Then use `<user>@localhost:3000` as the remoteStorage address in the client.
-WeFinger discovery and storage requests are served through the forwarded ports,
-so no TLS setup is needed.
 
 ### Adding npm modules to use with Stimulus controllers
 

app/controllers/services/remotestorage_controller.rb

@@ -1,7 +1,7 @@
 class Services::RemotestorageController < Services::BaseController
   before_action :authenticate_user!
   before_action :require_service_available
-  before_action :require_service_enabled
+  before_action :require_feature_enabled
 
   # Dashboard
   def show
@@ -17,8 +17,8 @@ class Services::RemotestorageController < Services::BaseController
       http_status :not_found unless Setting.remotestorage_enabled?
     end
 
-    def require_service_enabled
+    def require_feature_enabled
-      unless current_user.service_enabled?(:remotestorage)
+      unless Flipper.enabled?(:remotestorage, current_user)
         http_status :forbidden
       end
     end

app/controllers/services/rs_auths_controller.rb

@@ -1,7 +1,8 @@
 class Services::RsAuthsController < Services::BaseController
   before_action :authenticate_user!
-  before_action :require_service_enabled
+  before_action :require_feature_enabled
   before_action :require_service_available
+  # before_action :require_service_enabled
   before_action :find_rs_auth, only: [:destroy, :launch_app]
 
   def index
@@ -22,12 +23,9 @@ class Services::RsAuthsController < Services::BaseController
   end
 
   def launch_app
-    user_address =
+    user_address = Rails.env.development? ?
-      if Rails.env.development?
+      "#{current_user.cn}@localhost:3000" :
-        "#{current_user.cn}@#{request.host_with_port}"
+      current_user.address
-      else
-        current_user.address
-      end
 
     launch_url = "#{@auth.launch_url}#remotestorage=#{user_address}"
 
@@ -36,8 +34,8 @@ class Services::RsAuthsController < Services::BaseController
 
   private
 
-    def require_service_enabled
+    def require_feature_enabled
-      unless current_user.service_enabled?(:remotestorage)
+      unless Flipper.enabled?(:remotestorage, current_user)
         http_status :forbidden
       end
     end

app/controllers/webfinger_controller.rb

@@ -88,14 +88,8 @@ class WebfingerController < WellKnownController
   end
 
   def remotestorage_link
-    auth_url =
+    auth_url = new_rs_oauth_url(@username, host: Setting.rs_accounts_domain)
-      if Rails.env.development?
+    storage_url = "#{Setting.rs_storage_url}/#{@username}"
-        new_rs_oauth_url(@username)
-      else
-        new_rs_oauth_url(@username, host: Setting.rs_accounts_domain)
-      end
-
-    storage_url = "#{remotestorage_storage_base_url}/#{@username}"
 
     {
       rel: "http://tools.ietf.org/id/draft-dejong-remotestorage",
@@ -109,12 +103,4 @@ class WebfingerController < WellKnownController
       }
     }
   end
-
-  def remotestorage_storage_base_url
-    return Setting.rs_storage_url unless Rails.env.development?
-
-    uri = URI.parse(Setting.rs_storage_url)
-    uri.host = request.host
-    uri.to_s
-  end
 end

app/views/admin/users/show.html.erb

@@ -305,7 +305,7 @@
           <td>remoteStorage</td>
           <td>
             <%= render FormElements::ToggleComponent.new(
-              enabled: @services_enabled.include?("remotestorage"),
+              enabled: Flipper.enabled?(:remotestorage, current_user) && @services_enabled.include?("remotestorage"),
               input_enabled: false
             ) %>
           </td>

app/views/dashboard/index.html.erb

@@ -43,7 +43,8 @@
           <% end %>
         </div>
       <% end %>
-      <% if Setting.remotestorage_enabled? %>
+      <% if Setting.remotestorage_enabled? &&
+            Flipper.enabled?(:remotestorage, current_user) %>
         <div class="border border-gray-300 rounded-md hover:border-gray-400
                     bg-[length:80%] bg-[center_top_-156px] bg-no-repeat
                     bg-[url(/img/logos/icon_remotestorage.svg)]">

app/views/shared/_sidenav_settings.html.erb

@@ -25,7 +25,8 @@
   active: @settings_section.to_s == "lightning"
 ) %>
 <% end %>
-<% if Setting.remotestorage_enabled? %>
+<% if Setting.remotestorage_enabled? &&
+      Flipper.enabled?(:remotestorage, current_user) %>
 <%= render SidenavLinkComponent.new(
   name: "Storage", path: setting_path(:remotestorage), icon: "remotestorage",
   active: @settings_section.to_s == "remotestorage"

config/sidekiq.yml

@@ -1,4 +1,6 @@
 :concurrency: 2
+production:
+  :concurrency: 10
 :queues:
   - default
   - mailers

db/seeds.rb

@@ -1,9 +1,8 @@
+require 'sidekiq/testing'
+
 ldap = LdapService.new
 
-original_queue_adapter = ActiveJob::Base.queue_adapter
+Sidekiq::Testing.inline! do
-ActiveJob::Base.queue_adapter = :inline
-
-begin
   ldap.delete_all_users!
 
   puts "Create user: admin"
@@ -26,6 +25,4 @@ begin
       password: "user is user", confirmed: true
     })
   end
-ensure
-  ActiveJob::Base.queue_adapter = original_queue_adapter
 end

docker-compose.yml

@@ -12,20 +12,6 @@ services:
       DS_DM_PASSWORD: passthebutter
       SUFFIX_NAME: "dc=kosmos,dc=org"
 
-  ldap-init:
-    image: 4teamwork/389ds:latest
-    networks:
-      - internal_network
-    volumes:
-      - ./docker/ldap-init.sh:/ldap-init.sh:ro
-    environment:
-      LDAP_ADMIN_PASSWORD: passthebutter
-      LDAP_SUFFIX: "dc=kosmos,dc=org"
-    depends_on:
-      ldap:
-        condition: service_healthy
-    command: ["/bin/sh", "/ldap-init.sh"]
-
   redis:
     restart: always
     image: redis:7-alpine
@@ -40,7 +26,7 @@ services:
   web:
     build: .
     tty: true
-    command: ["bash", "docker/web-entrypoint.sh"]
+    command: bash -c "rm -f /akkounts/tmp/pids/server.pid; bin/dev"
     volumes:
       - .:/akkounts
       - /akkounts/node_modules
@@ -71,28 +57,20 @@ services:
       NOSTR_PRIVATE_KEY: 7c3ef7e448505f0615137af38569d01807d3b05b5005d5ecf8aaafcd40323cea
       NOSTR_RELAY_URL: ws://strfry:7777
     depends_on:
-      ldap:
+      - ldap
-        condition: service_started
+      - redis
-      ldap-init:
-        condition: service_completed_successfully
-      redis:
-        condition: service_started
 
-  garage:
+  minio:
-    image: dxflrs/garage:v2.4.1
+    image: quay.io/minio/minio:latest
-    command: ["/garage", "server", "--single-node", "--default-bucket"]
+    command: "server /data --console-address ':9001'"
     networks:
       - external_network
       - internal_network
     ports:
-      - "3900:3900"
+      - "9000:9000"
+      - "9001:9001"
     volumes:
-      - ./docker/garage/garage.toml:/etc/garage.toml:ro
+      - minio-data:/data
-      - garage-data:/var/lib/garage
-    environment:
-      GARAGE_DEFAULT_ACCESS_KEY: dev-key1
-      GARAGE_DEFAULT_SECRET_KEY: "1234567890123456"
-      GARAGE_DEFAULT_BUCKET: remotestorage
 
   liquor-cabinet:
     image: gitea.kosmos.org/5apps/liquor-cabinet:2.0.0-rc.1
@@ -101,21 +79,17 @@ services:
       - internal_network
     ports:
       - "4567:4567"
-    volumes:
-      - ./docker/liquor-cabinet/rainbows.conf.rb:/etc/liquor-cabinet/rainbows.conf.rb:ro
-    command: ["bundle", "exec", "rainbows", "-c", "/etc/liquor-cabinet/rainbows.conf.rb", "--listen", "0.0.0.0:4567"]
     environment:
       RACK_ENV: staging
       REDIS_HOST: redis
       REDIS_PORT: 6379
       REDIS_DB: 1
-      S3_ENDPOINT: http://garage:3900
+      S3_ENDPOINT: http://minio:9000
-      S3_REGION: garage
+      S3_ACCESS_KEY: dev-key
-      S3_ACCESS_KEY: dev-key1
+      S3_SECRET_KEY: 123456789
-      S3_SECRET_KEY: "1234567890123456"
       S3_BUCKET: remotestorage
     depends_on:
-      - garage
+      - minio
       - redis
 
   strfry:
@@ -153,7 +127,7 @@ networks:
 volumes:
   389ds-data:
     driver: local
-  garage-data:
+  minio-data:
     driver: local
   redis-data:
     driver: local

docker/garage/garage.toml

@@ -1,14 +0,0 @@
-metadata_dir = "/var/lib/garage/meta"
-data_dir = "/var/lib/garage/data"
-db_engine = "sqlite"
-
-replication_factor = 1
-
-rpc_bind_addr = "[::]:3901"
-rpc_public_addr = "127.0.0.1:3901"
-rpc_secret = "1799bccfd7411eddcf9ebd316bc1f5287ad12a68094e1c6ac6abde7e6feae1ec"
-
-[s3_api]
-s3_region = "garage"
-api_bind_addr = "[::]:3900"
-root_domain = ".s3.garage.localhost"

docker/ldap-init.sh

@@ -1,14 +0,0 @@
-#!/bin/sh
-set -e
-
-SUFFIX="${LDAP_SUFFIX:-dc=kosmos,dc=org}"
-URI="ldap://ldap:3389"
-
-if dsconf -D "cn=Directory Manager" -w "$LDAP_ADMIN_PASSWORD" "$URI" \
-     backend suffix list --suffix 2>/dev/null | grep -Fqx "$SUFFIX"; then
-  echo "LDAP backend for $SUFFIX already exists, skipping."
-else
-  echo "Creating LDAP backend for $SUFFIX..."
-  dsconf -D "cn=Directory Manager" -w "$LDAP_ADMIN_PASSWORD" "$URI" \
-    backend create --suffix "$SUFFIX" --be-name dev
-fi

docker/liquor-cabinet/rainbows.conf.rb

@@ -1,5 +0,0 @@
-Rainbows! do
-  use :ThreadPool
-  worker_connections 16
-  client_max_body_size 100 * 1024 * 1024
-end

docker/web-entrypoint.sh

@@ -1,14 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-if [ ! -f tmp/.setup-complete ]; then
-  echo "First start: setting up LDAP entries and databases..."
-  bin/rails ldap:setup
-  bin/rails db:setup
-  touch tmp/.setup-complete
-else
-  bin/rails db:prepare
-fi
-
-rm -f tmp/pids/server.pid
-exec bin/dev

package.json

@@ -11,7 +11,7 @@
     "postcss-preset-env": "^7.8.3",
     "tailwindcss": "^3.4.0"
   },
-  "version": "0.11.1",
+  "version": "0.11.0",
   "scripts": {
     "build:css:tailwind": "bun ./node_modules/tailwindcss/lib/cli.js --postcss -i ./app/assets/stylesheets/application.tailwind.css -o ./app/assets/builds/application.css",
     "build:css": "bun run build:css:tailwind"

spec/controllers/services/rs_auths_controller_spec.rb

@@ -6,9 +6,7 @@ RSpec.describe Services::RsAuthsController, type: :controller do
   before do
     allow_any_instance_of(AppCatalog::WebApp).to receive(:update_metadata).and_return(true)
     allow_any_instance_of(RemoteStorageAuthorization).to receive(:remove_token_expiry_job).and_return(nil)
-    allow_any_instance_of(LdapService).to receive(:fetch_users).and_return([
+    allow_any_instance_of(Flipper).to receive(:enabled?).and_return(true)
-      { services_enabled: ["remotestorage"] }
-    ])
   end
 
   describe "GET /services/storage/rs_auths/:id/launch_app" do
@@ -37,20 +35,6 @@ RSpec.describe Services::RsAuthsController, type: :controller do
           expect(response).to redirect_to(launch_url)
         end
       end
-
-      context "when remoteStorage is not enabled for the user" do
-        before do
-          allow_any_instance_of(LdapService).to receive(:fetch_users).and_return([
-            { services_enabled: [] }
-          ])
-
-          get :launch_app, params: { id: 1 }
-        end
-
-        it "responds with forbidden" do
-          expect(response).to have_http_status(:forbidden)
-        end
-      end
     end
   end
 end

spec/models/remote_storage_authorization_spec.rb

@@ -11,7 +11,6 @@ RSpec.describe RemoteStorageAuthorization, type: :model do
   end
 
   describe "#create" do
-    before(:each) { redis_rs_delete_keys("authorizations:*") }
     after(:each) { clear_enqueued_jobs }
     after(:all)  { redis_rs_delete_keys("authorizations:*") }
 
@@ -28,9 +27,10 @@ RSpec.describe RemoteStorageAuthorization, type: :model do
     end
 
     it "stores a token in redis" do
-      auth
+      user_auth_keys = redis_rs.keys("authorizations:#{user.cn}:*")
+      expect(user_auth_keys.length).to eq(1)
 
-      authorizations = redis_rs.smembers("authorizations:#{user.cn}:#{auth.token}")
+      authorizations = redis_rs.smembers(user_auth_keys.first)
       expect(authorizations.sort).to eq(%w(documents photos contacts:rw videos:r tasks/work:r).sort)
     end