Set up an instance of Mastodon for Kosmos

Refs #19

Use new application cookbook, update our cookbooks

site-cookbooks/kosmos-mastodon/CHANGELOG.md

@@ -0,0 +1,11 @@
+# kosmos-mastodon CHANGELOG
+
+This file is used to list changes made in each version of the kosmos-mastodon cookbook.
+
+## 0.1.0
+- [your_name] - Initial release of kosmos-mastodon
+
+- - -
+Check the [Markdown Syntax Guide](http://daringfireball.net/projects/markdown/syntax) for help with Markdown.
+
+The [Github Flavored Markdown page](http://github.github.com/github-flavored-markdown/) describes the differences between markdown on github and standard markdown.

site-cookbooks/kosmos-mastodon/README.md

@@ -0,0 +1,80 @@
+# kosmos-mastodon Cookbook
+
+TODO: Enter the cookbook description here.
+
+e.g.
+This cookbook makes your favorite breakfast sandwich.
+
+## Requirements
+
+TODO: List your cookbook requirements. Be sure to include any requirements this cookbook has on platforms, libraries, other cookbooks, packages, operating systems, etc.
+
+e.g.
+### Platforms
+
+- SandwichOS
+
+### Chef
+
+- Chef 12.0 or later
+
+### Cookbooks
+
+- `toaster` - kosmos-mastodon needs toaster to brown your bagel.
+
+## Attributes
+
+TODO: List your cookbook attributes here.
+
+e.g.
+### kosmos-mastodon::default
+
+<table>
+  <tr>
+    <th>Key</th>
+    <th>Type</th>
+    <th>Description</th>
+    <th>Default</th>
+  </tr>
+  <tr>
+    <td><tt>['kosmos-mastodon']['bacon']</tt></td>
+    <td>Boolean</td>
+    <td>whether to include bacon</td>
+    <td><tt>true</tt></td>
+  </tr>
+</table>
+
+## Usage
+
+### kosmos-mastodon::default
+
+TODO: Write usage instructions for each cookbook.
+
+e.g.
+Just include `kosmos-mastodon` in your node's `run_list`:
+
+```json
+{
+  "name":"my_node",
+  "run_list": [
+    "recipe[kosmos-mastodon]"
+  ]
+}
+```
+
+## Contributing
+
+TODO: (optional) If this is a public cookbook, detail the process for contributing. If this is a private cookbook, remove this section.
+
+e.g.
+1. Fork the repository on Github
+2. Create a named feature branch (like `add_component_x`)
+3. Write your change
+4. Write tests for your change (if applicable)
+5. Run the tests, ensuring they all pass
+6. Submit a Pull Request using Github
+
+## License and Authors
+
+Authors: TODO: List authors
+

site-cookbooks/kosmos-mastodon/attributes/default.rb

@@ -0,0 +1,4 @@
+node.default["kosmos-mastodon"]["directory"]      = "/opt/mastodon"
+node.default["kosmos-mastodon"]["puma_port"]      = 3000
+node.default["kosmos-mastodon"]["streaming_port"] = 4000
+node.default["kosmos-mastodon"]["server_name"]    = "kosmos.social"

site-cookbooks/kosmos-mastodon/metadata.rb

@@ -0,0 +1,15 @@
+name             'kosmos-mastodon'
+maintainer       'Kosmos'
+maintainer_email 'mail@kosmos.org'
+license          'All rights reserved'
+description      'Installs/Configures kosmos-mastodon'
+long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
+version          '0.1.0'
+
+depends "kosmos-nginx"
+depends "kosmos-nodejs"
+depends "kosmos-ruby"
+depends "application_ruby"
+depends "application_javascript"
+depends "postgresql"
+depends "database"

site-cookbooks/kosmos-mastodon/recipes/default.rb

@@ -0,0 +1,156 @@
+#
+# Cookbook Name:: kosmos-mastodon
+# Recipe:: default
+#
+# Copyright 2017, Kosmos
+#
+# All rights reserved - Do Not Redistribute
+#
+
+include_recipe "kosmos-nodejs"
+include_recipe "kosmos-ruby"
+node.override['postgresql']['enable_pgdg_apt'] = false
+include_recipe "postgresql::server"
+include_recipe "postgresql::ruby"
+unless node.chef_environment == "development"
+  node.override['postgresql']['config_pgtune']['db_type'] = "web"
+  include_recipe "postgresql::config_pgtune"
+end
+
+postgresql_database 'mastodon' do
+  connection(
+    :host      => '127.0.0.1',
+    :port      => 5432,
+    :username  => 'postgres',
+    :password  => node['postgresql']['password']['postgres']
+  )
+  action :create
+end
+
+mastodon_path = node["kosmos-mastodon"]["directory"]
+
+group "mastodon" do
+  gid 62786
+end
+
+user "mastodon" do
+  comment "mastodon user"
+  uid 62786
+  gid 62786
+  shell "/bin/bash"
+  home mastodon_path
+end
+
+package %w(imagemagick ffmpeg libxml2-dev libxslt1-dev file git curl)
+node_package %w(yarn)
+
+application mastodon_path do
+  owner "mastodon"
+  group "mastodon"
+
+  git do
+    user "mastodon"
+    group "mastodon"
+    repository "https://github.com/67P/mastodon.git"
+    revision   "redis_db"
+  end
+
+  mastodon_credentials = Chef::EncryptedDataBagItem.load('credentials', 'mastodon')
+
+  template ".env.production" do
+    source "env.production.erb"
+    mode  "0640"
+    owner "mastodon"
+    group "mastodon"
+    variables redis_db: 1,
+              redis_actioncable_db: 2,
+              domain: "kosmos.social",
+              paperclip_secret: mastodon_credentials['paperclip_secret'],
+              secret_key_base: mastodon_credentials['secret_key_base'],
+              otp_secret: mastodon_credentials['otp_secret'],
+              smtp_login: mastodon_credentials['smtp_user_name'],
+              smtp_password: mastodon_credentials['smtp_password'],
+              smtp_from_address: "mail@kosmos.social",
+              s3_bucket: "kosmos-social",
+              aws_access_key_id: mastodon_credentials['aws_access_key_id'],
+              aws_secret_access_key: mastodon_credentials['aws_secret_access_key'],
+              s3_region: "eu-west-1"
+  end
+
+  directory "#{mastodon_path}/public/.well-known" do
+    owner node['nginx']['user']
+    group node['nginx']['group']
+    recursive true
+  end
+
+  bundle_install do
+    user "mastodon"
+    deployment true
+    without %w{development test}
+  end
+
+  npm_install do
+    user "mastodon"
+  end
+
+  rails do
+    migrate true
+    rails_env "production"
+  end
+
+  execute "systemctl daemon-reload" do
+    command "systemctl daemon-reload"
+    action :nothing
+  end
+
+  # mastodon-web service
+  #
+  template "/lib/systemd/system/mastodon-web.service" do
+    source "mastodon-web.systemd.service.erb"
+    variables user: user,
+              app_dir: mastodon_path,
+              port: node["kosmos-mastodon"]["puma_port"]
+    notifies :run, "execute[systemctl daemon-reload]", :delayed
+    notifies :restart, "service[mastodon-web]", :delayed
+  end
+
+  service "mastodon-web" do
+    action [:enable, :start]
+  end
+
+  # mastodon-sidekiq service
+  #
+  template "/lib/systemd/system/mastodon-sidekiq.service" do
+    source "mastodon-sidekiq.systemd.service.erb"
+    variables user: user,
+              app_dir: mastodon_path
+    notifies :run, "execute[systemctl daemon-reload]", :delayed
+    notifies :restart, "service[mastodon-sidekiq]", :delayed
+  end
+
+  service "mastodon-sidekiq" do
+    action [:enable, :start]
+  end
+
+  # mastodon-streaming service
+  #
+  template "/lib/systemd/system/mastodon-streaming.service" do
+    source "mastodon-streaming.systemd.service.erb"
+    variables user: user,
+              app_dir: mastodon_path,
+              port: node["kosmos-mastodon"]["streaming_port"]
+    notifies :run, "execute[systemctl daemon-reload]", :delayed
+    notifies :restart, "service[mastodon-streaming]", :delayed
+  end
+
+  service "mastodon-streaming" do
+    action [:enable, :start]
+  end
+end
+
+# unless node.chef_environment == "development"
+#   node.override["backup"]["postgresql"]["host"]     = "localhost"
+#   node.override["backup"]["postgresql"]["username"] = "postgres"
+#   node.override["backup"]["postgresql"]["password"] = node['postgresql']['password']['postgres']
+#   include_recipe "backup"
+# end

site-cookbooks/kosmos-mastodon/recipes/nginx.rb

@@ -0,0 +1,48 @@
+#
+# Cookbook Name:: kosmos-mastodon
+# Recipe:: nginx
+#
+# Copyright 2017, Kosmos
+#
+# All rights reserved - Do Not Redistribute
+#
+
+mastodon_path = node["kosmos-mastodon"]["directory"]
+server_name   = node["kosmos-mastodon"]["server_name"]
+
+include_recipe "kosmos-nginx"
+
+directory "/var/www/mastodon/.well-known/acme-challenge" do
+  owner     node["nginx"]["user"]
+  group     node["nginx"]["group"]
+  recursive true
+  action    :create
+end
+
+template "#{node['nginx']['dir']}/sites-available/mastodon" do
+  source 'nginx_conf_mastodon.erb'
+  owner 'www-data'
+  mode 0640
+  variables streaming_port:          node["kosmos-mastodon"]["streaming_port"],
+            puma_port:               node["kosmos-mastodon"]["puma_port"],
+            server_name:             server_name,
+            ssl_cert:                "/etc/letsencrypt/live/#{server_name}/fullchain.pem",
+            ssl_key:                 "/etc/letsencrypt/live/#{server_name}/privkey.pem",
+            mastodon_path:           mastodon_path
+  notifies :reload, 'service[nginx]', :delayed
+end
+
+nginx_site 'mastodon' do
+  enable true
+end
+
+unless node.chef_environment == "development"
+  include_recipe "kosmos-base::letsencrypt"
+  execute "letsencrypt cert for kosmos.social" do
+    command "./certbot-auto certonly --webroot --agree-tos --email ops@5apps.com --webroot-path /var/www/mastodon -d #{server_name} -n"
+    cwd "/usr/local/certbot"
+    not_if { File.exist? "/etc/letsencrypt/live/#{server_name}/fullchain.pem" }
+    notifies :reload, "service[nginx]", :delayed
+    notifies :create, "template[#{node['nginx']['dir']}/sites-available/mastodon]", :immediately
+  end
+end

site-cookbooks/kosmos-mastodon/templates/default/config-environment-production.rb.erb

@@ -0,0 +1,119 @@
+Rails.application.configure do
+  # Settings specified here will take precedence over those in config/application.rb.
+
+  # Code is not reloaded between requests.
+  config.cache_classes = true
+
+  # Eager load code on boot. This eager loads most of Rails and
+  # your application in memory, allowing both threaded web servers
+  # and those relying on copy on write to perform better.
+  # Rake tasks automatically ignore this option for performance.
+  config.eager_load = true
+
+  # Full error reports are disabled and caching is turned on.
+  config.consider_all_requests_local       = false
+  config.action_controller.perform_caching = true
+  config.action_controller.asset_host      = ENV['CDN_HOST'] if ENV.key?('CDN_HOST')
+
+  # Disable serving static files from the `/public` folder by default since
+  # Apache or NGINX already handles this.
+  config.public_file_server.enabled = ENV['RAILS_SERVE_STATIC_FILES'].present?
+
+  # Compress JavaScripts and CSS.
+  config.assets.js_compressor = Uglifier.new(mangle: false)
+  # config.assets.css_compressor = :sass
+
+  # Do not fallback to assets pipeline if a precompiled asset is missed.
+  config.assets.compile = false
+
+  # `config.assets.precompile` and `config.assets.version` have moved to config/initializers/assets.rb
+
+  # Specifies the header that your server uses for sending files.
+  # config.action_dispatch.x_sendfile_header = 'X-Sendfile' # for Apache
+  config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' # for NGINX
+
+  # Allow to specify public IP of reverse proxy if it's needed
+  config.action_dispatch.trusted_proxies = [IPAddr.new(ENV['TRUSTED_PROXY_IP'])] unless ENV['TRUSTED_PROXY_IP'].blank?
+
+  # Force all access to the app over SSL, use Strict-Transport-Security, and use secure cookies.
+  config.force_ssl = false
+
+  # Use the lowest log level to ensure availability of diagnostic information
+  # when problems arise.
+  config.log_level = :debug
+
+  # Prepend all log lines with the following tags.
+  config.log_tags = [:request_id]
+
+  # Use a different logger for distributed setups.
+  # config.logger = ActiveSupport::TaggedLogging.new(SyslogLogger.new)
+
+  # Parse and split the REDIS_URL if passed (used with hosting platforms such as Heroku).
+  # Set ENV variables because they are used elsewhere.
+  if ENV['REDIS_URL']
+    redis_url = URI.parse(ENV['REDIS_URL'])
+    ENV['REDIS_HOST'] = redis_url.host
+    ENV['REDIS_PORT'] = redis_url.port.to_s
+    ENV['REDIS_PASSWORD'] = redis_url.password
+  end
+
+  # Use a different cache store in production.
+  config.cache_store = :redis_store, {
+    host: ENV.fetch('REDIS_HOST') { 'localhost' },
+    port: ENV.fetch('REDIS_PORT') { 6379 },
+    password: ENV.fetch('REDIS_PASSWORD') { false },
+    db: <%= @redis_db %>,
+    namespace: 'cache',
+    expires_in: 20.minutes,
+  }
+
+  # Enable serving of images, stylesheets, and JavaScripts from an asset server.
+  # config.action_controller.asset_host = 'http://assets.example.com'
+
+  # Ignore bad email addresses and do not raise email delivery errors.
+  # Set this to true and configure the email server for immediate delivery to raise delivery errors.
+  # config.action_mailer.raise_delivery_errors = false
+
+  # Enable locale fallbacks for I18n (makes lookups for any locale fall back to
+  # the I18n.default_locale when a translation cannot be found).
+  config.i18n.fallbacks = true
+
+  # Send deprecation notices to registered listeners.
+  config.active_support.deprecation = :notify
+
+  # Use default logging formatter so that PID and timestamp are not suppressed.
+  config.log_formatter = ::Logger::Formatter.new
+
+  # Better log formatting
+  config.lograge.enabled = true
+
+  # Do not dump schema after migrations.
+  config.active_record.dump_schema_after_migration = false
+
+  config.action_mailer.perform_caching = false
+
+  # E-mails
+  config.action_mailer.smtp_settings = {
+    :port           => ENV['SMTP_PORT'],
+    :address        => ENV['SMTP_SERVER'],
+    :user_name      => ENV['SMTP_LOGIN'],
+    :password       => ENV['SMTP_PASSWORD'],
+    :domain         => ENV['SMTP_DOMAIN'] || config.x.local_domain,
+    :authentication => :plain,
+  }
+
+  config.action_mailer.delivery_method = :smtp
+
+  config.react.variant = :production
+
+  config.to_prepare do
+    StatsD.backend = StatsD::Instrument::Backends::NullBackend.new if ENV['STATSD_ADDR'].blank?
+  end
+
+  config.action_dispatch.default_headers = {
+    'Server'                 => 'Mastodon',
+    'X-Frame-Options'        => 'DENY',
+    'X-Content-Type-Options' => 'nosniff',
+    'X-XSS-Protection'       => '1; mode=block',
+  }
+end

site-cookbooks/kosmos-mastodon/templates/default/env.production.erb

@@ -0,0 +1,49 @@
+# Service dependencies
+REDIS_HOST=localhost
+REDIS_PORT=6379
+REDIS_DB=<%= @redis_db %>
+REDIS_ACTIONCABLE_DB=<%= @redis_actioncable_db %>
+DB_HOST=localhost
+DB_USER=postgres
+DB_NAME=mastodon
+DB_PASS=<%= node['postgresql']['password']['postgres'] %>
+DB_PORT=5432
+
+# Federation
+LOCAL_DOMAIN=<%= @domain %>
+LOCAL_HTTPS=true
+
+# Application secrets
+# Generate each with the `rake secret` task (`docker-compose run --rm web rake secret` if you use docker compose)
+PAPERCLIP_SECRET=<%= @paperclip_secret %>
+SECRET_KEY_BASE=<%= @secret_key_base %>
+OTP_SECRET=<%= @otp_secret %>
+
+# Registrations
+# Single user mode will disable registrations and redirect frontpage to the first profile
+# SINGLE_USER_MODE=true
+# Prevent registrations with following e-mail domains
+# EMAIL_DOMAIN_BLACKLIST=example1.com|example2.de|etc
+
+# E-mail configuration
+SMTP_SERVER=smtp.mailgun.org
+SMTP_PORT=587
+SMTP_LOGIN=<%= @smtp_login %>
+SMTP_PASSWORD=<%= @smtp_password %>
+SMTP_FROM_ADDRESS=<%= @smtp_from_address %>
+
+# Optional asset host for multi-server setups
+# CDN_HOST=assets.example.com
+
+# S3 (optional)
+S3_ENABLED=true
+S3_BUCKET=<%= @s3_bucket %>
+AWS_ACCESS_KEY_ID=<%= @aws_access_key_id %>
+AWS_SECRET_ACCESS_KEY=<%= @aws_secret_access_key %>
+S3_REGION=<%= @s3_region %>
+
+# Optional alias for S3 if you want to use Cloudfront or Cloudflare in front
+# S3_CLOUDFRONT_HOST=
+
+# Streaming API integration
+# STREAMING_API_BASE_URL=

site-cookbooks/kosmos-mastodon/templates/default/mastodon-sidekiq.systemd.service.erb

@@ -0,0 +1,17 @@
+[Unit]
+Description=mastodon-sidekiq
+Requires=redis-server.service
+After=redis-server.service
+
+[Service]
+Type=simple
+User=<%= @user %>
+WorkingDirectory=<%= @app_dir %>
+Environment="RAILS_ENV=production"
+Environment="DB_POOL=5"
+ExecStart=/usr/local/bin/bundle exec sidekiq -c 5 -q default -q mailers -q pull -q push
+TimeoutSec=15
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

site-cookbooks/kosmos-mastodon/templates/default/mastodon-streaming.systemd.service.erb

@@ -0,0 +1,15 @@
+Description=mastodon-streaming
+After=network.target
+
+[Service]
+Type=simple
+User=<%= @user %>
+WorkingDirectory=<%= @app_dir %>
+Environment="NODE_ENV=production"
+Environment="PORT=<%= @port %>"
+ExecStart=/usr/local/bin/npm run start
+TimeoutSec=15
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

site-cookbooks/kosmos-mastodon/templates/default/mastodon-web.systemd.service.erb

@@ -0,0 +1,19 @@
+[Unit]
+Description=mastodon-web
+Requires=redis-server.service
+After=redis-server.service
+Requires=postgresql@9.4-main.service
+After=postgresql@9.4-main.service
+
+[Service]
+Type=simple
+User=<%= @user %>
+WorkingDirectory=<%= @app_dir %>
+Environment="RAILS_ENV=production"
+Environment="PORT=3000"
+ExecStart=/usr/local/bin/bundle exec puma -C config/puma.rb
+TimeoutSec=15
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

site-cookbooks/kosmos-mastodon/templates/default/nginx_conf_mastodon.erb

@@ -0,0 +1,88 @@
+map $http_upgrade $connection_upgrade {
+  default upgrade;
+  ''      close;
+}
+
+server {
+  listen 80;
+  server_name <%= @server_name %>;
+
+  access_log "/var/log/nginx/mastodon.access.log";
+  error_log "/var/log/nginx/mastodon.error.log";
+
+  location /.well-known {
+    root "/var/www/mastodon";
+  }
+  location / {
+    return 301 https://$host$request_uri;
+  }
+}
+
+server {
+  listen 443 ssl;
+  server_name <%= @server_name %>;
+
+  access_log "/var/log/nginx/mastodon.access.log";
+  error_log "/var/log/nginx/mastodon.error.log";
+
+  ssl_protocols TLSv1.2;
+  ssl_ciphers EECDH+AESGCM:EECDH+AES;
+  ssl_ecdh_curve secp384r1;
+  ssl_prefer_server_ciphers on;
+  ssl_session_cache shared:SSL:10m;
+
+  <% if File.exist?(@ssl_cert) &&
+        File.exist?(@ssl_key) -%>
+  ssl_certificate     <%= @ssl_cert %>;
+  ssl_certificate_key <%= @ssl_key %>;
+  <% end -%>
+
+  keepalive_timeout    70;
+  sendfile             on;
+  client_max_body_size 0;
+  gzip off;
+
+  root <%= @mastodon_path %>/public;
+
+  # add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
+
+  location / {
+    try_files $uri @proxy;
+  }
+
+  location @proxy {
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto https;
+
+    proxy_pass_header Server;
+
+    proxy_pass http://localhost:<%= @puma_port %>;
+    proxy_buffering off;
+    proxy_redirect off;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection $connection_upgrade;
+
+    tcp_nodelay on;
+  }
+
+  location /api/v1/streaming {
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto https;
+
+    proxy_pass http://localhost:<%= @streaming_port %>;
+    proxy_buffering off;
+    proxy_redirect off;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection $connection_upgrade;
+
+    tcp_nodelay on;
+  }
+
+  error_page 500 501 502 503 504 /500.html;
+}