Mitigate Gitea crawler load: ban Perplexity, throttle expensive routes

Gitea was CPU-saturated (load ~6 on 4 cores, 153 concurrent git
cat-file children) by mostly AI/SEO crawlers. PerplexityBot alone made
~43k requests/day to /kosmos/chef/pulls, enumerating tens of thousands
of distinct filter-permutation URLs; others deep-crawled blame/compare
and commit diff/patch pages. The existing edge rate-limit only matched
three user agents and caught none of the traffic.

- Add custom/public/robots.txt: disallow PerplexityBot entirely and,
  for everyone else, disallow filter permutations (*?*) and the
  expensive render routes (blame/compare/raw/archive/patch/diff).
- Hard-ban PerplexityBot at the edge (403); it ignores robots.txt.
- Add a per-IP "expensive" limit_req zone (10r/m, burst 5) covering the
  expensive routes and pulls/issues/milestones filter queries. Normal
  requests use an empty key and are unaffected.
- Wire the ban and the limit into the Gitea vhost, and reload OpenResty
  when rate_limits.conf changes.
- Bump the openresty submodule so it builds under Ruby 3.4 (Cinc 19):
  File.exists? was removed, use File.exist?.

site-cookbooks/kosmos_gitea/files/custom/public/robots.txt

@@ -0,0 +1,16 @@
+# Hard-disallow the AI crawler that floods Gitea with combinatorial
+# filter-permutation requests.
+User-agent: PerplexityBot
+Disallow: /
+
+# Everyone else: keep normal pages crawlable, but never crawl filter
+# permutations on list pages or the expensive render routes.
+User-agent: *
+Disallow: /*?*
+Disallow: /*/blame/
+Disallow: /*/compare/
+Disallow: /*/raw/
+Disallow: /*/archive/
+Disallow: /*.patch$
+Disallow: /*.diff$
+Allow: /

site-cookbooks/kosmos_gitea/files/rate_limits.conf

@@ -23,3 +23,30 @@ limit_req_zone $bot_limit_key zone=bots:20m rate=240r/m; # 4r/s per IP+bot
 
 # Status code returned when the limit is exceeded
 limit_req_status 429;
+
+# Hard ban: Perplexity ignores robots.txt, so deny it at the edge.
+map $http_user_agent $deny_bot {
+    default              0;
+    ~*PerplexityBot      1;
+}
+
+# Expensive routes and filter-permutation list pages get a hard per-IP limit.
+# These are the pages crawlers enumerate combinatorially and that are CPU-heavy
+# to render (blame/compare/diff/syntax-highlight/signature verification).
+map $request_uri $expensive_request {
+    default                                       0;
+    ~*^/[^/]+/[^/]+/blame/                        1;
+    ~*^/[^/]+/[^/]+/compare/                      1;
+    ~*^/[^/]+/[^/]+/raw/                          1;
+    ~*^/[^/]+/[^/]+/archive/                      1;
+    ~*\.(patch|diff)(\?|$)                        1;
+    ~*^/[^/]+/[^/]+/(pulls|issues|milestones)\?   1;
+}
+
+# Only non-empty keys are counted, so normal requests are never throttled here.
+map $expensive_request $expensive_key {
+    default              "";
+    "1"                  $binary_remote_addr;
+}
+
+limit_req_zone $expensive_key zone=expensive:20m rate=10r/m; # ~1 per 6s per IP

site-cookbooks/kosmos_gitea/recipes/default.rb

@@ -50,7 +50,7 @@ directory working_directory do
   mode  "0750"
 end
 
-%w(custom custom/options custom/options/label custom/templates custom/templates/custom).each do |path|
+%w(custom custom/options custom/options/label custom/public custom/templates custom/templates/custom).each do |path|
   directory "#{working_directory}/#{path}" do
     owner "git"
     group "git"
@@ -76,6 +76,15 @@ cookbook_file "#{working_directory}/custom/templates/custom/header.tmpl" do
   notifies :restart, "service[gitea]", :delayed
 end
 
+# robots.txt: disallow Perplexity and crawler-unfriendly filter permutations
+# and expensive render routes
+cookbook_file "#{working_directory}/custom/public/robots.txt" do
+  source "custom/public/robots.txt"
+  owner "git"
+  group "git"
+  mode  "0644"
+end
+
 directory config_directory do
   owner "git"
   group "git"

site-cookbooks/kosmos_gitea/recipes/nginx.rb

@@ -17,12 +17,13 @@ tls_cert_for domain do
   action :create
 end
 
-# Slow down requests from bots
+# Slow down / block unwanted bot traffic and hard-limit expensive routes
 cookbook_file "#{node["openresty"]["dir"]}/conf.d/rate_limits.conf" do
   source "rate_limits.conf"
   owner "root"
   group "root"
   mode "0644"
+  notifies :reload, "service[openresty]", :delayed
 end
 
 openresty_site domain do

site-cookbooks/kosmos_gitea/templates/default/nginx_conf_web.erb

@@ -20,9 +20,17 @@ server {
 
   proxy_intercept_errors on;
 
+  # Hard ban Perplexity (does not honour robots.txt). Defined in
+  # /etc/openresty/conf.d/rate_limits.conf
+  if ($deny_bot) { return 403; }
+
   # Rate limit for bots. Defined in /etc/openresty/conf.d/rate_limits.conf
   limit_req zone=bots burst=10; # allow a larger spike for asset bursts
 
+  # Hard limit for expensive render routes and filter-permutation list pages.
+  # Only non-empty keys are counted, so normal requests are unaffected.
+  limit_req zone=expensive burst=5 nodelay;
+
   location ~ ^/(avatars|repo-avatars)/.*$ {
     proxy_buffers 1024 8k;
     proxy_pass http://_gitea_web;