Gitea was CPU-saturated (load ~6 on 4 cores, 153 concurrent git
cat-file children) by mostly AI/SEO crawlers. PerplexityBot alone made
~43k requests/day to /kosmos/chef/pulls, enumerating tens of thousands
of distinct filter-permutation URLs; others deep-crawled blame/compare
and commit diff/patch pages. The existing edge rate-limit only matched
three user agents and caught none of the traffic.
- Add custom/public/robots.txt: disallow PerplexityBot entirely and,
for everyone else, disallow filter permutations (*?*) and the
expensive render routes (blame/compare/raw/archive/patch/diff).
- Hard-ban PerplexityBot at the edge (403); it ignores robots.txt.
- Add a per-IP "expensive" limit_req zone (10r/m, burst 5) covering the
expensive routes and pulls/issues/milestones filter queries. Normal
requests use an empty key and are unaffected.
- Wire the ban and the limit into the Gitea vhost, and reload OpenResty
when rate_limits.conf changes.
- Bump the openresty submodule so it builds under Ruby 3.4 (Cinc 19):
File.exists? was removed, use File.exist?.
raucao