Mitigate Gitea crawler load: ban Perplexity, throttle expensive routes #699

2026-11-09T15:52:04Z

raucao commented

2026-11-09 15:52:04 +00:00

Why

Gitea was CPU-saturated (load ~6 on 4 cores, 153 concurrent git cat-file children, 224k error log lines/hour). The traffic was mostly AI/SEO crawlers:

PerplexityBot made ~43k requests/day to /kosmos/chef/pulls, enumerating ~43k distinct filter-permutation URLs (type, sort, state, labels, assignee).
Epochal-Search, Applebot and others deep-crawled blame, compare, commit .patch/.diff and issue/PR list pages.

No robots.txt existed, and the edge rate limit only matched ClaudeBot/Sogou/meta-externalagent, so it caught none of it.

What this does

Adds custom/public/robots.txt, disallowing PerplexityBot entirely and, for everyone else, the filter permutations (/*?*) and expensive render routes (blame, compare, raw, archive, *.patch, *.diff).
Hard-bans PerplexityBot at the edge (403), since it does not honour robots.txt.
Adds a per-IP expensive limit_req zone (10 r/m, burst 5) covering those routes and pulls/issues/milestones filter queries. Normal requests use an empty key and are unaffected.
Wires the ban and limit into the Gitea vhost and reloads OpenResty when rate_limits.conf changes.
Bumps the openresty submodule so it builds under Ruby 3.4 (Cinc 19): File.exists? was removed, replaced with File.exist?. Without this, every OpenResty converge aborts in openresty::luarocks.

Verified on production

PerplexityBot UA → 403; Googlebot and facebookexternalhit → 200.
25 concurrent filter-permutation requests → 6×200 + 19×429; 25 concurrent normal requests → 25×200.
Gitea load average 6.15 → 2.00, git cat-file children 153 → 5.

Notes

The openresty submodule change is pushed to 67P/chef-openresty master (223c5e9); this PR bumps the pointer. draco and fornax were migrated from Chef 18 to Cinc Client to apply the proxy changes.

## Why Gitea was CPU-saturated (load ~6 on 4 cores, 153 concurrent `git cat-file` children, 224k error log lines/hour). The traffic was mostly AI/SEO crawlers: - **PerplexityBot** made ~43k requests/day to `/kosmos/chef/pulls`, enumerating ~43k distinct filter-permutation URLs (`type`, `sort`, `state`, `labels`, `assignee`). - Epochal-Search, Applebot and others deep-crawled `blame`, `compare`, commit `.patch`/`.diff` and issue/PR list pages. No `robots.txt` existed, and the edge rate limit only matched ClaudeBot/Sogou/meta-externalagent, so it caught none of it. ## What this does - Adds `custom/public/robots.txt`, disallowing PerplexityBot entirely and, for everyone else, the filter permutations (`/*?*`) and expensive render routes (`blame`, `compare`, `raw`, `archive`, `*.patch`, `*.diff`). - Hard-bans PerplexityBot at the edge (403), since it does not honour `robots.txt`. - Adds a per-IP `expensive` `limit_req` zone (10 r/m, burst 5) covering those routes and `pulls`/`issues`/`milestones` filter queries. Normal requests use an empty key and are unaffected. - Wires the ban and limit into the Gitea vhost and reloads OpenResty when `rate_limits.conf` changes. - Bumps the `openresty` submodule so it builds under Ruby 3.4 (Cinc 19): `File.exists?` was removed, replaced with `File.exist?`. Without this, every OpenResty converge aborts in `openresty::luarocks`. ## Verified on production - PerplexityBot UA → 403; Googlebot and facebookexternalhit → 200. - 25 concurrent filter-permutation requests → 6×200 + 19×429; 25 concurrent normal requests → 25×200. - Gitea load average 6.15 → 2.00, `git cat-file` children 153 → 5. ## Notes The `openresty` submodule change is pushed to `67P/chef-openresty` master (`223c5e9`); this PR bumps the pointer. `draco` and `fornax` were migrated from Chef 18 to Cinc Client to apply the proxy changes.

raucao added 1 commit 2026-11-09 15:52:04 +00:00

Mitigate Gitea crawler load: ban Perplexity, throttle expensive routes 8cdb6e2d5e

Gitea was CPU-saturated (load ~6 on 4 cores, 153 concurrent git
cat-file children) by mostly AI/SEO crawlers. PerplexityBot alone made
~43k requests/day to /kosmos/chef/pulls, enumerating tens of thousands
of distinct filter-permutation URLs; others deep-crawled blame/compare
and commit diff/patch pages. The existing edge rate-limit only matched
three user agents and caught none of the traffic.

- Add custom/public/robots.txt: disallow PerplexityBot entirely and,
  for everyone else, disallow filter permutations (*?*) and the
  expensive render routes (blame/compare/raw/archive/patch/diff).
- Hard-ban PerplexityBot at the edge (403); it ignores robots.txt.
- Add a per-IP "expensive" limit_req zone (10r/m, burst 5) covering the
  expensive routes and pulls/issues/milestones filter queries. Normal
  requests use an empty key and are unaffected.
- Wire the ban and the limit into the Gitea vhost, and reload OpenResty
  when rate_limits.conf changes.
- Bump the openresty submodule so it builds under Ruby 3.4 (Cinc 19):
  File.exists? was removed, use File.exist?.

raucao added the

service

gitea

kredits-1 labels 2026-11-09 15:53:56 +00:00

raucao requested review from greg 2026-11-09 15:53:59 +00:00

greg approved these changes

2026-11-09 17:53:28 +00:00

greg left a comment

👍

greg merged commit 6c6408b7c9 into master

2026-11-09 17:53:33 +00:00

greg deleted branch gitea-crawler-mitigation