Move Gitea bot controls into encrypted data bag #703

2026-11-13T07:08:49Z

raucao commented

2026-11-13 07:08:49 +00:00

Broadens the Gitea edge bot blocking and moves the bot/IP lists out of the cookbook into the encrypted credentials/gitea data bag (bot_filter), so the rate-limited crawler names, hard-banned user agents and denied networks are no longer published in this public repository and can be changed by editing the data bag.

rate_limits.conf and robots.txt are now rendered from the data bag. The rate values and the expensive-route/robots route patterns stay static in the templates. This also adds several AI crawlers to the rate-limited set, tightens the bots zone to 60r/m so the limit actually engages, hard-blocks a datacenter scraper network and disallows combinatorial commit-browsing routes in robots.txt.

The bot_filter schema is limit (list of {match, key}), deny_user_agents (list of user-agent patterns) and deny_networks (list of CIDRs). The cookbook is bumped to 0.3.9 and the nodes submodule ref is updated for gitea-2's zerotier state.

Broadens the Gitea edge bot blocking and moves the bot/IP lists out of the cookbook into the encrypted `credentials/gitea` data bag (`bot_filter`), so the rate-limited crawler names, hard-banned user agents and denied networks are no longer published in this public repository and can be changed by editing the data bag. `rate_limits.conf` and `robots.txt` are now rendered from the data bag. The rate values and the expensive-route/robots route patterns stay static in the templates. This also adds several AI crawlers to the rate-limited set, tightens the bots zone to 60r/m so the limit actually engages, hard-blocks a datacenter scraper network and disallows combinatorial commit-browsing routes in robots.txt. The `bot_filter` schema is `limit` (list of `{match, key}`), `deny_user_agents` (list of user-agent patterns) and `deny_networks` (list of CIDRs). The cookbook is bumped to 0.3.9 and the nodes submodule ref is updated for gitea-2's zerotier state.

raucao added 1 commit 2026-11-13 07:08:49 +00:00

Move Gitea bot controls into encrypted data bag 207f0309f1

Broaden the edge bot blocking and move the bot/IP lists out of the
cookbook so they are no longer published in the public repository.

- rate_limits.conf and robots.txt are now rendered from the encrypted
  credentials/gitea data bag (bot_filter): rate-limited crawler names,
  hard-banned user agents and denied networks.
- Add several AI crawlers to the rate-limited set, tighten the bots zone
  to 60r/m so the limit actually engages, hard-block a datacenter scraper
  network and disallow combinatorial commit-browsing routes in robots.txt.
- Rate values and the expensive-route/robots route patterns stay static.
- Bump cookbook to 0.3.9; include updated nodes submodule ref.

raucao added the kredits-2 ops labels 2026-11-13 07:13:47 +00:00

raucao requested review from greg 2026-11-13 07:13:53 +00:00

greg approved these changes

2026-11-13 08:38:59 +00:00

greg left a comment

👍

greg merged commit ea04af784e into master

2026-11-13 08:39:08 +00:00

greg deleted branch gitea-bot-filter-data-bag

2026-11-13 08:39:08 +00:00

raucao added the

service

gitea

label 2026-11-14 06:27:02 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/chef#703