WIP: Set up Prometheus and Alertmanager

greg commented

2026-07-03 16:00:58 +00:00

Owner

Add recipe for setting up Node Exporter on all machines (via role included in base role)
Add all node_exporter nodes as targets in prometheus.yml (via node/role search)
Write custom AltertManager config
- Add Prometheus webhooks to hubot-incoming-webhook
- Configure AltertManager to send to email and hubot/hal by default
Configure alerts for some basic problems like low disk space, low memory/swap, and CPUs running hot
Find all (interesting) existing prometheus metrics endpoints for services we run and add them as targets
- Garage
- PostgreSQL
- Redis
- Nginx (Openresty)
- haproxy
- ...

* [x] Add recipe for setting up [Node Exporter](https://prometheus.io/docs/guides/node-exporter/) on all machines (via role included in base role) * [x] Add all `node_exporter` nodes as targets in `prometheus.yml` (via node/role search) * [ ] Write custom AltertManager config * [ ] Add Prometheus webhooks to `hubot-incoming-webhook` * [ ] Configure AltertManager to send to email and hubot/hal by default * [ ] Configure alerts for some basic problems like low disk space, low memory/swap, and CPUs running hot * [ ] Find all (interesting) existing prometheus metrics endpoints for services we run and add them as targets * [x] Garage * [ ] PostgreSQL * [ ] Redis * [ ] Nginx (Openresty) * [ ] haproxy * [ ] ...

greg added 4 commits 2026-07-03 16:00:58 +00:00

Add prometheus node 850db344b7

Set Chef environment for node ec73dd5b57

Add community prometheus cookbook 4cd6c41254

WIP Initial kosmos_prometheus wrapper cookbook 765d0b080e

raucao commented

2026-07-03 17:46:47 +00:00

Owner

Would be nice to get #263 solved for looking up the targets' private IPs.

raucao self-assigned this 2026-07-03 17:54:39 +00:00

greg was assigned by raucao

2026-07-03 17:54:39 +00:00

raucao referenced this pull request

2026-07-03 17:55:30 +00:00

Choose monitoring/alerting/logging stack/software #432

raucao added the feature label 2026-07-03 17:57:33 +00:00

raucao added 5 commits 2026-07-04 13:27:48 +00:00

Remove obsolete CLI option df8c8d1742

Was left over from trying to overwrite the defaults before

WIP Add node exporter 7396af5ca4

Apply changed configs to prometheus and alertmanager e21797b402

Re-add global config, change values 2d835335b5

Remove upstream prometheus cookbook, migrate to our own 63534e1cf5

raucao added 1 commit 2026-07-04 13:45:37 +00:00

Use base role instead of recipe in all runlists 5813a45987

raucao added 2 commits 2026-07-04 13:47:00 +00:00

Add prometheus node exporter to base role ea69c7cec6

Add all nodes with base role as node exporter targets 153b1e77c5

raucao added 1 commit 2026-07-04 13:51:48 +00:00

Remove old garage nodes 7dc4895da3

raucao added 1 commit 2026-07-04 14:14:50 +00:00

Scrape garage metrics 67f62ebd6c

raucao changed title from ~~WIP: Set up Prometheus~~ to WIP: Set up Prometheus and AlertManager

2026-07-04 14:24:38 +00:00

raucao changed title from ~~WIP: Set up Prometheus and AlertManager~~ to WIP: Set up Prometheus and Alertmanager

2026-07-04 14:24:55 +00:00

raucao added this to the 2026 project 2026-07-04 14:25:32 +00:00

raucao moved this to In Progress in 2026 on 2026-07-04 14:25:46 +00:00

raucao added 1 commit 2026-07-04 14:30:34 +00:00

Update node configs f2ebda4a1a

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feature/prometheus_cookbook:feature/prometheus_cookbook

git checkout feature/prometheus_cookbook

WIP: Set up Prometheus and Alertmanager #632

Checkout