kosmos/chef
8
3
Fork 0
Code Issues 39 Pull Requests Projects 2 Activity

Set up Prometheus and Alertmanager #632

Merged
raucao merged 50 commits from feature/prometheus_cookbook into master 2026-07-09 13:33:38 +00:00
Conversation 2 Commits 50 Files Changed 67
+1175 -415
greg commented 2026-07-03 16:00:58 +00:00
Owner
Copy Link
Copy Source
  • Add recipe for setting up Node Exporter on all machines (via role included in base role)
  • Deploy node_exporter on all machines
  • Add all node_exporter nodes as targets in prometheus.yml (via node/role search)
  • Write custom AltertManager config
    • send to hubot/hal by default
    • send to email (critical only)
  • Configure alerts for some basic problems like low disk space, low memory/swap, and CPUs running hot
  • Find all (interesting) existing prometheus metrics endpoints for services we run and add them as targets (split out into separate topics?)
    • Garage
    • see separate issues for rest

Note: mtail can be used to scrape any log files and export metrics from them in Prometheus format.

* [x] Add recipe for setting up [Node Exporter](https://prometheus.io/docs/guides/node-exporter/) on all machines (via role included in base role) * [x] Deploy node_exporter on all machines * [x] Add all `node_exporter` nodes as targets in `prometheus.yml` (via node/role search) * [x] Write custom AltertManager config * [x] send to hubot/hal by default * [x] send to email (critical only) * [x] Configure alerts for some basic problems like low disk space, low memory/swap, and CPUs running hot * [x] Find all (interesting) existing prometheus metrics endpoints for services we run and add them as targets (split out into separate topics?) * [x] Garage * see separate issues for rest Note: [mtail](https://github.com/google/mtail) can be used to scrape *any* log files and export metrics from them in Prometheus format.
greg added 4 commits 2026-07-03 16:00:58 +00:00
Add prometheus node 850db344b7
Set Chef environment for node ec73dd5b57
Add community prometheus cookbook 4cd6c41254
WIP Initial kosmos_prometheus wrapper cookbook 765d0b080e
raucao commented 2026-07-03 17:46:47 +00:00
Owner
Copy Link
Copy Source

Would be nice to get #263 solved for looking up the targets' private IPs.

Would be nice to get #263 solved for looking up the targets' private IPs.
raucao self-assigned this 2026-07-03 17:54:39 +00:00
greg was assigned by raucao 2026-07-03 17:54:39 +00:00
raucao added the feature label 2026-07-03 17:57:33 +00:00
raucao added 5 commits 2026-07-04 13:27:48 +00:00
Remove obsolete CLI option df8c8d1742 
Was left over from trying to overwrite the defaults before
WIP Add node exporter 7396af5ca4
Apply changed configs to prometheus and alertmanager e21797b402
Re-add global config, change values 2d835335b5
Remove upstream prometheus cookbook, migrate to our own 63534e1cf5
raucao added 1 commit 2026-07-04 13:45:37 +00:00
Use base role instead of recipe in all runlists 5813a45987
raucao added 2 commits 2026-07-04 13:47:00 +00:00
Add prometheus node exporter to base role ea69c7cec6
Add all nodes with base role as node exporter targets 153b1e77c5
raucao added 1 commit 2026-07-04 13:51:48 +00:00
Remove old garage nodes 7dc4895da3
raucao added 1 commit 2026-07-04 14:14:50 +00:00
Scrape garage metrics 67f62ebd6c
raucao changed title from WIP: Set up Prometheus to WIP: Set up Prometheus and AlertManager 2026-07-04 14:24:38 +00:00
raucao changed title from WIP: Set up Prometheus and AlertManager to WIP: Set up Prometheus and Alertmanager 2026-07-04 14:24:55 +00:00
raucao added this to the 2026 project 2026-07-04 14:25:32 +00:00
raucao moved this to In Progress in 2026 on 2026-07-04 14:25:46 +00:00
raucao added 1 commit 2026-07-04 14:30:34 +00:00
Update node configs f2ebda4a1a
raucao added 2 commits 2026-07-05 10:42:14 +00:00
Use full domain as fqdn for leo e29d90d3d0
Set node name as prometheus instance name 821b56e6ab 
And DRY up the recipe so it's easy and expressive to add more
metrics/targets
raucao added 1 commit 2026-07-05 11:19:29 +00:00
Add altermanager section to prometheus config 171db0fb7c
raucao added 2 commits 2026-07-05 15:21:25 +00:00
Remove decommissioned node 639800a838
Merge branch 'master' into feature/prometheus_cookbook 53de8ea119
raucao added 5 commits 2026-07-05 15:26:32 +00:00
Add util script for sanitizing nodes 6c3da8a711 
Deletes the "override" and "default" properties, which may contain
sensitive data
Add pre-commit hook for sanitizing node configs 8c71261fc9
Remove newline from all node files 959d506cfb 
Was actually added by manually editing the files, but Chef writes them
without newline, so our sanitization script also does it that way
Remove decommissioned nodes 53cf95c4e7
Merge pull request 'Sanitize node configs via script, and do it automatically in a Git hook' (#635) from dev/sanitize_node_configs into feature/prometheus_cookbook c762dfe8c1 
Reviewed-on: #635
raucao added 2 commits 2026-07-05 15:32:53 +00:00
Add vm_host label to prometheus targets 0e9bcca72d
Add new vm_host attribute/value to all VM nodes 6370069e9d
raucao added 1 commit 2026-07-05 15:39:36 +00:00
Merge branch 'master' into feature/prometheus_cookbook c736468259
greg added 1 commit 2026-07-05 16:03:31 +00:00
Fix year in license file 8601e33764
greg added 1 commit 2026-07-05 16:16:42 +00:00
Move web port to an attribute 5e7a0c07b0
raucao added 2 commits 2026-07-06 10:38:13 +00:00
WIP Add default alertmanager route and receiver 65c1933a99
Add icons to visualize alert severity 81aae3d1bb
raucao added 5 commits 2026-07-06 12:56:13 +00:00
Fix port attribute 33987402e3
Critical icon only for critical severity 21513e6d10
Fix postfix relay sending (on newer Ubuntu versions) 4ce77e1a6e 
Requires relay restrictions to be configured explicitly
Change prometheus scrape/eval intervals 20f67d7261
Send critical alerts via email (in addition to XMPP) 1ae5210813
raucao added 1 commit 2026-07-06 16:02:09 +00:00
WIP Add prometheus rules, improve alerts 2a14837f64
raucao added 1 commit 2026-07-06 16:48:10 +00:00
Update node configs 02dc95b7de
raucao added 1 commit 2026-07-06 16:50:15 +00:00
Document upgrading Chef client 704d0bca0e
raucao commented 2026-07-07 08:30:21 +00:00
Owner
Copy Link
Copy Source

Added information about the various export options for the listed services. We should split them out into separate issues once we're done with the rest here.

Added separate issues for more metrics. See below.

<del>Added information about the various export options for the listed services. We should split them out into separate issues once we're done with the rest here.</del> Added separate issues for more metrics. See below.
raucao added the monitoring label 2026-07-07 11:08:09 +00:00
raucao added a new dependency 2026-07-07 11:23:09 +00:00
#641 Set up Grafana
raucao added a new dependency 2026-07-07 11:24:57 +00:00
#640 Set up metrics/scraping for strfry
raucao added a new dependency 2026-07-07 11:25:13 +00:00
#639 Set up metrics/scraping for HAProxy
raucao added a new dependency 2026-07-07 11:25:20 +00:00
#638 Set up metrics/scraping for OpenResty
raucao added a new dependency 2026-07-07 11:25:27 +00:00
#637 Set up metrics/scraping for Redis
raucao added a new dependency 2026-07-07 11:25:33 +00:00
#636 Set up metrics/scraping for PostgreSQL
raucao added 1 commit 2026-07-07 12:27:06 +00:00
Show severity for new alerts, number of firing vs resolved 94a3c41964
raucao added 1 commit 2026-07-07 13:05:48 +00:00
Update dirsrv node configs e4ad52d413
raucao added 2 commits 2026-07-07 13:20:56 +00:00
Remove unnecessary include 4976888358
Update node configs b6ba5ba679
raucao added 2 commits 2026-07-07 14:58:45 +00:00
Increase padding for NodeDown rule 9fca5386f3
Update node configs 6f93820936
raucao added 1 commit 2026-07-09 12:10:58 +00:00
Configure alerts for disk/filesystem usage, memory, CPU b700baac37
greg added 1 commit 2026-07-09 13:18:42 +00:00
Move the node_exporter port to an attribute ae14a1f353
greg added 1 commit 2026-07-09 13:24:05 +00:00
Move the alertmanager port to an attribute f15d5cb958
greg marked the pull request as ready for review 2026-07-09 13:26:27 +00:00
greg added 1 commit 2026-07-09 13:31:03 +00:00
Fix node attribute 51af9c76e0
raucao added the kredits-3 label 2026-07-09 13:33:16 +00:00
raucao merged commit 86f7b68726 into master 2026-07-09 13:33:38 +00:00
raucao deleted branch feature/prometheus_cookbook 2026-07-09 13:33:38 +00:00
raucao moved this to Done in 2026 on 2026-07-09 15:11:14 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
service
discourse
Kosmos Community Forums
monitoring
Infrastructure metrics, alerts, notifications, etc.
service
accounts
Kosmos Accounts
service
drone-ci
Kosmos Drone CI
service
email
mail.kosmos.org
service
garage
S3-compatible object storage
service
gitea
Kosmos Gitea
service
ipfs
Kosmos IPFS
service
mastodon
kosmos.social
service
nostr
Relays, Blossom server, etc.
service
postgres
Database cluster
service
remotestorage
Portable data storage for the Web
service
wiki
Kosmos Wiki
service
xmpp
Kosmos Chat
bug
Something is not working
 design
Graphic/visual design
 dev environment
Config, builds, CI, deployment, etc.
 docs
Documentation
 duplicate
This issue or pull request already exists
 enhancement
Improving existing functionality
 feature
New functionality
 good first issue
Dive in, and start contributing
 idea
Something to consider
 invalid
Not a bug
 kredits-1
Small contribution
 kredits-2
Medium contribution
 kredits-3
Large contribution
 on hold
Currently not actionable
 ops
Manual IT ops activities
 question
Looking for an answer
release
major
release
minor
release
patch
security
All your base are belong to us
 ui/ux
User interface, process design, etc.
 wontfix
This won't be fixed
No Label monitoring feature kredits-3
Milestone
No items
No Milestone
Projects
Clear projects
No project 2026
Assignees
Clear assignees
bkero (bkero) bumi (bumi) fsmanuel (fsmanuel) galfert (galfert) greg (Greg) hueso (hueso) raucao (Râu Cao) slvrbckt (Nick Jennings)
No Assignees raucao greg
2 Participants
Notifications
Due Date
No due date set.
Blocks
#636 Set up metrics/scraping for PostgreSQL
kosmos/chef
#637 Set up metrics/scraping for Redis
kosmos/chef
#638 Set up metrics/scraping for OpenResty
kosmos/chef
#639 Set up metrics/scraping for HAProxy
kosmos/chef
#640 Set up metrics/scraping for strfry
kosmos/chef
#641 Set up Grafana
kosmos/chef
Reference: kosmos/chef#632
Delete Branch "feature/prometheus_cookbook"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?