raucao
8a60ef2aea
Job containers were pinned to a single shared `gitea-actions` bridge network, added with the Actions cache to give service containers DNS and a fixed gateway for the built-in cache server. Because every job of every run attached to that one network, concurrent jobs' service containers collided on hostnames: two simultaneous jobs that both expose a `redis` service meant either could resolve the other's container, causing intermittent cross-job CI failures that vanished when runs did not overlap. Set `container.network` back to empty so the runner auto-creates an isolated user-defined network per job. Embedded DNS still works on each per-job network, so service containers remain reachable by name, but jobs can no longer resolve each other's containers. `cache.host` is left empty for auto-detection; the runner here is a host binary (systemd), not containerized, so the detected address is reachable from job containers. The cache port stays fixed per runner. The cache firewall rule now allows Docker's default address pool (172.16.0.0/12) because per-job subnets are dynamic, and the now-unused `gitea-actions` network is removed on converge once idle.
This repository contains all infrastructure automation code that we use to set up and configure servers, virtual machines, and applications for Kosmos hosted services.
Cookbooks are written in Ruby, and based on Chef Infra resources. Some cookbooks contain integration test suites based on Test Kitchen.
Nodes run the community Cinc Client distribution of Chef
Infra Client instead of the license-gated Chef Infra Client. Everything — the
knife zero workflow and the Test Kitchen suites — is driven by
Cinc Workstation, and on-node
configuration lives in /etc/cinc.
Note: Manual configuration of servers and applications is highly discouraged, and can be overwritten or lost without notice!
Setup
Install Cinc Workstation
Cinc Workstation replaces Chef Workstation. Install it following
https://cinc.sh/docs/workstation/ — or, on Arch Linux, the cinc-workstation-bin
AUR package. It installs to /opt/cinc-workstation and reads its user config
from ~/.cinc-workstation.
rbenv
If you use rbenv to manage Ruby versions on your system, install the
rbenv-cinc-workstation plugin (the Cinc port of
rbenv-chef-workstation),
create an empty $(rbenv root)/versions/cinc-workstation directory, and run
rbenv rehash. This repository pins the version in .ruby-version.
Install knife-zero
knife-zero is not bundled with Cinc Workstation. Install it into Cinc's user
gem directory:
cinc gem install knife-zero
Configure knife
Copy .cinc/config.rb.example to .cinc/config.rb and adjust
encrypted_data_bag_secret if your secret lives elsewhere.
Common tasks
Bootstrap a new host server
knife zero bootstrap root@server-name.kosmos.org --run-list "role[base],role[kvm_host]" --secret-file .cinc/encrypted_data_bag_secret
Bootstrap a new VM
knife zero bootstrap ubuntu@zerotier-ip-address -x ubuntu --sudo --run-list "role[base],role[kvm_guest]" --secret-file .cinc/encrypted_data_bag_secret
Bootstrap a new VM with environment and role/app (postgres replica as example)
knife zero bootstrap ubuntu@10.1.1.134 -x ubuntu --sudo --environment production --run-list "role[base],role[kvm_guest],role[postgresql_replica]" --secret-file .cinc/encrypted_data_bag_secret
Run Chef Zero on a host server
knife zero converge -p2222 name:server-name.kosmos.org
Run Chef Zero on a VM
knife zero converge -a knife_zero.host name:vm-name-23
Update Cinc Client on a server
knife zero converge name:dev.kosmos.org --client-version latest
Upgrade Cinc Client on a node
knife ssh -a knife_zero.host 'name:foo-1' 'curl -L https://omnitruck.cinc.sh/install.sh | sudo bash -s -- -v latest'
The migration script (below) installs the current stable Cinc Client for the
node's platform (-v latest); omnitruck picks the right build per OS. Pass a
concrete version (e.g. -v 19.2.12) instead of latest if you need a true
freeze.
Migrate an existing node to Cinc Client
scripts/knife/migrate_node_to_cinc.sh node-name-1 # then converge as usual
Managing cookbooks
Cookbooks are managed via Berkshelf. Run berks --help for command help.
Install cookbooks listed in Berksfile:
berks install
Vendor installed cookbooks to the cookbooks/ dir:
berks vendor cookbooks/ --delete
Integration tests
Some cookbooks contain Test Kitchen suites. They are configured to run against
Cinc Client (provisioner: product_name: cinc) and use Cinc Workstation's
bundled kitchen (which ships a Cinc-capable kitchen-dokken):
kitchen test
(use kitchen converge/verify while iterating on a single suite).