Downsizing the GKE setup #29
Loading…
x
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Last week I have downsized the GKE setup from 4 to 2 nodes. I cannot find an option to change the machine type (both nodes are 1vCPU right now), so it looks like we would have to recreate a node pool. GKE offers the option of a half shared CPU, we need to check if the performance is acceptable for Gitea
Yes, of course it has to be a new node pool. A cluster can have many pools.
And I think there's no question about any type of CPU being enough for Gitea. It uses hardly any resources at all, because it's all compiled machine code. No need for spending any time on testing that beforehand.
Here are the current pods running on our two nodes:
The big system pods are fluentd-gcp (2 pods, requests 200MB RAM, 10% of a CPU), kube-dns (2 pods, requests 110MB RAM, 27% of a CPU), metrics-server (1 pod, requests 105MB RAM, 5% of a CPU)
If we switch to 3 small cluster nodes (g1-small), we'd have 3 x 1.7GB RAM (5.1GB) and 3 x shared vCPUs as opposed to 2 x 2 GB (4GB) and 2 x 1 vCPUs right now.
We should be able to switch to 3 small nodes without changing the resource requests and limits, and then probably adjust the CPU requests. These 3 small nodes should cost around USD 43.38 / month according to the price calculator (https://cloud.google.com/products/calculator/)
So what you're saying is that you want to run 5.1GB of memory for Gitea only needing 400MB max?
I don't understand this reasoning. It sounds like you're saying the smallest node type doesn't work at all by default. So Google is offering a node type that cannot work with GKE. Which is both unreasonable and also doesn't map the blog posts around the net about using $5 nodes for GKE.
Btw, running two nodes, if that's what is happening right now, is actually the first thing that the official docs (as well as others) say you should never do, because the config orchestration needs one master and two slaves minimum. Hence, a minimum of 3 nodes is always recommended, even for the smallest of clusters.
I found a resource that lists the actual allocatable memory for different node types: https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu
The system pods do not count towards that limit. But they still use actual RAM, around 400MB per node according to the docs, ~500MB right now for us
3 f1-micro would be USD 12.88 / month. A micro has 0.6GB RAM, 240 of it allocatable (1.8GB RAM total, 720MB allocatable). So that should work for Gitea and its database (plus the system pods)
On Monday when I tried to migrate to 2 micros we ran out of CPU, but that should not happen with 3 micros.
We could do the switch to 3 micros tomorrow, what do you think?
I think my opinion doesn't matter anymore. You just explained why it makes sense to switch to micro instances.
Note from chat: the Gitea resource limits have to be adjusted before downsizing the cluster again.
It turns out we can get rid of one of the 3 small nodes after all. The master is managed by GKE (https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#master). The minimum of 3 nodes in a cluster is when you run your own cluster, because etcd needs at least 3 nodes for a production cluster; this is what the Kubernetes docs and blog posts are talking about, not the case of a managed service like Amazon EKS or GKE. This is confusing because they are still "nodes", but they don't have a role and only run pods, they do not provide the actual Kubernetes API (the master managed by GKE does)
Running only one node would mean that Gitea would go down completely if the node goes down, but two would be just fine for our usage for now
Edit: The GKE docs call these nodes "worker machines" as opposed to the "cluster master"