Add default and Kosmos label sets #15

raucao · 2019-01-27T08:23:06Z

raucao commented

2019-01-27 08:23:06 +00:00

Adds custom label set configs, overriding the default set and adding a new one for Kosmos (that includes kredits labels).

To do:

Add config to deploy /custom/ directory contents to our Gitea on GKE

closes #4

Adds custom label set configs, overriding the default set and adding a new one for Kosmos (that includes kredits labels). To do: * [x] Add config to deploy `/custom/` directory contents to our Gitea on GKE closes #4

raucao commented

2019-01-27 08:23:38 +00:00

@gregkare IIRC you had plans for how to add the /custom/ files to the deployment?

@gregkare IIRC you had plans for how to add the `/custom/` files to the deployment?

raucao closed this pull request

2019-01-27 08:24:54 +00:00

raucao reopened this pull request

2019-01-27 08:24:58 +00:00

greg commented

2019-01-28 16:22:57 +00:00

My plan was to upload them to the persistent volume, or create a small Docker container that contains these files, whichever makes more sense. kubectl cp can be used to copy files/directories to a container

My plan was to upload them to the persistent volume, or create a small Docker container that contains these files, whichever makes more sense. `kubectl cp` can be used to copy files/directories to a container

raucao commented

2019-01-28 22:43:53 +00:00

OK. Can you take it from here?

greg self-assigned this 2019-01-29 09:11:59 +00:00

greg commented

2019-01-29 09:12:16 +00:00

OK. Can you take it from here?

Yes, assigned myself

> OK. Can you take it from here? Yes, assigned myself

greg commented

2019-02-05 19:32:04 +00:00

Done in eba722992f. I created the labels as ConfigMaps and copied them to the persistent data volume as part of the init container.

This is running on our gitea (can be checked here: https://gitea.kosmos.org/kosmos/chef/labels)

Done in eba722992f. I created the labels as ConfigMaps and copied them to the persistent data volume as part of the init container. This is running on our gitea (can be checked here: https://gitea.kosmos.org/kosmos/chef/labels)

raucao commented

2019-02-06 04:37:24 +00:00

Thanks!

However, the task was actually to copy the entire content of /custom to the deployed Gitea, because that's where themes, template changes, etc. go as well. Anything added to the /custom folder of this repo should end up in the /custom folder of our production instance after deployment.

Thanks! However, the task was actually to copy the *entire* content of `/custom` to the deployed Gitea, because that's where themes, template changes, etc. go as well. Anything added to the `/custom` folder of this repo should end up in the `/custom` folder of our production instance after deployment.

raucao commented

2019-02-06 04:41:19 +00:00

FYI: also added the label set to this repo.

raucao added the

kredits-2

label 2019-02-06 04:43:26 +00:00

raucao self-assigned this 2019-02-06 04:43:31 +00:00

raucao added the

enhancement

label 2019-02-06 04:43:41 +00:00

greg commented

2019-02-06 08:36:37 +00:00

I think it's not a big change from this PR to achieve that (create ConfigMaps for the entire folder instead of just options/label, copy the mounted files to /data/gitea/, this is what the Gitea docs call /custom). The secrets (app.ini and the dummy TLS cert) will have to be moved respectively to custom/conf and custom/https

I think it's not a big change from this PR to achieve that (create ConfigMaps for the entire folder instead of just `options/label`, copy the mounted files to `/data/gitea/`, this is what the Gitea docs call `/custom`). The secrets (app.ini and the dummy TLS cert) will have to be moved respectively to `custom/conf` and `custom/https`

raucao commented

2019-02-06 09:43:16 +00:00

this is what the Gitea docs call /custom

I don't know what that means. The files live in an actual /custom folder with my local Gitea that I'm developing them in. The docs call that folder /custom. If you put them in the normal data folder, then almost certainly they will overwrite actual Gitea files, instead of adding custom ones in the custom folder.

> this is what the Gitea docs call /custom I don't know what that means. The files live in an actual `/custom` folder with my local Gitea that I'm developing them in. The docs call that folder `/custom`. If you put them in the normal data folder, then almost certainly they will overwrite actual Gitea files, instead of adding custom ones in the custom folder.

raucao commented

2019-02-06 09:44:27 +00:00

create ConfigMaps for the entire folder instead of just options/label

These are all static files. I'm not sure ConfigMaps are the correct way of copying a static folder to a volume or container. What about the other approach you mentioned? I think this is usually done in an init container.

> create ConfigMaps for the entire folder instead of just options/label These are all static files. I'm not sure ConfigMaps are the correct way of copying a static folder to a volume or container. What about the other approach you mentioned? I think this is usually done in an init container.

raucao commented

2019-02-24 05:13:16 +00:00

@gregkare Did you see my comments?

greg commented

2019-02-25 08:54:24 +00:00

I don’t know what that means. The files live in an actual /custom folder with my local Gitea that I’m developing them in. The docs call that folder /custom. If you put them in the normal data folder, then almost certainly they will overwrite actual Gitea files, instead of adding custom ones in the custom folder.

/data/gitea/ isn't the "normal data folder", it's our data folder, this is our /custom/

These are all static files. I’m not sure ConfigMaps are the correct way of copying a static folder to a volume or container. What about the other approach you mentioned? I think this is usually done in an init container.

ConfigMaps can store static files (https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#define-the-key-to-use-when-creating-a-configmap-from-a-file). My current approach is to store the files as ConfigMaps and then use the init container to copy them to the persistent storage, that seems like a good solution. What don't you like about it?

> I don’t know what that means. The files live in an actual /custom folder with my local Gitea that I’m developing them in. The docs call that folder /custom. If you put them in the normal data folder, then almost certainly they will overwrite actual Gitea files, instead of adding custom ones in the custom folder. `/data/gitea/` isn't the "normal data folder", it's our data folder, this is our `/custom/` > These are all static files. I’m not sure ConfigMaps are the correct way of copying a static folder to a volume or container. What about the other approach you mentioned? I think this is usually done in an init container. ConfigMaps can store static files (https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#define-the-key-to-use-when-creating-a-configmap-from-a-file). My current approach is to store the files as ConfigMaps and then use the init container to copy them to the persistent storage, that seems like a good solution. What don't you like about it?

greg commented

2019-02-25 08:56:33 +00:00

Here's the Gitea docs where they say /data/gitea is the folder for customization when using Docker: https://docs.gitea.io/en-us/install-with-docker/#customization

Here's the Gitea docs where they say `/data/gitea` is the folder for customization when using Docker: https://docs.gitea.io/en-us/install-with-docker/#customization

raucao commented

2019-02-26 04:09:26 +00:00

What don’t you like about it?

Exactly what I wrote before. In the documentation you linked the static file clearly contains actual configuration variables. Hence the name "config map". It's not "file map" or "file link" or something. The feature is meant for config keys and values. So it just seems wrong and hacky to use it for linking and copying entire directories of random files.

/data/gitea/ isn’t the “normal data folder”, it’s our data folder, this is our /custom/

Then why is there also a custom folder, like you said before:

The secrets (app.ini and the dummy TLS cert) will have to be moved respectively to custom/conf and custom/https

> What don’t you like about it? Exactly what I wrote before. In the documentation you linked the static file clearly contains actual configuration variables. Hence the name "config map". It's not "file map" or "file link" or something. The feature is meant for config keys and values. So it just seems wrong and hacky to use it for linking and copying entire directories of random files. > /data/gitea/ isn’t the “normal data folder”, it’s our data folder, this is our /custom/ Then why is there also a `custom` folder, like you said before: > The secrets (app.ini and the dummy TLS cert) will have to be moved respectively to custom/conf and custom/https

greg commented

2019-02-26 14:00:22 +00:00

From the ConfigMaps docs:

"ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable."

Configuration artifacts is a perfect description for what we're doing here

Then why is there also a custom folder, like you said before:

There's no custom folder on the persistent data volume. The Gitea docs are just confusing. I called it custom locally because it contains our customization files, but this data is going into /data/gitea/, which is the default data directory for Gitea when using Docker

From the ConfigMaps docs: "ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable." Configuration artifacts is a perfect description for what we're doing here > Then why is there also a custom folder, like you said before: There's no `custom` folder on the persistent data volume. The Gitea docs are just confusing. I called it `custom` locally because it contains our customization files, but this data is going into `/data/gitea/`, which is the default data directory for Gitea when using Docker

raucao commented

2019-02-27 06:45:48 +00:00

OK, good. So then what's missing here is to actually copy the entire custom folder there, right? The TODO has been marked complete, but I cannot see a commit to copy the folder, only one for copying the labels specifically.

greg commented

2019-02-27 11:46:21 +00:00

I'm on it

greg commented

2019-02-27 12:17:58 +00:00

There is no way to create a configmap from a directory containing subdirectories and files (you need to pass every file), so I need to rethink the process if we want to add more custom files easily. I had also misunderstood ConfigMaps, the GKE docs are more explicit about it being for key-value pairs, so it would not work for every file (https://cloud.google.com/kubernetes-engine/docs/concepts/configmap). Copying a folder recursively to a location on a pod seems possible with kubectl cp, I'm trying that out. Creating a Docker image just for copying these custom files seems overkill

There is no way to create a configmap from a directory containing subdirectories and files (you need to pass every file), so I need to rethink the process if we want to add more custom files easily. I had also misunderstood ConfigMaps, the GKE docs are more explicit about it being for key-value pairs, so it would not work for every file (https://cloud.google.com/kubernetes-engine/docs/concepts/configmap). Copying a folder recursively to a location on a pod seems possible with `kubectl cp`, I'm trying that out. Creating a Docker image just for copying these custom files seems overkill

raucao commented

2019-02-27 13:28:37 +00:00

What's wrong with simply deploying them to a PV and mounting that as /custom? I thought that was the intial approach with the init container (which would also mount the PV and copy the files oves).

What's wrong with simply deploying them to a PV and mounting that as `/custom`? I thought that was the intial approach with the init container (which would also mount the PV and copy the files oves).

greg commented

2019-03-29 14:26:48 +00:00

What’s wrong with simply deploying them to a PV and mounting that as /custom? I thought that was the intial approach with the init container (which would also mount the PV and copy the files oves).

I missed this question, it's what I'm doing in bbfa3f2 (script that copies everything from the custom dir to the persistent volume we use as the gitea data directory)

> What’s wrong with simply deploying them to a PV and mounting that as /custom? I thought that was the intial approach with the init container (which would also mount the PV and copy the files oves). I missed this question, it's what I'm doing in bbfa3f2 (script that copies everything from the custom dir to the persistent volume we use as the gitea data directory)

raucao commented

2019-03-29 14:44:12 +00:00

Then why does the title say "Add a script to copy the content of the custom folder to a running pod"? Shouldn't it say "copy the custom folder to the persistent volume" in that case?

Also, why is it not mounted as "/custom", but overwriting actual data in the normal Gitea data directory? I still haven't seen any evidence that "/custom" isn't used just because Gitea is run in a different server environment.

Then why does the title say "Add a script to copy the content of the custom folder to a running pod"? Shouldn't it say "copy the custom folder to the persistent volume" in that case? Also, why is it not mounted as "/custom", but overwriting actual data in the normal Gitea data directory? I still haven't seen any evidence that "/custom" isn't used just because Gitea is run in a different server environment.

greg commented

2019-03-29 14:56:56 +00:00

Then why does the title say “Add a script to copy the content of the custom folder to a running pod”? Shouldn’t it say “copy the custom folder to the persistent volume” in that case?

Because you need a running pod to mount a persistent volume

Also, why is it not mounted as “/custom”, but overwriting actual data in the normal Gitea data directory? I still haven’t seen any evidence that “/custom” isn’t used just because Gitea is run in a different server environment.

I had linked to this before: https://docs.gitea.io/en-us/install-with-docker/#customization

Also there has never been a /custom dir, it's a custom dir relative to the gitea binary (see https://docs.gitea.io/en-us/customizing-gitea/), which does not make sense in production.

Here is the line in the Dockerfile that sets the GITEA_CUSTOM env variable to /data/gitea: https://github.com/go-gitea/gitea/blob/master/Dockerfile#L52

> Then why does the title say “Add a script to copy the content of the custom folder to a running pod”? Shouldn’t it say “copy the custom folder to the persistent volume” in that case? Because you need a running pod to mount a persistent volume > Also, why is it not mounted as “/custom”, but overwriting actual data in the normal Gitea data directory? I still haven’t seen any evidence that “/custom” isn’t used just because Gitea is run in a different server environment. I had linked to this before: https://docs.gitea.io/en-us/install-with-docker/#customization Also there has never been a `/custom` dir, it's a `custom` dir relative to the gitea binary (see https://docs.gitea.io/en-us/customizing-gitea/), which does not make sense in production. Here is the line in the Dockerfile that sets the `GITEA_CUSTOM` env variable to `/data/gitea`: https://github.com/go-gitea/gitea/blob/master/Dockerfile#L52

raucao commented

2019-03-30 11:18:01 +00:00

Because you need a running pod to mount a persistent volume

I don't think that's the case. This seems to be exactly what init containers are for, as you suggested in the beginning:

"Init Containers support all the fields and features of app Containers, including resource limits, volumes, and security settings."

See https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

Also there has never been a /custom dir, it’s a custom dir relative to the gitea binary (see https://docs.gitea.io/en-us/customizing-gitea/), which does not make sense in production.

Yes, it's custom/ by default, when not overridden. No need to split hairs over that. My main point is still that the custom dir is different from the data dir, from everything I read so far, and also from how I customized Gitea locally. There's simply no documentation that I could find, that suggests that mixing custom and data together in one directory is how it works. I asked repeatedly to link to such documentation if it exists, so I can understand why merging those two different directories makes sense in our case, but not locally or when deployed without Docker containers.

> Because you need a running pod to mount a persistent volume I don't think that's the case. This seems to be exactly what init containers are for, as you suggested in the beginning: "Init Containers support all the fields and features of app Containers, including resource limits, volumes, and security settings." See https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ > Also there has never been a /custom dir, it’s a custom dir relative to the gitea binary (see https://docs.gitea.io/en-us/customizing-gitea/), which does not make sense in production. Yes, it's `custom/` by default, when not overridden. No need to split hairs over that. My main point is still that the custom dir is different from the `data` dir, from everything I read so far, and also from how I customized Gitea locally. There's simply no documentation that I could find, that suggests that mixing `custom` and `data` together in one directory is how it works. I asked repeatedly to link to such documentation if it exists, so I can understand why merging those two different directories makes sense in our case, but not locally or when deployed without Docker containers.

greg commented

2019-03-30 11:24:32 +00:00

I don’t think that’s the case. This seems to be exactly what init containers are for, as you suggested in the beginning:

An init container is only initialized before running the actual container, so I don't think you can use it to copy files from the local filesystem

Yes, it’s custom/ by default, when not overridden. No need to split hairs over that. My main point is still that the custom dir is different from the data dir, from everything I read so far, and also from how I customized Gitea locally. There’s simply no documentation that I could find, that suggests that mixing custom and data together in one directory is how it works. I asked repeatedly to link to such documentation if it exists, so I can understand why merging those two different directories makes sense in our case, but not locally or when deployed without Docker containers.

As far as I understand there is only one thing that they call the custom dir, and that is a badly named data dir

> I don’t think that’s the case. This seems to be exactly what init containers are for, as you suggested in the beginning: An init container is only initialized before running the actual container, so I don't think you can use it to copy files from the local filesystem > Yes, it’s custom/ by default, when not overridden. No need to split hairs over that. My main point is still that the custom dir is different from the data dir, from everything I read so far, and also from how I customized Gitea locally. There’s simply no documentation that I could find, that suggests that mixing custom and data together in one directory is how it works. I asked repeatedly to link to such documentation if it exists, so I can understand why merging those two different directories makes sense in our case, but not locally or when deployed without Docker containers. As far as I understand there is only one thing that they call the custom dir, and that is a badly named data dir

raucao commented

2019-03-30 11:28:07 +00:00

An init container is only initialized before running the actual container, so I don’t think you can use it to copy files from the local filesystem

You can copy it from the Git repo, if you don't compile an image with the files. That's a suggested use case in the docs for init containers even.

As far as I understand there is only one thing that they call the custom dir, and that is a badly named data dir

Then you haven't actually run Gitea locally and followed the documentation for customizing it. Which I have, and which my changes are based upon. It's clearly a different directory for overriding instead of overwriting things in the data directory.

> An init container is only initialized before running the actual container, so I don’t think you can use it to copy files from the local filesystem You can copy it from the Git repo, if you don't compile an image with the files. That's a suggested use case in the docs for init containers even. > As far as I understand there is only one thing that they call the custom dir, and that is a badly named data dir Then you haven't actually run Gitea locally and followed the documentation for customizing it. Which I have, and which my changes are based upon. It's clearly a different directory for overriding instead of overwriting things in the data directory.

greg commented

2019-03-30 11:48:02 +00:00

OK, I have finally understood.

Gitea files

    GITEA_WORK_DIR: Absolute path of working directory.

    GITEA_CUSTOM: Gitea uses GITEA_WORK_DIR/custom folder by default. Use this variable to change custom directory.

From https://docs.gitea.io/en-us/specific-variables/

OK, I have finally understood. ``` Gitea files GITEA_WORK_DIR: Absolute path of working directory. GITEA_CUSTOM: Gitea uses GITEA_WORK_DIR/custom folder by default. Use this variable to change custom directory. ``` From https://docs.gitea.io/en-us/specific-variables/

greg commented

2019-04-01 15:06:39 +00:00

I have switched over to ship the custom files as a Docker image, that is based on the busybox image (see e0741b4 for details). The files are copied to the persistent volume in the init container. This is much better than what I had come up with previously and is running on GKE

I have switched over to ship the custom files as a Docker image, that is based on the busybox image (see e0741b4 for details). The files are copied to the persistent volume in the init container. This is much better than what I had come up with previously and is running on GKE

raucao commented

2019-05-02 14:39:36 +00:00

@gregkare Any reason this is still open?

greg closed this pull request

2019-05-02 15:27:31 +00:00

greg deleted branch feature/4-label_sets

2019-05-02 15:27:38 +00:00

This repo is archived. You cannot comment on pull requests.

No reviewers