kosmos/chef
kosmos
/
chef
8
3
Fork 0
Code Issues 34 Pull Requests 1 Projects 1 Activity

Bitcoin (Core) cookbook #121

Merged
greg merged 20 commits from feature/bitcoin_cookbook into master 2020-08-20 17:17:31 +00:00
Conversation 19 Commits 20 Files Changed 20 +670 -3
raucao commented 2020-01-21 20:26:54 +00:00
Owner
Copy Link

Installs and configures Bitcoin Core on Ubuntu via the official snap package. (Update: we now only use the new source recipe, which compiles from source.)

@gregkare The installation of the actual bitcoin-core snap package via the snap_package resource makes the chef client process hang for me, and I'm stuck trying to figure out why. I tried to attach strace, but it's too late and I don't see a way of starting the chef run as an argument for strace from within the VM. Maybe you could see if it hangs on your machine, too, and maybe you'd have an idea for how to find out more?

Installs and configures [Bitcoin Core](https://github.com/bitcoin/bitcoin) on Ubuntu via the official snap package. (Update: we now only use the new `source` recipe, which compiles from source.) @gregkare The installation of the actual `bitcoin-core` snap package via the `snap_package` resource makes the chef client process hang for me, and I'm stuck trying to figure out why. I tried to attach strace, but it's too late and I don't see a way of starting the chef run as an argument for strace from within the VM. Maybe you could see if it hangs on your machine, too, and maybe you'd have an idea for how to find out more?
greg commented 2020-01-22 10:44:05 +00:00
Owner
Copy Link

Sure, I will investigate

Sure, I will investigate
greg commented 2020-01-22 11:35:47 +00:00
Owner
Copy Link

It's not just the bitcoin-core snap package, even the hello-world package has the same behavior. I found this issue on the chef repo: https://github.com/chef/chef/issues/8827

So it looks like snap_package is completely broken, at least on Ubuntu 18.04 (and 19.04, as the person who reported the issue is running that). In the mean time we can work around it by not using it, instead doing something like:

execute "snap install bitcoin-core"

snap install returns 0 when a package already exists so no need for a guard clause

I have pushed a few changes, including running snap install as an execute command and fixing the mounting of the directory (typo in an attribute name)

It's not just the bitcoin-core snap package, even the hello-world package has the same behavior. I found this issue on the chef repo: https://github.com/chef/chef/issues/8827 So it looks like `snap_package` is completely broken, at least on Ubuntu 18.04 (and 19.04, as the person who reported the issue is running that). In the mean time we can work around it by not using it, instead doing something like: ```ruby execute "snap install bitcoin-core" ``` `snap install` returns 0 when a package already exists so no need for a guard clause I have pushed a few changes, including running `snap install` as an execute command and fixing the mounting of the directory (typo in an attribute name)
raucao commented 2020-01-22 14:08:33 +00:00
Author
Owner
Copy Link

Great, thanks!

Great, thanks!
greg was assigned by raucao 2020-01-22 14:08:40 +00:00
raucao self-assigned this 2020-01-22 14:08:40 +00:00
greg commented 2020-01-23 16:56:09 +00:00
Owner
Copy Link

I created an issue about the invalid systemd unit when passing a hash as a config: https://github.com/chef/chef/issues/9259

I created an issue about the invalid systemd unit when passing a hash as a config: https://github.com/chef/chef/issues/9259
raucao commented 2020-01-23 17:15:30 +00:00
Author
Owner
Copy Link

Doesn't it also add newlines around every key/value pair? I'm pretty sure it did that on my machine.

Doesn't it also add newlines around every key/value pair? I'm pretty sure it did that on my machine.
greg commented 2020-01-24 14:15:22 +00:00
Owner
Copy Link

For me it only adds a newline before the start of a section, so that behavior is correct

For me it only adds a newline before the start of a section, so that behavior is correct
raucao commented 2020-01-24 15:37:37 +00:00
Author
Owner
Copy Link

Strange. Maybe a difference between Linux and macOS hosts?

Strange. Maybe a difference between Linux and macOS hosts?
raucao commented 2020-08-17 08:46:42 +00:00
Author
Owner
Copy Link

Status update:

I wrote a whole new source recipe yesterday (see latest commit), and got everything working as intended. However, I ended up with a leveldb issue when starting bitcoin, which I wasn't able to fix. So I have it re-indexing the whole chain database now.

Status update: I wrote a whole new source recipe yesterday (see latest commit), and got everything working as intended. However, I ended up with a leveldb issue when starting bitcoin, which I wasn't able to fix. So I have it re-indexing the whole chain database now.
raucao commented 2020-08-17 09:10:22 +00:00
Author
Owner
Copy Link

After a while of indexing, it runs into the same issue. Goes something like:

LevelDB read failure: IO error: /mnt/data/bitcoin/chainstate/1629202.ldb: Too many open files

I have already upped the ulimit for open files to unlimited in the systemd service config (and tried doing it via ulimit before). But somehow it seems to be stuck at the 1024 default, no matter what I do.

I'm wondering if there's a different issue, and this error message is just a symptom. In any case, I'm completely stuck now. Already spent hours trying to fix it.

Anyone has ideas for debugging it further?

After a while of indexing, it runs into the same issue. Goes something like: > LevelDB read failure: IO error: /mnt/data/bitcoin/chainstate/1629202.ldb: Too many open files I have already upped the ulimit for open files to unlimited in the systemd service config (and tried doing it via ulimit before). But somehow it seems to be stuck at the 1024 default, no matter what I do. I'm wondering if there's a different issue, and this error message is just a symptom. In any case, I'm completely stuck now. Already spent hours trying to fix it. Anyone has ideas for debugging it further?
raucao commented 2020-08-17 10:27:37 +00:00
Author
Owner
Copy Link

Hmm:

Note that most process resource limits configured with these options are per-process, and processes may fork in order to acquire a new set of resources that are accounted independently of the original process, and may thus escape limits set

So maybe it's worth fixing the ulimit after all.

Unfortunately, the current ulimit cookbook and resource didn't work for me. When I opened a shell as the satoshi user, it would always show me the default value again. Same when I tried setting it in /etc/security/limits.conf.

I also set a high value in /proc/sys/fs/file-max and ran sysctl -p to apply it, and that also didn't work.

I must be doing something terribly wrong for all the normal methods to fail like that.

Hmm: > Note that most process resource limits configured with these options are per-process, and processes may fork in order to acquire a new set of resources that are accounted independently of the original process, and may thus escape limits set So maybe it's worth fixing the ulimit after all. Unfortunately, the current ulimit cookbook and resource didn't work for me. When I opened a shell as the `satoshi` user, it would always show me the default value again. Same when I tried setting it in `/etc/security/limits.conf`. I also set a high value in `/proc/sys/fs/file-max` and ran `sysctl -p` to apply it, and that also didn't work. I must be doing something terribly wrong for all the normal methods to fail like that.
greg commented 2020-08-17 11:22:45 +00:00
Owner
Copy Link

We're using the ulimit cookbook to set the file handle limit for dirsrv (2c40ad4591/site-cookbooks/kosmos-dirsrv/resources/instance.rb (L16-L18)) and that works on barnard:

$ ulimit -a | grep "open files"
open files                      (-n) 40960

Can you post the code you're using?

We're using the ulimit cookbook to set the file handle limit for dirsrv (https://gitea.kosmos.org/kosmos/chef/src/commit/2c40ad4591e277ae1d74d7510fdfd06f88a94222/site-cookbooks/kosmos-dirsrv/resources/instance.rb#L16-L18) and that works on barnard: ``` $ ulimit -a | grep "open files" open files (-n) 40960 ``` Can you post the code you're using?
raucao commented 2020-08-17 11:46:22 +00:00
Author
Owner
Copy Link

I used that exact code when trying it with that resource. The question is not so much about that resource as about none of the various methods working at all.

I used that exact code when trying it with that resource. The question is not so much about that resource as about *none* of the various methods working at all.
raucao commented 2020-08-17 13:47:11 +00:00
Author
Owner
Copy Link

I found out that limits from limits.conf don't apply to systemd services at all, but it still doesn't explain why opening a shell for that user still shows the default limit when there are entries in the config file that change it.

https://bugzilla.redhat.com/show_bug.cgi?id=754285

I found out that limits from `limits.conf` don't apply to systemd services at all, but it still doesn't explain why opening a shell for that user still shows the default limit when there are entries in the config file that change it. https://bugzilla.redhat.com/show_bug.cgi?id=754285
raucao commented 2020-08-17 14:15:39 +00:00
Author
Owner
Copy Link

OK, I also just confirmed that this is happening with a ulimit set correctly in a shell session when running the exec command manually. So the issue is something else.

Unfortunately, there's nobody on the Web who had the same issue with this program apparently. :/

OK, I also just confirmed that this is happening with a ulimit set correctly in a shell session when running the exec command manually. So the issue is something else. Unfortunately, there's nobody on the Web who had the same issue with this program apparently. :/
raucao commented 2020-08-17 14:50:57 +00:00
Author
Owner
Copy Link

Turns out that this is an unknown issue with encfs (and/or leveldb).

I pushed a change to only store the wallet data in encfs, but use the default directory (still configurable) for the rest.

As I nuked all data, it is currently re-syncing the whole chain. So I'm going to revisit this when sync is done.

Turns out that this is an unknown issue with encfs (and/or leveldb). I pushed a change to only store the wallet data in encfs, but use the default directory (still configurable) for the rest. As I nuked all data, it is currently re-syncing the whole chain. So I'm going to revisit this when sync is done.
raucao commented 2020-08-18 12:46:49 +00:00
Author
Owner
Copy Link

Another few improvements later, and it's finally finished! 🎉

Another few improvements later, and it's finally finished! 🎉
raucao changed title from WIP: Bitcoin (Core) cookbook to Bitcoin (Core) cookbook 2020-08-18 12:47:42 +00:00
greg requested changes 2020-08-19 12:51:51 +00:00
greg left a comment
Owner
Copy Link

Looks good to me apart from the Vagrant file change and the license file. I also don't see the ulimit change anywhere, did you forget to push it?

Looks good to me apart from the Vagrant file change and the license file. I also don't see the ulimit change anywhere, did you forget to push it?
Vagrantfile Outdated
@ -96,3 +96,3 @@
chef.add_recipe 'kosmos-base'
chef.add_recipe 'kosmos_encfs'
# chef.add_recipe 'kosmos-postgresql::default'
# chef.add_recipe 'kosmos-bitcoin::snapd'
greg commented 2020-08-19 12:46:58 +00:00
Owner
Copy Link

I don't think that should have been checked in

I don't think that should have been checked in
raucao commented 2020-08-19 13:27:36 +00:00
Author
Owner
Copy Link

Why not? It's commented, so you can uncomment it to easily run the recipe locally. Same as all the other commented recipes.

Why not? It's commented, so you can uncomment it to easily run the recipe locally. Same as all the other commented recipes.
site-cookbooks/kosmos-bitcoin/LICENSE Outdated
@ -0,0 +1,3 @@
Copyright 2019 The Authors
greg commented 2020-08-19 12:48:49 +00:00
Owner
Copy Link

The license should be MIT like the other kosmos cookbooks

The license should be MIT like the other kosmos cookbooks
raucao marked this conversation as resolved
raucao commented 2020-08-19 13:25:56 +00:00
Author
Owner
Copy Link

I also don’t see the ulimit change anywhere, did you forget to push it?

Which ulimit change are you referring to?

> I also don’t see the ulimit change anywhere, did you forget to push it? Which ulimit change are you referring to?
raucao requested review from greg 2020-08-19 13:29:51 +00:00
greg commented 2020-08-20 10:07:42 +00:00
Owner
Copy Link

Which ulimit change are you referring to?

I don't see the code that sets the file handle limit for the satoshi user

> Which ulimit change are you referring to? I don't see the code that sets the file handle limit for the satoshi user
raucao commented 2020-08-20 11:39:16 +00:00
Author
Owner
Copy Link

That's because...

I found out that limits from limits.conf don’t apply to systemd services at all

With systemd services, this is done via the LimitNOFILE config.

See https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process%20Properties

(Which also makes the ulimit cookbook obsolete for anything that is run as a systemd service.)

That's because... > I found out that limits from limits.conf don’t apply to systemd services at all With systemd services, this is done via the `LimitNOFILE` config. See https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process%20Properties (Which also makes the ulimit cookbook obsolete for anything that is run as a systemd service.)
greg approved these changes 2020-08-20 15:20:51 +00:00
greg left a comment
Owner
Copy Link

LGTM!

I didn't see the LimitNOFILE option in the systemd unit

LGTM! I didn't see the `LimitNOFILE` option in the systemd unit
greg merged commit 414a595719 into master 2020-08-20 17:17:31 +00:00
greg deleted branch feature/bitcoin_cookbook 2020-08-20 17:17:38 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
greg
Labels
Clear labels   
service
accounts
Kosmos Accounts

  
service
discourse
Kosmos Community Forums

  
service
drone-ci
Kosmos Drone CI

  
service
email
mail.kosmos.org

  
service
garage
S3-compatible object storage

  
service
gitea
Kosmos Gitea

  
service
ipfs
Kosmos IPFS

  
service
mastodon
kosmos.social

  
service
postgres
Database cluster

  
service
remotestorage
Portable data storage for the Web

  
service
wiki
Kosmos Wiki

  
service
xmpp
Kosmos Chat

  
bug

Something is not working

  
design

Graphic/visual design

  
dev environment

Config, builds, CI, deployment, etc.

  
docs

Documentation

  
duplicate

This issue or pull request already exists

  
enhancement

Improving existing functionality

  
feature

New functionality

  
good first issue

Dive in, and start contributing

  
idea

Something to consider

  
invalid

Not a bug

  
kredits-1

Small contribution

  
kredits-2

Medium contribution

  
kredits-3

Large contribution

  
on hold

Currently not actionable

  
ops

Manual IT ops activities

  
question

Looking for an answer

  
release
major

  
release
minor

  
release
patch

  
security

All your base are belong to us

  
ui/ux

User interface, process design, etc.

  
wontfix

This won't be fixed

No Label
service
accounts
service
discourse
service
drone-ci
service
email
service
garage
service
gitea
service
ipfs
service
mastodon
service
postgres
service
remotestorage
service
wiki
service
xmpp
bug
design
dev environment
docs
duplicate
enhancement
feature
good first issue
idea
invalid
kredits-1
kredits-2
kredits-3
on hold
ops
question
release
major
release
minor
release
patch
security
ui/ux
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
greg
raucao
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/chef#121
No description provided.
Delete Branch "feature/bitcoin_cookbook"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?