Improve IPFS memory usage #52

New Issue

2019-05-01T12:03:22Z

raucao commented

2019-05-01 12:03:22 +00:00

I found some hints for reducing memory usage in these GitHub comments:

Bandwidth metric tracking. Unfortunately, we never forget old peers. You can disable bandwidth tracking by setting ipfs config --json "Swarm.DisableBandwidthMetrics true".

If you're memory constrained, try running the daemon with --routing=dhtclient

I found some hints for reducing memory usage [in these GitHub comments](https://github.com/ipfs/go-ipfs/issues/3532): > Bandwidth metric tracking. Unfortunately, we never forget old peers. You can disable bandwidth tracking by setting ipfs config --json "Swarm.DisableBandwidthMetrics true". > If you're memory constrained, try running the daemon with --routing=dhtclient

raucao changed title from ~~IPFS memory usage~~ to Improve IPFS memory usage

2019-05-01 12:03:37 +00:00

raucao commented

2019-05-01 12:16:57 +00:00

By the way, the config contains lots of settings that we most likely want to change for the small DO boxes that are supposed to mostly keep our own data replicated. E.g. minimum and maximum number of peer connections: https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#basic-connection-manager

greg commented

2019-05-02 09:37:45 +00:00

Cool, giving these a try manually on barnard

greg commented

2019-05-02 11:50:29 +00:00

I tried all of these and the memory usage was still getting up to 1GB in only a few minutes. I have tried something else: removing all the bootstrap nodes from the config. After this change, the ipfs process is using ~150MB after 30 minutes, let's see how this evolves.

I think every ipfs node needs to have the other members of the cluster as bootstrap nodes in case they don't have 100% uptime. If the cluster is up they get all the pins, but if the cluster daemon is down while a document is created they wouldn't be able to load the document

I tried all of these and the memory usage was still getting up to 1GB in only a few minutes. I have tried something else: removing all the bootstrap nodes from the config. After this change, the ipfs process is using ~150MB after 30 minutes, let's see how this evolves. I think every ipfs node needs to have the other members of the cluster as bootstrap nodes in case they don't have 100% uptime. If the cluster is up they get all the pins, but if the cluster daemon is down while a document is created they wouldn't be able to load the document

raucao commented

2019-05-02 13:40:51 +00:00

if the cluster daemon is down while a document is created they wouldn’t be able to load the document

How does that make sense, when we create documents through the cluster node's API instead of the IPFS daemon one? Doesn't that mean you cannot create docs when the cluster process is down?

> if the cluster daemon is down while a document is created they wouldn’t be able to load the document How does that make sense, when we create documents through the cluster node's API instead of the IPFS daemon one? Doesn't that mean you cannot create docs when the cluster process is down?

greg commented

2019-05-02 15:27:16 +00:00

No, I'm just saying if you're not connected to the cluster when the document is created, you wouldn't be able to get the document after the fact (if you're not connected to any of our nodes that have the document)

greg commented

2019-05-02 15:28:39 +00:00

Update: even removing all bootstrap nodes from the config doesn't help with the ridiculous memory consumption. After 4 hours the ipfs daemon on barnard is using 625M of RAM

raucao commented

2019-05-02 20:10:30 +00:00

No, I’m just saying if you’re not connected to the cluster when the document is created, you wouldn’t be able to get the document after the fact (if you’re not connected to any of our nodes that have the document)

I don't understand what this means. Who is "you" in this case? And why wouldn't one be able to get a document, when it's available on the IPFS network in general?

> No, I’m just saying if you’re not connected to the cluster when the document is created, you wouldn’t be able to get the document after the fact (if you’re not connected to any of our nodes that have the document) I don't understand what this means. Who is "you" in this case? And why wouldn't one be able to get a document, when it's available on the IPFS network in general?

greg commented

2019-05-03 07:48:01 +00:00

There's a WIP PR for ipfs-go that's supposed to fix the memory leaks caused by storing the peerstore in memory: https://github.com/ipfs/go-ipfs/pull/6080

On barnard go-ipfs is now using 716M of RAM after 15 hours

There's a WIP PR for ipfs-go that's supposed to fix the memory leaks caused by storing the peerstore in memory: https://github.com/ipfs/go-ipfs/pull/6080 On barnard go-ipfs is now using 716M of RAM after 15 hours

greg commented

2019-05-03 08:36:37 +00:00

I don’t understand what this means. Who is “you” in this case? And why wouldn’t one be able to get a document, when it’s available on the IPFS network in general?

Yeah I got confused, even without bootstrap nodes it's still connecting to the network

> I don’t understand what this means. Who is “you” in this case? And why wouldn’t one be able to get a document, when it’s available on the IPFS network in general? Yeah I got confused, even without bootstrap nodes it's still connecting to the network

greg commented

2019-05-03 09:07:07 +00:00

In the meantime we can use Systemd's MemoryHigh and MemoryMax to keep the memory usage of go-ipfs under control: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#MemoryHigh=bytes

Testing it on barnard

In the meantime we can use Systemd's MemoryHigh and MemoryMax to keep the memory usage of go-ipfs under control: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#MemoryHigh=bytes Testing it on barnard

👍 1

greg commented

2019-05-07 12:40:13 +00:00

We have significantly lowered memory usage on barnard (by lowering the min and max number of connections, disabling bandwidth metrics). The daemon is using 80MB of RAM after 40 minutes on barnard.

The master branch of our chef repo now uses this WIP PR in the ipfs-cookbook repo: https://github.com/67P/ipfs-cookbook/pull/4 (I need to make the attributes configurable, but first we're continuing the migration)

We have significantly lowered memory usage on barnard (by lowering the min and max number of connections, disabling bandwidth metrics). The daemon is using 80MB of RAM after 40 minutes on barnard. The master branch of our chef repo now uses this WIP PR in the ipfs-cookbook repo: https://github.com/67P/ipfs-cookbook/pull/4 (I need to make the attributes configurable, but first we're continuing the migration)

greg commented

2019-06-17 09:08:29 +00:00

We forgot to close this one. We merged https://github.com/67P/ipfs-cookbook/pull/4

No PR since the change were done to an external cookbook, so I'm adding a label and @raucao and I as assignees to this issue

We forgot to close this one. We merged https://github.com/67P/ipfs-cookbook/pull/4 No PR since the change were done to an external cookbook, so I'm adding a label and @raucao and I as assignees to this issue

greg self-assigned this 2019-06-17 09:08:39 +00:00

raucao was assigned by greg

2019-06-17 09:08:40 +00:00

greg closed this issue

2019-06-17 09:08:52 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kosmos/chef#52