Firefox - crashes with excessive thread activity

[mitra42]

• Version: ipfs 0.25.1 (output of npm list | grep ipfs)
• Platform: Mac OSX 10.11.6 and 10.12.6
• Subsystem: JS - https://unpkg.com/ipfs-iiif-db/dist/browser.js - but unsure what part

Type: Bug

Severity: Severe

Description:

The most serious bug - that makes the system unusable - is the extreme thread usage on Firefox. Firefox normally runs with about 60 threads (according to Mac Activity Monitor)

As soon as I open a page which starts IPFS/IIIF it adds about another 40 threads, BUT the Idle Wake Ups starts hitting the 10’s of thousands (three orders of magnitude more than anything else on the box). Load Average (as reported by “top” or “uptime" starts rising and quickly exceeds 100, by which time the machine is running too slow to do much more than kill Firefox.

Is anyone else running js-ipfs in browser, are you seeing this especially with iiif ?

Note - on Chrome the behavior is different - there aren't as many wake ups, but CPU load grows (slower) until it hits a point where Chrome gives the "Aw Snap! Something went wrong while displaying this webpage." error.

Steps to reproduce the error:

[haadcode]

I've seen this has started happening in Orbit, too (see orbitdb-archive/orbit-web#9 (comment) and note that the bug itself is not related).

"That said, I've noticed Orbit has started randomly crashing (both Chrome and FF) and on Firefox it just kills the browser completely (hangs). This is seemingly random and I haven't been able to debug it yet."

This is indeed the same behaviour as described above by @mitra42.

Important to note here is that Orbit's code (deployed at orbit.chat) hasn't changed since April and as far as I can tell, only the browsers have gotten updated. So that makes me think that a) we always had this problem or b) it's something shared between the browsers (eg. webrtc). Or both.

There's an open bug for Firefox at https://bugzilla.mozilla.org/show_bug.cgi?id=1389812 which also has some interesting details re. webrtc usage, perhaps related.

[daviddias]

Thanks for reporting. This mostly due to WebRTC being hungry for machine resources and the fact that so many operations are happening on the Main thread.

Solutions for this problem are:

• Ideal - Figure out Connection Closing for IPFS and implement it. This is doable and one of our priorities.
• Also ideal - WebRTC supported in Service and Web Workers. You can help this cause by contributing to the discussions on Awesome WebWorkers Support #725 (comment).
• Things you can do now to mitigate the memory consumption issues:
  • Avoid using the open WebRTC rendezvous point (which is connecting you to the whole network) and use your own. https://github.com/libp2p/js-libp2p-webrtc-star#hosted-signalling-server
  • Avoid using WebRTC at all and put IPFS running on a Service or WebWorker
  • Contribute to our performance reviews \o/ (read more at Video player with js-ipfs #128)

[daviddias]

Another thing that is specially annoying is Chrome's aggressive resource throttling and the lack of way to disable that (other than having things on a Service Worker).

[mitra42]

OK - this sounds pretty bad ... If I understand you.

• We could run our own server at the Internet Archive following instructions at that address.
• BUT wouldn't that disconnect us from the rest of IPFS, which somewhat removes the point of using a Content-addressed system in the first place.
• ALSO - that still has scaling issues, i.e. as soon as any apps scaled they'd hit the same issue.
• I don't quite get your point "Avoid using WebRTC at all and put IPFS running on a Service or WebWorker". We are using WebRTC because in order to get the ability to post/retrieve/notify on a list one of you suggested I used IIIF which specifies WebRTC.

And moving to a WebWorker seems like a massive re-architecting of the upper layers to get around this bug in lower layers, and I'm not sure it would even work since the problem is that the WebRTC (or how IIIF/IPFS is using it) causes it to consume huge amounts of resources.

I find on Chrome our demos crash within about 5 minutes, just sitting there with no UI or interaction with IPFS other than having started it.
IpfsIiifDb( {"ipfs":{"repo":"/tmp/ipfs_dweb20170820","config":{"Addresses":{"Swarm":["/libp2p-webrtc-star/dns4/star-signal.cloud.ipfs.team/wss"]},"Discovery":{"webRTCStar":{"Enabled":true}}},"EXPERIMENTAL":{"pubsub":true}},"store":"indexeddb","partition":"dweb20170820"} )

[daviddias]

@mitra42 seems that my points were missing some context. To clarify:

Ideal - Figure out Connection Closing for IPFS and implement it. This is doable and one of our priorities.

Connection Closing is a feature that any P2P network requires, it is currently a bottleneck both in go-ipfs and js-ipfs that we've been actively working towards a scalable solution. We've multiple ideas and are discussing how to implement it.

You just happen to notice it faster in js-ipfs because a) The browser has way less resources than a go-ipfs running a desktop and b) WebRTC is super resource intensive, try any video chat app that uses it nowadays and your laptop fans will start flying :)

You can implement an application level connection closing policy and close connections that no longer interest you.

We could run our own server at the Internet Archive following instructions at that address.
BUT wouldn't that disconnect us from the rest of IPFS, which somewhat removes the point of using a Content-addressed system in the first place.

Yes, I proposed that and the reason is that it would be a way to introduce the nodes that are interested into the same content without having to connect to every other node in the network that is being booted for other apps.

It is a good policy to connect nodes that are interested in a topic faster (i.e orbit with orbit nodes, IIIF with IIIF nodes, etc).

Content Addressing will still work.

I don't quite get your point "Avoid using WebRTC at all and put IPFS running on a Service or WebWorker". We are using WebRTC because in order to get the ability to post/retrieve/notify on a list one of you suggested I used IIIF which specifies WebRTC.

WebRTC is resource intensive and browsers do not like that. You could actually turn on WebRTC transport and disable WebRTC-Star discovery that connects all the nodes automatically, enabling you to pick the ones you are using.

We are developing Relay, which will enable Browser nodes to connect to any other node in the network through a WebSockets connection, reducing vastly the resources consumption.

And moving to a WebWorker seems like a massive re-architecting of the upper layers to get around this bug in lower layers,

Separating the View layer from the Transport Layer shouldn't come as a foreign idea, that already happens for all the other transports in the Browser. In fact, we are already working on a Browser integration that would put IPFS in a background process so that you don't even have to load it in your app and one of the other solutions is exposing IPFS through an Extension.

There is a lot of work ahead of us, but progress moving steadily and today we are able to run full P2P protocol on the browser, things will just get better moving forward :)

@mitra42 if you and your team would like to work more closely with us, either by creating tests, identifying performance benchmarks and so on, please let us know, we appreciate help from our open source contributors and I'm happy to give you pointers on how to improve certain things :)

[mitra42]

You can implement an application level connection closing policy and close connections that no longer interest you.

How can we do this at the app level ? We aren't opening and closing connections at the app level, this is happening entirely inside IPFS library code.

We can run our own server, but as I said there will still be a scaling problem when the number of nodes on any app exceeds enough to bring WebRTC to its knees.

Content Addressing will still work.

Maybe I'm misunderstanding something, but my assumption was that if we have the app connect to a separate webrtc server and store a block, and someone on some other app refers to the same block elsewhere (by its hash) will they still get it ?

Re your point on webrtc, I want to be clear - we aren't explicitly using WebRTC from choice and certainly aren't explicitly conncting to other nodes since anything will be running in the browser. I was told (by Matt) that the best way to do what we needed was to use IIIF and the IIIF examples used WebRTC (I'm pretty sure that's where the "WebRTC" link came from).

Separating the View layer from the Transport Layer shouldn't come as a foreign idea .

Agreed, and they are separate in our framework, its the Transport Layer that is causing the problems ! Its still a massive job to rearchitect something to use WebWorkers.

@diasdavid - I'd be happy to jump on a call to figure out to make this move more smoothly, and be more of help.

[daviddias]

How can we do this at the app level ? We aren't opening and closing connections at the app level, this is happening entirely inside IPFS library code.

You can disable Discovery. It defaults to true (see https://github.com/ipfs/js-ipfs/blob/master/src/core/runtime/config-browser.json#L13-L16) but you can overload the config as it is explained in this section of the README https://github.com/ipfs/js-ipfs#advanced-options-when-creating-an-ipfs-node

We can run our own server, but as I said there will still be a scaling problem when the number of nodes on any app exceeds enough to bring WebRTC to its knees.

The number of open WebRTC Connections needs to be under control more strictly than other types of connections.

Relay will help this too because we will be able to multiplex multiple connections to multiple peers over one socket.

I was told (by Matt) that the best way to do what we needed was to use IIIF and the IIIF examples used WebRTC (I'm pretty sure that's where the "WebRTC" link came from).

And that was a good suggestion. The IIIF team enabled shared annotations over IPFS using IIIF-DB which was designed to fit the needs of the project. I believe @flyingzumwalt was suggesting to use IIIF-DB as an inspiration and a learning tool of how to create a CRDT powered DB over IPFS and not as here is the off the shelf solution that solves all your problems. Bare in mind that IIIF DB is still under active development. You read me more about CRDT and the work being actively developed at https://blog.ipfs.io/30-js-ipfs-crdts.md.

As we talked in the past over email some months ago, it would be excellent if you could share with us your goal for the application and the architecture that you are building, I feel that there is some shoehorn happening here and that is creating some obstacles. Happy to chat more next week :)

[mitra42]

Yes - our goal was, and still is, to build on top of IPFS, however pretty much everything we tried didn't function or didn't function as documented, or wasn't available on JS-IPFS (e.g. IPNS) or depended on something else which didn't work. That's how we ended up with IIIF, because something else (I think Y-connector) wasn't working as documented.

Our docs - very much a work in progress so are still on Google Docs, but it doesn't really cover the key primitive we are trying to get IPFS to do - which works, except that it crashes because of this WRTP issue. Lets setup a call for next week - I have a lot of flexibility.

[mitra42]

Its unclear from those links what the implication of disabling discover is. If we disable discovery as suggested above will the pubsub feature still work, and will a resource stored at one node be viewable at another?

[daviddias]

@mitra42 PubSub will work still. You can learn more about what Discovery is and how things get plugged in the Tutorials of libp2p https://github.com/libp2p/js-libp2p/tree/master/examples

• Peer Discovery - https://github.com/libp2p/js-libp2p/tree/master/examples/discovery-mechanisms
• PubSub - https://github.com/libp2p/js-libp2p/tree/master/examples/pubsub

I'll review your Google Doc over the weekend and ping you to chat next week :)