Speed/disconnection issues when using raf and dat-storage

[joehand]

@blahah and I were debugging some speed + connection issues today. There are a few things causing bad transfer speeds. I can debug this a bit tomorrow and try to open more specific issues/test PRs. Hungry now and wanted to file before I forget =).

1. Frequent disconnects coming from dat-storage (I think not being able to locate data, see below).
2. raf storage (single file or multiple) causing hyperdrive to go slower. This one needs more debugging. Sometimes seems like raf, default hyperdrive storage, is much slower than using ram.

1 is definitely related to writing files + raf. I see it most with dat-storage and sometimes (I think, harder to reproduce) with regular raf on the content/data file.

The dat-storage error, could not locate data causes the replication stream to end and forces client to reconnect. Example error:

events.js:163
      throw er; // Unhandled 'error' event
      ^

Error: Could not locate data
    at /Users/rik/.nvm/versions/node/v7.7.4/lib/node_modules/dat-next/node_modules/dat-storage/index.js:50:30
    at loop (/Users/rik/.nvm/versions/node/v7.7.4/lib/node_modules/dat-next/node_modules/dat-storage/index.js:108:19)
    at /Users/rik/.nvm/versions/node/v7.7.4/lib/node_modules/dat-next/node_modules/dat-storage/index.js:75:5
    at /Users/rik/.nvm/versions/node/v7.7.4/lib/node_modules/dat-next/node_modules/hypercore/index.js:888:5
    at onread (/Users/rik/.nvm/versions/node/v7.7.4/lib/node_modules/dat-next/node_modules/random-access-file/index.js:106:25)
    at FSReqWrap.wrapper [as oncomplete] (fs.js:629:17)

[mafintosh]

I think we solved most of this

[derhuerst]

I got this error today, trying to add files to a dat-node dat instance via dat.archive.writeFile. See also the where I pass in dat.archive. It always seems to crash at the same location/point in time, after ~1 minute. For context: I'm working on dat-ecosystem-archive/dat-node#165.

[joehand]

@derhuerst can you send the full error message you got? The code for dat-storage has changed quite a bit and I'm not sure what code the original error corresponds to anymore.

It always seems to crash at the same location/point in time, after ~1 minute.

Is the writing happening when it crashes or something else going on after the write finishes?

[derhuerst]

Right after running npm update before inside build-wikipedia-feed:

cat enwiki-20170720-stub-meta-current.xml.gz | gunzip | build-wikipedia-feed/revisions-list.js 2>/dev/null | build-wikipedia-feed/store-revisions.js

dat /root/.local/share/p2p-wiki/db eb4890cc39eb966cddb35117510405ed8fe26221901ff3d25a9d1e4bdc83a26a
13 @ 783865149 ✓
14 @ 783865160 ✓
10 @ 767284433 ✓
15 @ 783865293 ✓
18 @ 783865299 ✓
19 @ 783821589 ✓
20 @ 783821738 ✓
…
640 @ 789562124 ✓
642 @ 780371012 ✓
643 @ 789562051 ✓
644 @ 375387155 ✓
648 @ 769390556 ✓
Error: Could not locate data
    at /root/wiki-dumps/build-wikipedia-feed/node_modules/dat-storage/index.js:61:30
    at loop (/root/wiki-dumps/build-wikipedia-feed/node_modules/dat-storage/index.js:132:19)
    at /root/wiki-dumps/build-wikipedia-feed/node_modules/dat-storage/index.js:97:5
    at /root/wiki-dumps/build-wikipedia-feed/node_modules/hypercore/index.js:1061:5
    at onread (/root/wiki-dumps/build-wikipedia-feed/node_modules/random-access-file/index.js:130:14)
    at FSReqWrap.wrapper [as oncomplete] (fs.js:664:17)

Is the writing happening when it crashes or something else going on after the write finishes?

I'm opening the dat archive using dat-node, without joining the network (or anything else), and then write many files in parallel, as described above.

[derhuerst]

Any news on this? I still run across this error.