Update documentation

[daviddias]

• get importer up to the new path + stream format
• integrate exporter onto js-ipfs
• create docs of how importer and exporter works

Some notes:

var importer = new unixFSE.import()
    importer.add({path: , stream:)
        chunk the file, store the hash
    importer.finish()

    - default “no flush”, in the future, support flush
    - each time a file is hashed and added, emit an event with that filepath and hash
    - when the dir gets added, also emit an event for that
    - use a env variable to use main IPFS repo, just for the tests
    - I can sort (at ipfs-core) before printing :)

var exporter = new unixFSE.export()
    each time it hits a file emit an event with a file and write to the stream

Proposal:

I want to rename this module to unixfs-engine, because it is what it really is, the agnostic file importer will be done later, this is not prepared in any sense to support different layouts (nor do we know yet how to achieve that).

[nginnever]

A name change is something that I think is a really good idea since data-importing does not cover the other half of what the module is doing. I really like the name Unixfs-engine.

[daviddias]

moar notes (from last call)

var importer = new unixFSE.importer()

importer.add({path: , stream:)
    chunk the file, store the hash

importer.finish()

- default “no flush”, in the future, support flush
- each time a file is hashed and added, emit an event with that filepath and hash
- when the dir gets added, also emit an event for that
- use a env variable to use main IPFS repo, just for the tests
- It can sort (at ipfs-core) before printing :)

NO FLUSH

importer.add({path: /a/foo.txt, stream: <>}
->

1. foo gets chunked and converted to unixfs/merkledag
2. importer.on('something', {path:/a/foo.txt, hash: QmHASHafootxt}
   importer.add({path: /a/baz.txt, stream: <>}
3. foo gets chunked and converted to unixfs/merkledag
4. importer.on('something', {path:/a/foo.txt, hash: QmHASHafootxt}

...
importer.finish()
importer.on('something', {path: /a, hash: QmHASHofdirA}

FLUSH ON

importer.add({path: /a/foo.txt, stream: <>}
->

1. foo gets chunked and converted to unixfs/merkledag
2. importer.on('something', {path:/a/foo.txt, hash: QmHASHafootxt}
3. importer.on('something', {path:/a, hash: QmHASHdirofA}
   importer.add({path: /a/baz.txt, stream: <>}
4. foo gets chunked and converted to unixfs/merkledag
5. importer.on('something', {path:/a/foo.txt, hash: QmHASHafootxt}
6. importer.on('something', {path:/a, hash: QmHASHnewDirofA}

...
importer.finish()
importer.on('something', {path: /a, hash: QmHASHofdirA}

[daviddias]

@nginnever can we get a status update on the js-ipfs integration? Thank you.

[daviddias]

Tracking at: ipfs/js-ipfs#156

[daviddias]

@nginnever would you be able to update the documentation of this module, please? Thank you :)

[nginnever]

Sure, will do that today!