Merge branch 'master' into fix-webpack-example

Author: daviddias

README.md

@@ -116,12 +116,18 @@ The CLI is available by using the command `jsipfs` in your terminal. This is ali
 Learn how to bundle with browserify and webpack in the [`examples`](https://github.com/ipfs/js-ipfs/tree/master/examples) folder.
 
 
-You can also load it using a `<script>` using the [unpkg](https://unpkg.com) CDN. Inserting one of the following lines will make a `Ipfs` object available in the global namespace.
+You can also load it using a `<script>` using the [unpkg](https://unpkg.com) CDN or the [jsDelivr](https://www.jsdelivr.com/package/npm/ipfs) CDN. Inserting one of the following lines will make a `Ipfs` object available in the global namespace.
 
 ```html
-<script src="https://unpkg.com/ipfs/dist/index.min.js"></script> <!-- loading the minified version -->
-<script src="https://unpkg.com/ipfs/dist/index.js"></script>     <!-- loading the human-readable (not minified) version -->
+<!-- loading the minified version -->
+<script src="https://unpkg.com/ipfs/dist/index.min.js"></script>
+<script src="https://cdn.jsdelivr.net/npm/ipfs/dist/index.min.js"></script>
+
+<!-- loading the human-readable (not minified) version -->
+<script src="https://unpkg.com/ipfs/dist/index.js"></script>
+<script src="https://cdn.jsdelivr.net/npm/ipfs/dist/index.js"></script>
 ```
+Inserting one of the above lines will make an `Ipfs` object available in the global namespace.
 
 ## Usage
 
@@ -309,7 +315,7 @@ Every IPFS instance also exposes the libp2p API at `ipfs.libp2p`. The formal int
 
 ##### [object](https://github.com/ipfs/interface-ipfs-core/tree/master/API/object)
 
-> Consider using the [dag API](#dag) API instead.
+> Consider using the [dag API](#dag) instead.
 
 - [`ipfs.object.new([template][, callback])`](https://github.com/ipfs/interface-ipfs-core/tree/master/API/object#objectnew)
 - [`ipfs.object.put(obj, [options, callback])`](https://github.com/ipfs/interface-ipfs-core/tree/master/API/object#objectput)
@@ -487,7 +493,7 @@ Then, update your IPFS Daemon config to include the multiaddr for this new trans
 
 ### Lint
 
-*Conforming to linting rules is a prerequisite to commit to js-ipfs.*
+**Conforming to linting rules is a prerequisite to commit to js-ipfs.**
 
 ```sh
 > npm run lint
@@ -522,11 +528,11 @@ src                 # Main source code folder
 ### Monitoring
 
 The HTTP API exposed with js-ipfs can also be used for exposing metrics about
-the running js-ipfs node and other nodejs metrics.
+the running js-ipfs node and other Node.js metrics.
 
 To enable it, you need to set the environment variable `IPFS_MONITORING` (any value)
 
-Once environment variable is set and the js-ipfs daemon is running, you can get
+Once the environment variable is set and the js-ipfs daemon is running, you can get
 the metrics (in prometheus format) by making a GET request to the following endpoint:
 
 ```
@@ -541,17 +547,17 @@ What does this image explain?
 
 - IPFS uses `ipfs-repo` which picks `fs` or `indexeddb` as its storage drivers, depending if it is running in Node.js or in the Browser.
 - The exchange protocol, `bitswap`, uses the Block Service which in turn uses the Repo, offering a get and put of blocks to the IPFS implementation.
-- The DAG api (previously Object) comes from the IPLD Resolver, it can support several IPLD Formats (i.e: dag-pb, dag-cbor, etc).
+- The DAG API (previously Object) comes from the IPLD Resolver, it can support several IPLD Formats (i.e: dag-pb, dag-cbor, etc).
 - The Files API uses `ipfs-unixfs-engine` to import and export files to and from IPFS.
-- Swarm, the component that offers a network API, uses libp2p to dial and listen for connections, use the DHT, discovery mechanisms and more. libp2p-ipfs-nodejs is used in case of running in Node.js and libp2p-ipfs-browser is used in the case of the Browser.
+- Swarm, the component that offers a network API, uses libp2p to dial and listen for connections, to use the DHT, for discovery mechanisms, and more. libp2p-ipfs-nodejs is used when running in Node.js and libp2p-ipfs-browser is used when running in the browser.
 
 ## Contribute
 
 IPFS implementation in JavaScript is a work in progress. As such, there's a few things you can do right now to help out:
 
   * Go through the modules below and **check out existing issues**. This would be especially useful for modules in active development. Some knowledge of IPFS may be required, as well as the infrastructure behind it - for instance, you may need to read up on p2p and more complex operations like muxing to be able to help technically.
-  * **Perform code reviews**. More eyes will help a) speed the project along b) ensure quality and c) reduce possible future bugs.
-  * Take a look at go-ipfs and some of the planning repositories or issues: for instance, the libp2p spec [here](https://github.com/ipfs/specs/pull/19). Contributions here that would be most helpful are **top-level comments** about how it should look based on our understanding. Again, the more eyes the better.
+  * **Perform code reviews**. More eyes will help (a) speed the project along, (b) ensure quality, and (c) reduce possible future bugs.
+  * Take a look at go-ipfs and some of the planning repositories or issues: for instance, the [libp2p spec](https://github.com/ipfs/specs/pull/19). Contributions here that would be most helpful are **top-level comments** about how it should look based on our understanding. Again, the more eyes the better.
   * **Add tests**. There can never be enough tests.
   * **Contribute to the [FAQ repository](https://github.com/ipfs/faq/issues)** with any questions you have about IPFS or any of the relevant technology. A good example would be asking, 'What is a merkledag tree?'. If you don't know a term, odds are, someone else doesn't either. Eventually, we should have a good understanding of where we need to improve communications and teaching together to make IPFS and IPN better.
 

examples/browser-webpack/package.json

@@ -13,13 +13,13 @@
     "babel-preset-react": "^6.24.1",
     "babel-preset-stage-0": "^6.24.1",
     "json-loader": "^0.5.4",
+    "react": "^15.5.4",
+    "react-dom": "^15.6.1",
     "react-hot-loader": "^1.3.1",
     "webpack": "^2.5.1",
     "webpack-dev-server": "^2.4.5"
   },
   "dependencies": {
-    "ipfs": "file:../../",
-    "react": "^15.6.1",
-    "react-dom": "^15.6.1"
+    "ipfs": "file:../../"
   }
 }

package.json

@@ -74,7 +74,7 @@
     "expose-loader": "^0.7.3",
     "form-data": "^2.3.1",
     "gulp": "^3.9.1",
-    "interface-ipfs-core": "~0.31.19",
+    "interface-ipfs-core": "~0.32.1",
     "ipfsd-ctl": "~0.23.0",
     "left-pad": "^1.1.3",
     "lodash": "^4.17.4",
@@ -93,16 +93,18 @@
     "async": "^2.5.0",
     "bl": "^1.2.1",
     "boom": "^5.2.0",
-    "debug": "^3.0.1",
+    "byteman": "^1.3.5",
     "cids": "^0.5.1",
+    "debug": "^3.0.1",
     "file-type": "^6.1.0",
     "filesize": "^3.5.10",
     "fsm-event": "^2.1.0",
+    "get-folder-size": "^1.0.0",
     "glob": "^7.1.2",
     "hapi": "^16.5.2",
     "hapi-set-header": "^1.0.2",
     "hoek": "^4.2.0",
-    "ipfs-api": "^14.3.5",
+    "ipfs-api": "^14.3.7",
     "ipfs-bitswap": "~0.17.2",
     "ipfs-block": "~0.6.0",
     "ipfs-block-service": "~0.12.0",
@@ -138,8 +140,11 @@
     "peer-book": "~0.5.1",
     "peer-id": "~0.10.1",
     "peer-info": "~0.11.0",
+    "progress": "^2.0.0",
     "promisify-es6": "^1.0.3",
+    "pull-abortable": "^4.1.1",
     "pull-file": "^1.0.0",
+    "pull-ndjson": "^0.1.1",
     "pull-paramap": "^1.2.2",
     "pull-pushable": "^2.1.1",
     "pull-sort": "^1.0.1",
@@ -216,4 +221,4 @@
     "Łukasz Magiera <magik6k@users.noreply.github.com>",
     "ᴠɪᴄᴛᴏʀ ʙᴊᴇʟᴋʜᴏʟᴍ <victorbjelkholm@gmail.com>"
   ]
-}
+}

src/cli/commands/files/add.js

@@ -8,8 +8,12 @@ const pull = require('pull-stream')
 const paramap = require('pull-paramap')
 const zip = require('pull-zip')
 const toPull = require('stream-to-pull-stream')
+const getFolderSize = require('get-folder-size')
+const byteman = require('byteman')
+const waterfall = require('async/waterfall')
 const utils = require('../../utils')
 const print = require('../../utils').print
+const createProgressBar = require('../../utils').createProgressBar
 
 const WRAPPER = 'wrapper/'
 
@@ -40,14 +44,21 @@ function checkPath (inPath, recursive) {
   return inPath
 }
 
+function getTotalBytes (path, recursive, cb) {
+  if (recursive) {
+    getFolderSize(path, cb)
+  } else {
+    fs.stat(path, (err, stat) => cb(err, stat.size))
+  }
+}
+
 function addPipeline (index, addStream, list, argv) {
   const {
     wrapWithDirectory,
     quiet,
     quieter,
     silent
   } = argv
-
   pull(
     zip(
       pull.values(list),
@@ -102,6 +113,12 @@ module.exports = {
   describe: 'Add a file to IPFS using the UnixFS data format',
 
   builder: {
+    progress: {
+      alias: 'p',
+      type: 'boolean',
+      default: true,
+      describe: 'Stream progress data'
+    },
     recursive: {
       alias: 'r',
       type: 'boolean',
@@ -185,34 +202,42 @@ module.exports = {
     }
     const ipfs = argv.ipfs
 
-    // TODO: revist when interface-ipfs-core exposes pull-streams
-    let createAddStream = (cb) => {
-      ipfs.files.createAddStream(options, (err, stream) => {
-        cb(err, err ? null : toPull.transform(stream))
-      })
-    }
-
-    if (typeof ipfs.files.createAddPullStream === 'function') {
-      createAddStream = (cb) => {
-        cb(null, ipfs.files.createAddPullStream(options))
-      }
-    }
+    let list = []
+    waterfall([
+      (next) => glob(path.join(inPath, '/**/*'), next),
+      (globResult, next) => {
+        list = globResult.length === 0 ? [inPath] : globResult
+
+        getTotalBytes(inPath, argv.recursive, next)
+      },
+      (totalBytes, next) => {
+        if (argv.progress) {
+          const bar = createProgressBar(totalBytes)
+          options.progress = function (byteLength) {
+            bar.update(byteLength / totalBytes, {progress: byteman(byteLength, 2, 'MB')})
+          }
+        }
 
-    createAddStream((err, addStream) => {
-      if (err) {
-        throw err
-      }
+        // TODO: revist when interface-ipfs-core exposes pull-streams
 
-      glob(path.join(inPath, '/**/*'), (err, list) => {
-        if (err) {
-          throw err
+        let createAddStream = (cb) => {
+          ipfs.files.createAddStream(options, (err, stream) => {
+            cb(err, err ? null : toPull.transform(stream))
+          })
         }
-        if (list.length === 0) {
-          list = [inPath]
+
+        if (typeof ipfs.files.createAddPullStream === 'function') {
+          createAddStream = (cb) => {
+            cb(null, ipfs.files.createAddPullStream(options))
+          }
         }
 
-        addPipeline(index, addStream, list, argv)
-      })
+        createAddStream(next)
+      }
+    ], (err, addStream) => {
+      if (err) throw err
+
+      addPipeline(index, addStream, list, argv)
     })
   }
 }

src/cli/utils.js

@@ -9,6 +9,8 @@ const path = require('path')
 const debug = require('debug')
 const log = debug('cli')
 log.error = debug('cli:error')
+const Progress = require('progress')
+const byteman = require('byteman')
 
 exports = module.exports
 
@@ -85,3 +87,16 @@ exports.print = (msg, newline) => {
     process.stdout.write(msg)
   }
 }
+
+exports.createProgressBar = (totalBytes) => {
+  const total = byteman(totalBytes, 2, 'MB')
+  const barFormat = `:progress / ${total} [:bar] :percent :etas`
+
+  // 16 MB / 34 MB [===========             ] 48% 5.8s //
+  return new Progress(barFormat, {
+    incomplete: ' ',
+    clear: true,
+    stream: process.stdout,
+    total: totalBytes
+  })
+}

src/core/components/files.js

@@ -20,6 +20,14 @@ module.exports = function files (self) {
       shardSplitThreshold: self._options.EXPERIMENTAL.sharding ? 1000 : Infinity
     }, options)
 
+    let total = 0
+    let prog = opts.progress || (() => {})
+    const progress = (bytes) => {
+      total += bytes
+      prog(total)
+    }
+
+    opts.progress = progress
     return pull(
       pull.map(normalizeContent),
       pull.flatten(),
@@ -54,7 +62,7 @@ module.exports = function files (self) {
     add: promisify((data, options, callback) => {
       if (typeof options === 'function') {
         callback = options
-        options = undefined
+        options = {}
       } else if (!callback || typeof callback !== 'function') {
         callback = noop
       }
@@ -66,9 +74,8 @@ module.exports = function files (self) {
       }
 
       pull(
-        pull.values(normalizeContent(data)),
-        importer(self._ipldResolver, options),
-        pull.asyncMap(prepareFile.bind(null, self, options)),
+        pull.values([data]),
+        createAddPullStream(options),
         sort((a, b) => {
           if (a.path < b.path) return 1
           if (a.path > b.path) return -1
@@ -80,7 +87,7 @@ module.exports = function files (self) {
 
     cat: promisify((ipfsPath, callback) => {
       if (typeof ipfsPath === 'function') {
-        return callback(new Error('You must supply a ipfsPath'))
+        return callback(new Error('You must supply an ipfsPath'))
       }
 
       pull(
@@ -89,6 +96,7 @@ module.exports = function files (self) {
           if (err) {
             return callback(err)
           }
+          if (!files || !files.length) return callback(new Error('No such file'))
           callback(null, toStream.source(files[files.length - 1].content))
         })
       )

src/core/components/init-assets.js

@@ -6,12 +6,12 @@ const glob = require('glob')
 const importer = require('ipfs-unixfs-engine').importer
 const pull = require('pull-stream')
 const file = require('pull-file')
-// const mh = require('multihashes')
+const CID = require('cids')
 
 // Add the default assets to the repo.
 module.exports = function addDefaultAssets (self, log, callback) {
   const initDocsPath = path.join(__dirname, '../../init-files/init-docs')
-  const index = __dirname.lastIndexOf('/')
+  const index = initDocsPath.lastIndexOf('/')
 
   pull(
     pull.values([initDocsPath]),
@@ -20,7 +20,7 @@ module.exports = function addDefaultAssets (self, log, callback) {
     }),
     pull.flatten(),
     pull.map((element) => {
-      const addPath = element.substring(index + 1, element.length)
+      const addPath = element.substring(index + 1)
       if (fs.statSync(element).isDirectory()) {
         return
       }
@@ -34,14 +34,10 @@ module.exports = function addDefaultAssets (self, log, callback) {
     pull.filter(Boolean),
     importer(self._ipldResolver),
     pull.through((el) => {
-      if (el.path === 'files/init-docs/docs') {
-        log('to get started, enter:')
-        log()
-        log(`\t jsipfs files cat /ipfs/QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB`)
-        // TODO when we support pathing in unixfs-engine
-        // const hash = mh.toB58String(el.multihash)
-        // log(`\t jsipfs files cat /ipfs/${hash}/readme`)
-        log()
+      if (el.path === 'init-docs') {
+        const cid = new CID(el.multihash)
+        log('to get started, enter:\n')
+        log(`\t jsipfs files cat /ipfs/${cid.toBaseEncodedString()}/readme\n`)
       }
     }),
     pull.collect((err) => {

src/core/runtime/libp2p-browser.js

@@ -26,6 +26,14 @@ class Node extends libp2p {
       modules.discovery.push(r)
     }
 
+    if (options.modules && options.modules.transport) {
+      options.modules.transport.forEach((t) => modules.transport.push(t))
+    }
+
+    if (options.modules && options.modules.discovery) {
+      options.modules.discovery.forEach((d) => modules.discovery.push(d))
+    }
+
     super(modules, peerInfo, peerBook, options)
   }
 }