Skip to content

Commit 5872e31

Browse files
dignifiedquiredaviddias
dignifiedquire
authored and
daviddias
committed
feat: migrate to datastore from pull-blob-store
1 parent 64f3062 commit 5872e31

File tree

49 files changed

+1048
-813
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+1048
-813
lines changed

‎README.md

Lines changed: 62 additions & 110 deletions
Original file line numberDiff line numberDiff line change
@@ -19,24 +19,13 @@ This is the implementation of the [IPFS repo spec](https://github.com/ipfs/specs
1919
## Table of Contents
2020

2121
- [Background](#background)
22-
- [Good to know (historical context)](#good-to-know-historical-context)
2322
- [Install](#install)
2423
- [npm](#npm)
2524
- [Use in Node.js](#use-in-nodejs)
2625
- [Use in a browser with browserify, webpack or any other bundler](#use-in-a-browser-with-browserify-webpack-or-any-other-bundler)
2726
- [Use in a browser Using a script tag](#use-in-a-browser-using-a-script-tag)
2827
- [Usage](#usage)
2928
- [API](#api)
30-
- [var repo = new IPFSRepo(path, opts)](#var-repo--new-ipfsrepopath-opts)
31-
- [repo.exists(cb)](#repoexistscb)
32-
- [repo.version.get(cb(err, version))](#repoversiongetcberr-version)
33-
- [repo.version.set(version, cb(err))](#repoversionsetversion-cberr)
34-
- [repo.config.get(cb(err, config))](#repoconfiggetcberr-config)
35-
- [repo.config.set(config, cb(err))](#repoconfigsetconfig-cberr)
36-
- [repo.keys](#repokeys)
37-
- [repo.blockstore.putStream()](#)
38-
- [repo.blockstore.getStream(key, extension)](#)
39-
- [repo.datastore](#repodatastore)
4029
- [Contribute](#contribute)
4130
- [License](#license)
4231

@@ -45,62 +34,60 @@ This is the implementation of the [IPFS repo spec](https://github.com/ipfs/specs
4534
Here is the architectural reasoning for this repo:
4635

4736
```bash
48-
┌─────────────────────────────────┐
49-
│ interface defined by Repo Spec │
50-
├─────────────────────────────────┤
51-
│ │ ┌──────────────────────┐
52-
│ │ │ interface-pull-blob-store │
53-
│ IPFS REPO │─────────────────────────────────▶│ interface │
54-
│ │ ├──────────────────────┤
55-
│ │ │ locks │
56-
└─────────────────────────────────┘ └──────────────────────┘
57-
│
58-
┌──────────┴────┬───────────────┬───────────────┬───────────────┬───────────────┐
59-
▼ ▼ ▼ ▼ ▼ ▼
60-
┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐
61-
│ abstract │ │ abstract │ │ abstract │ │ abstract │ │ abstract │ │ abstract │
62-
│ -blob │ │ -blob │ │ -blob │ │ -blob │ │ -blob │ │ -blob │
63-
│ -store │ │ -store │ │ -store │ │ -store │ │ -store │ │ -store │
64-
│ interface │ │ interface │ │ interface │ │ interface │ │ interface │ │ interface │
65-
├───────────┤ ├───────────┤ ├───────────┤ ├───────────┤ ├───────────┤ ├───────────┤
66-
│ │ │ │ │ │ │ │ │ │ │ │
67-
│ keys │ │ config │ │ blockstore │ │ datastore │ │ logs │ │ version │
68-
│ │ │ │ │ │ │ │ │ │ │ │
69-
└───────────┘ └───────────┘ └───────────┘ └───────────┘ └───────────┘ └───────────┘
37+
┌────────────────────────────────────────┐
38+
│ IPFSRepo │
39+
└────────────────────────────────────────┘
40+
┌─────────────────┐
41+
│ / │
42+
├─────────────────┤
43+
│ Datastore │
44+
└─────────────────┘
45+
│
46+
┌───────────┴───────────┐
47+
│ │
48+
┌─────────────────┐ ┌─────────────────┐
49+
│ /blocks │ │ /datastore │
50+
├─────────────────┤ ├─────────────────┤
51+
│ Datastore │ │ LevelDatastore │
52+
└─────────────────┘ └─────────────────┘
53+
54+
┌────────────────────────────────────────┐ ┌────────────────────────────────────────┐
55+
│ IPFSRepo - Default Node.js │ │ IPFSRepo - Default Browser │
56+
└────────────────────────────────────────┘ └────────────────────────────────────────┘
57+
┌─────────────────┐ ┌─────────────────┐
58+
│ / │ │ / │
59+
├─────────────────┤ ├─────────────────┤
60+
│ FsDatastore │ │LevelJSDatastore │
61+
└─────────────────┘ └─────────────────┘
62+
│ │
63+
┌───────────┴───────────┐ ┌───────────┴───────────┐
64+
│ │ │ │
65+
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
66+
│ /blocks │ │ /datastore │ │ /blocks │ │ /datastore │
67+
├─────────────────┤ ├─────────────────┤ ├─────────────────┤ ├─────────────────┤
68+
│ FlatfsDatastore │ │LevelDBDatastore │ │LevelJSDatastore │ │LevelJSDatastore │
69+
└─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘
7070
```
7171

72-
This provides a well defined interface for creating and interacting with an IPFS
73-
Repo backed by a group of abstract backends for keys, configuration, logs, and
74-
more. Each of the individual repos has an interface defined by
75-
[interface-pull-blob-store](https://github.com/ipfs/interface-pull-blob-store): this
76-
enables us to make IPFS Repo portable (running on Node.js vs the browser) and
77-
accept different types of storage mechanisms for each repo (fs, levelDB, etc).
78-
79-
### Good to know (historical context)
80-
81-
- The datastore folder holds the legacy version of datastore, still built in levelDB, there is a current endeavour of pushing it to fs completely.
82-
- The blocks folder is the current version of datastore.
83-
- The keys repo doesn't exist yet, as the private key is simply stored inside config
72+
This provides a well defined interface for creating and interacting with an IPFS repo.
8473

8574
## Install
8675

8776
### npm
8877

8978
```sh
90-
> npm i ipfs-repo
79+
> npm install ipfs-repo
9180
```
9281

9382
### Use in Node.js
9483

95-
```JavaScript
84+
```js
9685
var IPFSRepo = require('ipfs-repo')
9786
```
9887

9988
### Use in a browser with browserify, webpack or any other bundler
10089

101-
The code published to npm that gets loaded on require is in fact a ES5 transpiled version with the right shims added. This means that you can require it and use with your favourite bundler without having to adjust asset management process.
102-
103-
```JavaScript
90+
```js
10491
var IPFSRepo = require('ipfs-repo')
10592
```
10693

@@ -119,77 +106,42 @@ Loading this module through a script tag will make the `IpfsRepo` obj available
119106
Example:
120107

121108
```js
122-
var inMemoryBS = require('interface-pull-blob-store')
123-
// inMemoryBS is an "in memory" blob store, you can find others at:
124-
// https://github.com/ipfs/interface-pull-blob-store#modules-that-use-this
109+
const Repo = require('ipfs-repo')
110+
const repo = new Repo('/tmp/ipfs-repo')
125111

126-
var IPFSRepo = require('ipfs-repo')
127-
var repo = new IPFSRepo('/Users/someone/.ipfs', {
128-
stores: inMemoryBS
129-
})
130-
```
112+
repo.init({ cool: 'config' }, (err) => {
113+
if (err) {
114+
throw err
115+
}
131116

132-
## API
117+
repo.open((err) => {
118+
if (err) {
119+
throw err
120+
}
133121

134-
```js
135-
var IPFSRepo = require('ipfs-repo')
122+
console.log('repo is ready')
123+
})
124+
})
136125
```
137126

138-
### var repo = new IPFSRepo(path, opts)
139-
140-
Creates a **reference** to an IPFS repository at the path `path`. This does
141-
*not* create the repo, but is an object that refers to the repo at such a path.
142-
143-
Valid keys for `opts` include:
127+
This now has created the following structure, either on disk or as an in memory representation:
144128

145-
- `stores`: either an
146-
[interface-pull-blob-store](https://github.com/ipfs/interface-pull-blob-store), or a
147-
map of the form
148-
149-
```js
150-
{
151-
keys: someBlobStore,
152-
config: someBlobStore,
153-
datastore: someBlobStore,
154-
logs: someBlobStore,
155-
locks: someBlobStore,
156-
version: someBlobStore
157-
}
129+
```
130+
├── blocks
131+
│   ├── SHARDING
132+
│ └── _README
133+
├── config
134+
├── datastore
135+
└── version
158136
```
159137

160-
If you use the former form, all of the sub-blob-stores will use the same store.
161-
162-
### repo.exists(cb)
163-
164-
Check if the repo you are going to access already exists. Calls the callback
165-
`cb(err, exists)`, where `exists` is true or false.
166-
167-
### repo.version.get(cb(err, version))
168-
### repo.version.set(version, cb(err))
169-
170-
Read/write the version number of the repository. The version number is the repo version number.
171-
172-
### repo.config.get(cb(err, config))
173-
### repo.config.set(config, cb(err))
174-
175-
Read/write the configuration object of the repository.
176-
177-
### repo.keys
178-
179-
Read/write keys inside the repo. This feature will be expanded once
180-
[IPRS](https://github.com/ipfs/specs/tree/master/records) and
181-
[KeyChain](https://github.com/ipfs/specs/tree/master/keychain) are finalized and implemented on go-ipfs.
182-
183-
### repo.blockstore.putStream()
184-
### repo.datastore.getStream(key, extension)
185-
### repo.datastore.has(key, extension, cb)
186-
### repo.datastore.delete(key, extension, cb)
138+
## API
187139

188-
Read and write buffers to/from the repo's block store.
140+
See https://ipfs.github.io/js-ipfs-repo
189141

190-
### repo.datastore
142+
## Notes
191143

192-
**WIP**
144+
- [Explanation of how repo is structured](https://github.com/ipfs/js-ipfs-repo/pull/111#issuecomment-279948247)
193145

194146
## Contribute
195147

‎example.js

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
'use strict'
2+
3+
const Repo = require('ipfs-repo')
4+
const repo = new Repo('/Users/awesome/.jsipfs')
5+
6+
repo.init({my: 'config'}, (err) => {
7+
if (err) {
8+
throw err
9+
}
10+
11+
repo.open((err) => {
12+
if (err) {
13+
throw err
14+
}
15+
16+
console.log('repo is ready')
17+
})
18+
})

‎package.json

Lines changed: 32 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -3,55 +3,66 @@
33
"version": "0.11.3",
44
"description": "IPFS Repo implementation",
55
"main": "src/index.js",
6+
"browser": {
7+
"./src/lock.js": "./src/lock-memory.js",
8+
"./src/default-options.js": "./src/default-options-browser.js"
9+
},
610
"scripts": {
711
"test": "aegir-test",
812
"test:node": "aegir-test node",
913
"test:browser": "aegir-test browser",
1014
"build": "aegir-build",
1115
"lint": "aegir-lint",
12-
"release": "aegir-release",
13-
"release-minor": "aegir-release --type minor",
14-
"release-major": "aegir-release --type major",
16+
"release": "aegir-release --docs",
17+
"release-minor": "aegir-release --type minor --docs",
18+
"release-major": "aegir-release --type major --docs",
1519
"coverage": "aegir-coverage",
16-
"coverage-publish": "aegir-coverage publish"
20+
"coverage-publish": "aegir-coverage publish",
21+
"docs": "aegir-docs"
1722
},
1823
"repository": {
1924
"type": "git",
2025
"url": "https://github.com/ipfs/js-ipfs-repo.git"
2126
},
2227
"keywords": [
23-
"IPFS"
28+
"IPFS",
29+
"libp2p",
30+
"datastore"
2431
],
2532
"pre-commit": [
2633
"lint",
2734
"test"
2835
],
2936
"homepage": "https://github.com/ipfs/js-ipfs-repo",
3037
"engines": {
31-
"node": ">=4.0.0"
38+
"node": ">=4.0.0",
39+
"npm": ">=3.0.0"
3240
},
3341
"devDependencies": {
34-
"aegir": "^10.0.0",
35-
"buffer-loader": "^0.0.1",
42+
"aegir": "^11.0.0",
3643
"chai": "^3.5.0",
37-
"fs-pull-blob-store": "~0.4.1",
38-
"idb-pull-blob-store": "~0.5.1",
39-
"interface-pull-blob-store": "~0.6.0",
44+
"dirty-chai": "^1.2.2",
4045
"lodash": "^4.17.4",
41-
"multihashes": "~0.4.0",
46+
"memdown": "^1.2.4",
47+
"multihashes": "~0.4.4",
4248
"ncp": "^2.0.0",
4349
"pre-commit": "^1.2.2",
44-
"rimraf": "^2.5.4"
50+
"rimraf": "^2.6.1"
4551
},
4652
"dependencies": {
47-
"async": "^2.1.4",
53+
"async": "^2.1.5",
4854
"base32.js": "^0.1.0",
49-
"ipfs-block": "~0.5.5",
50-
"lock": "^0.1.3",
51-
"multihashes": "~0.3.3",
52-
"pull-defer": "^0.2.2",
53-
"pull-stream": "^3.5.0",
54-
"pull-write": "^1.1.1",
55+
"cids": "^0.4.2",
56+
"datastore-core": "^0.1.0",
57+
"datastore-fs": "^0.1.1",
58+
"datastore-level": "^0.1.0",
59+
"debug": "^2.6.3",
60+
"interface-datastore": "^0.1.1",
61+
"ipfs-block": "~0.6.0",
62+
"level-js": "^2.2.4",
63+
"leveldown": "^1.6.0",
64+
"lock-me": "^1.0.2",
65+
"multiaddr": "^2.2.2",
5566
"safe-buffer": "^5.0.1"
5667
},
5768
"license": "MIT",
@@ -69,4 +80,4 @@
6980
"nginnever <ginneversource@gmail.com>",
7081
"npmcdn-to-unpkg-bot <npmcdn-to-unpkg-bot@users.noreply.github.com>"
7182
]
72-
}
83+
}

0 commit comments

Comments
 (0)