Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexing design for w3s #29

Open
wants to merge 31 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
6528294
Indexing design for w3s
gammazero May 29, 2024
4eab388
embed diagram
gammazero Jun 2, 2024
f6870b4
rename query diagram
gammazero Jun 2, 2024
24197d3
Add diagram tldr file
gammazero Jun 3, 2024
f878d55
requested changes from review
gammazero Jun 4, 2024
4f66930
Update rfc/indexing-design.md
gammazero Jun 4, 2024
64015b6
Update rfc/indexing-design.md
gammazero Jun 4, 2024
f470b46
Update rfc/indexing-design.md
gammazero Jun 4, 2024
ca152c6
Update rfc/indexing-design.md
gammazero Jun 4, 2024
b898f36
review updates
gammazero Jun 4, 2024
4d4049c
Add discussion summary
gammazero Jun 7, 2024
592811c
Update rfc/indexing-design.md
gammazero Jun 11, 2024
88e1689
Update rfc/indexing-design.md
gammazero Jun 11, 2024
0eb08ec
Update rfc/indexing-design.md
gammazero Jun 11, 2024
32ca892
Update rfc/indexing-design.md
gammazero Jun 11, 2024
9f10b69
Update rfc/indexing-design.md
gammazero Jun 11, 2024
3e7758a
Update rfc/indexing-design.md
gammazero Jun 11, 2024
55214d9
Update rfc/indexing-design.md
gammazero Jun 11, 2024
038225a
Update rfc/indexing-design.md
gammazero Jun 11, 2024
95e01bb
Update rfc/indexing-design.md
gammazero Jun 11, 2024
9dd095c
detail operation description
gammazero Jun 12, 2024
1e4c3db
revierw changes
gammazero Jun 12, 2024
05d30ae
update query description
gammazero Jun 12, 2024
e215eba
Combine w3up-index and ipni-cache
gammazero Jun 17, 2024
f532427
fix diagram error
gammazero Jun 17, 2024
cc99f9e
update diagrams
gammazero Jun 17, 2024
703643a
update API description
gammazero Jun 17, 2024
c999858
Update rfc/indexing-design.md
gammazero Jun 17, 2024
0c5b91d
Update rfc/indexing-design.md
gammazero Jun 17, 2024
9182f9d
Update rfc/indexing-design.md
gammazero Jun 17, 2024
ba31778
Update rfc/indexing-design.md
gammazero Jun 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update rfc/indexing-design.md
Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
gammazero and Alan Shaw authored Jun 17, 2024
commit 9182f9df13d9c7a4a78b83d58a6953b8064480db
2 changes: 1 addition & 1 deletion rfc/indexing-design.md
Original file line number Diff line number Diff line change
@@ -20,7 +20,7 @@ A user stores content in shards of opaque blobs. They are typically a series of

The agent uploading user content builds an IPLD DAG and encodes it into one or more CAR files (blobs). It also derives a ([sharded-dag-index](https://github.com/w3s-project/specs/blob/main/w3-index.md#sharded-dag-index-example)) that describe the blobs in terms of (multihash, offset, length) tuples for all the blocks. This enables the user to make choices about which blocks should be make discoverable and retrievable over the network. For example, they may want only the DAG root to be indexed.

The sharded-dag-index is stored in the user's space along with the CAR file(s) containing the blob data. This is done in the [w3 add-blob operation](https://github.com/w3s-project/specs/blob/main/w3-blob.md#add-blob). The w3s service creates [location commitments](https://github.com/w3s-project/specs/blob/main/w3-blob.md#location-commitment) for the blob shards and for the sharded-dag-index.
The sharded-dag-index is stored in the user's space along with the CAR file(s) containing the blob data. This is done in the [w3 add-blob operation](https://github.com/w3s-project/specs/blob/main/w3-blob.md#add-blob). The w3s service creates a [location commitment](https://github.com/w3s-project/specs/blob/main/w3-blob.md#location-commitment) for each blob (the sharded-dag-index included).

After the add-blob process completes, and the user receives an [accept-blob receipt](https://github.com/w3s-project/specs/blob/main/w3-blob.md#accept-blob-receipt), the user may then choose to make their data publicly queryable by publishing it to W3UP's indexing system. The user optionally invokes an [index-add](https://github.com/w3s-project/specs/blob/main/w3-index.md#index-add) capability to publish the sharded-dag-index multihashes into W3UP's indexing system and eventually to [IPNI](https://github.com/ipni/specs/blob/main/IPNI.md) so that they can be used to look up location commitments and retrieve sharded-dag-index information and blob data. See [W3 Index](https://github.com/w3s-project/specs/blob/main/w3-index.md#w3-index) for more.