Skip to content
This repository was archived by the owner on Mar 28, 2023. It is now read-only.

Represent storetheindex v0 finder protocol in schema #7

Closed
wants to merge 1 commit into from
Closed

Represent storetheindex v0 finder protocol in schema #7

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 22 additions & 3 deletions ipld/routing.ipldsch
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,39 @@ type Envelope union {
| GetP2PProvideResponse "get-p2p-provide-response"
} representation keyed
Comment on lines 4 to 5
Copy link
Contributor

@aschmahmann aschmahmann Dec 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@warpfork continuing from #7 (comment) (GitHub's lack of threads are brutal and I'd like to isolate this topic a bit).

hen there's not much useful that can be done other than describing the data as Any, as far as I can figure.

Can Any currently be parsed by codegen?

but that's an accurate representation of what would be mechanically needed, even if there were syntactic facades over it; I'm not actually sure we want such a thing to be easy to describe: it's sort of a negative-feedback form of mechanical sympathy

Is this necessarily true? Say I have a keyed union for the keys "foo" and "bar", but my parser sees a "baz". Why would having a keyed union like the below be so rough to process, as soon as we notice the key isn't "foo" or "bar" we just treat it like an Any which offhand doesn't seem like it'd be more work than processing a regular field that happens to be an Any.

type OpenKeyedUnion union {
     | String "foo"
     | String "baz"
     | Any
} representation keyed

Also, some union types are more efficient than others for parsing so adding one more that eases consumer pain here doesn't seem too bad, but I could be missing things so lmk 😄.


type Multihash Bytes

type GetP2PProvideRequest struct {
key Bytes
Multihashes [Multihash]
}

type GetP2PProvideResponse struct {
providers [Provider]
MultihashResults [MultihashResult]
}

type MultihashResult struct {
Multihash Multihash
ProviderResults [ProviderResult]
}

type ProviderResult struct {
Metadata Metadata
Provider Provider
}
Comment on lines +22 to +25
Copy link
Contributor

@aschmahmann aschmahmann Dec 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@willscott I wanted to clarify + document some things I understood from our conversation today.

  1. This schema is representing how the indexers are currently storing their data internally
  2. The system is trying to be agnostic about the types of records it supports and as a result has defined the things it cares about (e.g. who published who has the data rather than who has the data) while relegating everything else to the metadata field
  3. The metadata field then becomes the primary point of interest for us. Could you or @gammazero give a (even if rough) schema for what the current metadata things look like?

Copy link
Contributor

@aschmahmann aschmahmann Dec 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, following up from discussions today the metadata schema seems like it'll be important within the indexer as well as once you separate the "who published these records" from "who has these records" you'll want some way to evaluate if the records are good for your own reputation system purposes which means understanding some (even if not all) of the record types.

Copy link

@gammazero gammazero Jan 4, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aschmahmann

  1. The schema is representing an answer to a client asking, "who has the data". Who published this record is not part of this, and that is a separate concern from delegated routing and responses to clients.
  2. Only the metadata is trying to be agnostic about what records it supports since that is consumed mostly by the provider (who has the data) to determine where/how to retrieve that data.
  3. The metadata contains a "protocol" field which determines the protocol used to retrieve data (graphsync, bitswap). This protocol field is used by the client that wants the data. The remainder of the metadata is a payload that is used only by the provider. It is likely something like a deal ID, but could be anything (e.g. record key for their internal database) that tells the provider how to find and retrieve the data.


# XXX: represent as union of known options
type Metadata struct {
ProtocolID Int
Data Bytes
}

type Provider union {
| Peer "peer"
} representation keyed

type Peer struct {
Multiaddress [Bytes]
ID Bytes
Addrs [Bytes]
}

# XXX: can we have a union representation that tries to parse in order of rules?
Expand Down