any plan to support full text keyword search?

[redmouth]

Currently it only support searching by file content hash as keyword

[ec1oud]

In order for it to support any kind of search, there has to be an index, right? In a database it would typically be a b-tree. So it seems to me there needs to be thought put into how to store such fast-changing things as indices on ipfs. I've been thinking that I'd like to be able to use ipfs blocks as tree nodes, for many distributed-data-structure purposes, but it's not really lightweight enough to store small nodes. When you build a tree in memory, pointers are 64-bit, so you can afford to use lots of them to make very extensible data structures. But on IPFS, a pointer is a hash.

So far the only answer we have about data structures is to use JSON-LD, right? And the ideal size for a chunk of JSON would be somewhere near the block size. So if you have to build data structures on that kind of substrate, it makes you rethink the data structure, into something very different than what you'd normally have with byte-addressable storage. It's all about packing the right amount of data into each node, not too coarse, not too fine.

Or can we come up with a way to contextualize and thereby shorten a hash somehow? Within the context of a known tree root, somehow pointers to tree nodes would be shorter hashes, so that pointers from one node to another would be short (64 bits or less). (Now someone will say just publish a directory structure instead, I suppose. Then instead of pointers you have names: you navigate to the leaf content by going into the right subdirectories.)

But regardless, doesn't it mean every update to such a data structure would mean updating the IPNS record which points to the whole thing? because the latest version of that data structure, which has been updated, would have a different overall hash, even if you just made one little change somewhere. Just like amending a git commit by changing one byte will make the commit's hash totally different.

Anyway, if we could have such things, then a search engine would consist of a crawler which fetches content and updates a big index, I suppose. It could publish those updates periodically, so maybe the IPNS record only gets updated once a minute or so, instead of with every change. But I don't know much about the optimal data structure for a full-text database, actually; there are lot of them for inspiration. But I figure you should be able to do the search on your own client machine by pursuing links from the root. Maybe it could be a trie, for example. But if a leaf node gives you all links relevant to one word, it would be too many. You'd need to find which links are in common, given multiple words. But it's still not necessarily specific enough. Google would refine it further by the page ranking algorithms and by practically reading our minds (previous searches influence future ones by giving them a clue about the domain we're searching).

It's a very important world problem to decentralize this somehow so that we can stop relying on big central indices like Google. And yet if the index-building were decentralized in a way that anybody could voluntarily contribute to it, it would just turn into a big spam repository. The rules that are followed, to extract relevant keywords from the actual published content and put them into the index (and ignore content which is just spam), need to be collectively enforced, before the search engine can be decentralized. And that sounds like the same class of problems that blockchains are trying to solve: distributed consensus. But blockchains are so expensive in every way: lots of redundant communication and computation going on, taking up a significant fraction of the whole world's electricity supply just to give us the bitcoin blockchain. Lots of storage space taken up on your machine if you even run a bitcoin client, because you have to store the whole blockchain. Ethereum would give us the ability to compute something in a distributed way, but it has the same problem with not scaling to such large problems. There's no way it would ever scale to storing an index of the whole Internet. The index has to be stored in a distributed way, and yet we need to have distributed consensus too. Which means only a small fraction of the total client base can be involved in re-computing and verifying any one part of the index. You want some redundancy, but much less than 100%. And yet prevent the distributed consensus from being corrupted by colluding actors... It seems like a very hard problem to me so far.

[udatny]

this is indeed a interesting topic

@ec1oud i ask myself how much you did already towards an index.

let me summarize your points (and add my 2cents), thanks for giving me a base to build up on:

ideal size for a chunk of JSON would be somewhere near the block size

: yes. this is mainly related to the key structure: max length and type (hex, alphanumeric, numeric, base58, etc) - the entropy. based on this information you may model the shape of the tree. ipld. sure.

doesn't it mean every update to such a data structure would mean updating the IPNS record which points to the whole thing?

ipns is not what comes to mind at the first place. pubsub or just publishing the root node is good enough... and yes. the update leads to a new root node.

...then a search engine would consist of a crawler which fetches content and updates a big index...

this is not the indexing itself anymore. this is some specific use. as an index of backlinks, which me and my buddy were thinking of several times....

trie

yes.

But if a leaf node gives you all links relevant to one word, it would be too many. You'd need to find which links are in common, given multiple words.

just attach another index ;) the index i imagine, is just a simple key / value (1:1) index. if you use a cid as the value, it would point to the value content - just as another link. its up to the indexer what the cid points to and how to handle the value node in such a case.

ok, and the last passage of your thoughts.... i definetely agree. that are the real issues. but i think its time to come up with some ideas / solutions how to adress them.

• how to decentralize / validate / verify index updates
• how not to loose tree nodes

lets go...

[icoco]

any API / solution for this purpose ?