feat: partial DAG provides with Reprovider.Strategy=mfs|pinned+mfs #10754

hsanjuan · 2025-03-13T17:44:31Z

Fixes Include local MFS entity roots in Reprovider.Strategy=roots|pinned+mfs|mfs #10386
Overseeds Add MFS Support to Reprovider Strategy #10704
Part of Kubo users can more directly control which data is advertised — IPFS/2025 ipshipyard/roadmaps#8
Requires the boxo PR at feat: NewDAGProvider to walk partial DAGs in offline mode boxo#905

This augments the current provider KeyChanFuncs with the CIDs coming from the MFS DAG. MFS CIDs are listed using the offline fetcher, which has been updated with SkipNotFound so that no errors happen during traversal when nodes are missing.

Strategies have been updated:

For "roots" and "all" strategy we announce the MFS root only, after the pins.
For the "pinned+mfs" strategy, we announce the MFS Cids after the pin cids.
A new "mfs" strategy is added, only for MFS cids.

The code around strategy selection has been slightly change for the better (I hope).

also:

Fixes #10386. Overseeds #10704. This augments the current provider KeyChanFuncs with the CIDs coming from the MFS DAG. MFS CIDs are listed using the offline fetcher, which has been updated with SkipNotFound so that no errors happen during traversal when nodes are missing. Strategies have been updated: - For "roots" and "all" strategy we announce the MFS root only, after the pins. - For the "pinned" strategy, we announce the MFS Cids after the pin cids. - A new "mfs" strategy is added, only for MFS cids. The code around strategy selection has been slightly change for the better (I hope).

…fort

core/node/core.go

core/node/provider.go

hsanjuan · 2025-03-14T14:02:40Z

core/node/provider.go

+		case "pinned":
+			return provider.NewPrioritizedProvider(
+				provider.NewBufferedProvider(provider.NewPinnedProvider(false, in.Pinner, in.IPLDFetcher)),
+				mfsProvider(in.MFSRoot, in.OfflineUnixFSFetcher),


The downside here is that there is no strategy that only provide "recursive-pinned" content, and no mfs.

If we don't provide mfs here, there will be no policy that provides only "recursive-pinned+mfs" content... but it could be added as pinned+mfs or so...

MFS is implicitly pinned (protected from GC) so I think it is fine to keep it under pinned.
But things get hairy with roots (we dont have ability to only announce dir and file roots as we walk MFS, right?).

Maybe keeping pinned and roots as-is, and adding separate pinned+mfs for now is the right way of handling this? (we dont change behavior for existing users, and new users can choose pinned+mfs).

Is it a problem to only announce the MFS-root? (consider it a big file)... Or you mean this could cause misunderstandings?

in the meantime I restored "pinned" and "roots" and added "pinned+mfs"

I'm thinking:

if you pin something then that pin is usually recursive

pinned announces recursive pins by walking the entire DAG behind each recursively pinned CID

roots announces only root cids of pins (no DAG walk at all)

pinned+mfs should do the same as pinned – walk the MFS DAG and announce MFS recursively (only local blocks).

roots

easy: could also announce only MFS root (no need to create roots+mfs just for one CID, fine to announce MFS root with other pin roots)

harder: make MFS walk smart enough to only yield root CIDs of files and directories (imo this comes with extra overhead, and no longer can be roots and requires separate profile roots+mfs)

I think its fine to do "easy" roots for now – announce the MFS root for now (without walking dag and discovering sub-entity roots).
We can add roots+mfs if users ask for it.

could also announce only MFS root (no need to create roots+mfs just for one CID, fine to announce MFS root with other pin roots)

This is how it was.

core/node/provider.go

…fort

lidel · 2025-03-29T00:38:41Z

Looking good. Updated docs + ipfs/boxo#847 and started local test of pinned+mfs over the weekend. Likely merge next week.

lidel · 2025-03-29T12:37:03Z

Cosmetic: providing MFS which does not have all blocks locally, prints this to logs:

2025-03-29T02:03:06.113Z	ERROR	provider.batched	provider/dagprovider.go:41	dagprovider dag traversal error: skip

the specific "skip" event should not be logged as error (ok to log it as DEBUG)

hsanjuan · 2025-03-31T09:56:31Z

Cosmetic: providing MFS which does not have all blocks locally, prints this to logs:
2025-03-29T02:03:06.113Z	ERROR	provider.batched	provider/dagprovider.go:41	dagprovider dag traversal error: skip
the specific "skip" event should not be logged as error (ok to log it as DEBUG)

Just to check, you removed the root MFS cid? It is the only way I can reproduce this.

hsanjuan · 2025-03-31T09:58:33Z

And if the root block is not there, maybe it should be logged as error? It means we have initiated a traversal from a block that is not there which is probably not intended

lidel · 2025-03-31T13:13:35Z

@hsanjuan no, I have MFS root.

Perhaps this "skip" is due to one of partial dags I have (e.g. ipfs files cp /ipfs/bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma /partial-1EiB or ipfs files cp /ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze /wikipedia-350GiB)?

Lmk if this is not enough, I could block some time to debug try to reproduce with smaller repo if that helps.

hsanjuan · 2025-04-01T07:27:10Z

@lidel the only way I get the dagprovider to spit out the error is when the root is missing. Walking partial DAGs should not cause it. Otherwise it would indicate that we are aborting partial-dag walks, which we shouldn't.

I have updated boxo to log the root cid when this error is produced. Can you test again? We can then decide at least if this root cid belongs to mfs or to a specific DAG or to something else.

lidel · 2025-04-02T18:41:10Z

@hsanjuan as far i can tell, the error is not due to MFS root root missing, but one of child branches missing from local store.

I was able to reproduce the problem with ipfs config Reprovider.Strategy mfs, ipfs config --json Routing.AcceleratedDHTClient true

To see it, you need to apply change to boxo to show the actual child CID that was skipped, and not the root of entire MFS (ipfs/boxo@e08fc17, added to ipfs/boxo#905)

I created MFS to be huge with lazy-loaded items:

$ ipfs files cp /ipfs/bafybeie6tqsijf7j6gpow5q3qdjmtqgscrmd2ijl4olpxanbbptmvh7sui /_Tsize_test
$ ipfs files cp /ipfs/bafybeie6tqsijf7j6gpow5q3qdjmtqgscrmd2ijl4olpxanbbptmvh7sui /_Tsize_test2
$ ipfs files cp /ipfs/bafybeie6tqsijf7j6gpow5q3qdjmtqgscrmd2ijl4olpxanbbptmvh7sui /_Tsize_test3
$ ipfs files cp /ipfs/bafybeie6tqsijf7j6gpow5q3qdjmtqgscrmd2ijl4olpxanbbptmvh7sui /_Tsize_test4

Then restarted the node, waited ~5min for accelerated client to finish mapping, and manually triggered ipfs bitswap reprovide.

ipfs daemon printed ERROR:

Daemon is ready
2025-04-02T19:51:08.768+0200	ERROR	provider.batched	provider/dagprovider.go:41	dagprovider dag traversal error from root QmRfuVHxBJWZTwpDpsFYrG545vgv2qCt5Biopbd2hmeKiF: error traversing node at "_Tsize_test/bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma/bafybeiewrbcx23v7sz7fgcmavanfxq4ki2rsuoguvrpiyc4sdyxjdhttwm": could not load link "bafybeiewrbcx23v7sz7fgcmavanfxq4ki2rsuoguvrpiyc4sdyxjdhttwm": unable to find bafybeiewrbcx23v7sz7fgcmavanfxq4ki2rsuoguvrpiyc4sdyxjdhttwm in local blockstore: skip
2025-04-02T20:01:08.770+0200	ERROR	provider.batched	provider/dagprovider.go:41	dagprovider dag traversal error from root QmRfuVHxBJWZTwpDpsFYrG545vgv2qCt5Biopbd2hmeKiF: error traversing node at "_Tsize_test/bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma/bafybeiewrbcx23v7sz7fgcmavanfxq4ki2rsuoguvrpiyc4sdyxjdhttwm": could not load link "bafybeiewrbcx23v7sz7fgcmavanfxq4ki2rsuoguvrpiyc4sdyxjdhttwm": unable to find bafybeiewrbcx23v7sz7fgcmavanfxq4ki2rsuoguvrpiyc4sdyxjdhttwm in local blockstore: skip

This reverts commit 249ee3f.

lidel · 2025-04-09T01:33:05Z

After staring at this for far too long, the error seems to be somehow bogus – perhaps it is the last internal skip being returned, or racy one due intenal lock?

With some extra debug log, confirmed the provide of all local blocks occurs every time, including the mfs root:

$ ipfs files cp /ipfs/bafybeie6tqsijf7j6gpow5q3qdjmtqgscrmd2ijl4olpxanbbptmvh7sui /_Tsize_test
$ ipfs files cp /ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze/wiki /en.wikipedia-on-ipfs.org-wiki
$ ipfs files stat / | head -1
QmTeDEtragGSErNjsN7czYqX8KWVSoG3WHKEqDuDiYvQ9U
$ ipfs files ls / -l
_Tsize_test/	bafybeie6tqsijf7j6gpow5q3qdjmtqgscrmd2ijl4olpxanbbptmvh7sui	0
en.wikipedia-on-ipfs.org-wiki/	bafybeihn2f7lhumh4grizksi2fl233cyszqadkn424ptjajfenykpsaiw4	0

2025-04-09T03:26:25.269+0200	DEBUG	provider.batched	provider/dagprovider.go:16	DAG provider loaded	{"root": "QmTeDEtragGSErNjsN7czYqX8KWVSoG3WHKEqDuDiYvQ9U"}
2025-04-09T03:26:25.269+0200	DEBUG	provider.batched	provider/dagprovider.go:25	calling BlockAll	{"root": "QmTeDEtragGSErNjsN7czYqX8KWVSoG3WHKEqDuDiYvQ9U"}
skip error triggered bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma block was not found locally (offline): ipld: could not find bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma
skip error triggered bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma block was not found locally (offline): ipld: could not find bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma
skip error triggered bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma block was not found locally (offline): ipld: could not find bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma
skip error triggered bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma block was not found locally (offline): ipld: could not find bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma
skip error triggered bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma block was not found locally (offline): ipld: could not find bafybeif54ksdqydq3rliwjqps3wup6nulfbbrzgcvrrd5cgqqvps46eyma
skip error triggered bafybeichnleonsnrhdmykteijc2klanhwcrzoek4bzgdkx6amq3gwtj6uq block was not found locally (offline): ipld: could not find bafybeichnleonsnrhdmykteijc2klanhwcrzoek4bzgdkx6amq3gwtj6uq
2025-04-09T03:26:25.270+0200	DEBUG	provider.batched	provider/dagprovider.go:45	dagprovider dag traversal error from root QmTeDEtragGSErNjsN7czYqX8KWVSoG3WHKEqDuDiYvQ9U: skip
2025-04-09T03:26:25.771+0200	DEBUG	provider.batched	provider/reprovider.go:328	provide key QmTeDEtragGSErNjsN7czYqX8KWVSoG3WHKEqDuDiYvQ9U
2025-04-09T03:26:25.771+0200	DEBUG	provider.batched	provider/reprovider.go:328	provide key bafybeie6tqsijf7j6gpow5q3qdjmtqgscrmd2ijl4olpxanbbptmvh7sui
2025-04-09T03:26:25.771+0200	DEBUG	provider.batched	provider/reprovider.go:328	provide key bafybeihn2f7lhumh4grizksi2fl233cyszqadkn424ptjajfenykpsaiw4
2025-04-09T03:26:25.771+0200	DEBUG	provider.batched	provider/reprovider.go:350	starting provide of 3 keys
2025-04-09T03:27:02.532+0200	DEBUG	provider.batched	provider/reprovider.go:366	finished providing of 3 keys. It took 36.760657837s with an average of 12.253552612s per provide

This makes me believe we should not log ERROR if err := fetcherhelpers.BlockAll is traversal.SkipMe and instead log it to debug. Will double check tomorrow with fresh eyes.

caused by boxo adding new bitswap one

lidel

Thank you @hsanjuan, tested as much I could and provides what it should provide.

We will be testing pinned+mfs in collab cluster once 0.35.0-rc1 ships (we use mfs for extra backups of latest versions of websites)

hsanjuan self-assigned this Mar 13, 2025

hsanjuan mentioned this pull request Mar 13, 2025

Add MFS Support to Reprovider Strategy #10704

Closed

hsanjuan added 2 commits March 14, 2025 14:31

provider: close mfsRootProvider channel

be53048

Merge remote-tracking branch 'origin/master' into provide-mfs-best-ef…

c606909

…fort

hsanjuan commented Mar 14, 2025

View reviewed changes

hsanjuan added 4 commits March 17, 2025 11:37

provider: add pinned+mfs strategy. Restored pinned, roots strategy.

4c1ae00

provider: use offlineIPLDFetcher to traverse the pinset.

d0241e0

docs: update provider strategies

946b8e3

Merge remote-tracking branch 'origin/master' into provide-mfs-best-ef…

7cd01b1

…fort

hsanjuan marked this pull request as ready for review March 17, 2025 10:54

hsanjuan requested a review from a team as a code owner March 17, 2025 10:54

hsanjuan requested a review from lidel March 17, 2025 10:54

This was referenced Mar 18, 2025

Release 0.34 #10685

Closed

Release 0.35 #10760

Open

lidel mentioned this pull request Mar 28, 2025

feat: NewDAGProvider to walk partial DAGs in offline mode ipfs/boxo#847

Closed

lidel added 4 commits March 28, 2025 22:57

Merge branch 'master' into provide-mfs-best-effort

973dd19

docs(config): clarify strategies

71edc64

chore: switch to updated boxo#847

e0175f2

docs(changelog): move to 0.35 + expand

42ccd86

update boxo

e3a8301

lidel mentioned this pull request Apr 2, 2025

Kubo users can more directly control which data is advertised — IPFS/2025 ipshipyard/roadmaps#8

Open

lidel mentioned this pull request Apr 2, 2025

feat: NewDAGProvider to walk partial DAGs in offline mode ipfs/boxo#905

Merged

hsanjuan added 4 commits April 4, 2025 23:33

go.mod update

f6f5a43

Boxo with Println statements

249ee3f

Revert "Boxo with Println statements"

15cb0b3

This reverts commit 249ee3f.

mfs provider: flush mfs before getting root CID.

d67bdc0

lidel added 2 commits April 9, 2025 17:12

Merge 'origin/master' into provide-mfs-best-effort

5b452eb

chore: latest boxo#905

0408e40

lidel changed the title ~~provider: provide MFS DAG using offline fetcher.~~ feat: support partial DAG provide with Reprovider.Strategy=mfs|pinned+mfs Apr 9, 2025

lidel changed the title ~~feat: support partial DAG provide with Reprovider.Strategy=mfs|pinned+mfs~~ feat: partial DAG provides with Reprovider.Strategy=mfs|pinned+mfs Apr 9, 2025

lidel mentioned this pull request Apr 9, 2025

Pinning large CID is "stuck" at same node #8103

Open

lidel added 2 commits April 9, 2025 17:37

chore: fix prometheus metric test

39b0086

caused by boxo adding new bitswap one

chore: latest boxo main

35d952a

lidel approved these changes Apr 9, 2025

View reviewed changes

lidel merged commit 996bcf3 into master Apr 9, 2025
16 checks passed

lidel deleted the provide-mfs-best-effort branch April 9, 2025 16:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: partial DAG provides with Reprovider.Strategy=mfs|pinned+mfs #10754

feat: partial DAG provides with Reprovider.Strategy=mfs|pinned+mfs #10754

hsanjuan commented Mar 13, 2025 •

edited by lidel

Loading

hsanjuan Mar 14, 2025

lidel Mar 14, 2025

hsanjuan Mar 17, 2025

hsanjuan Mar 17, 2025

lidel Mar 17, 2025 •

edited

Loading

hsanjuan Mar 17, 2025

lidel commented Mar 29, 2025

lidel commented Mar 29, 2025

hsanjuan commented Mar 31, 2025

hsanjuan commented Mar 31, 2025

lidel commented Mar 31, 2025 •

edited

Loading

hsanjuan commented Apr 1, 2025

lidel commented Apr 2, 2025 •

edited

Loading

lidel commented Apr 9, 2025

lidel left a comment •

edited

Loading

feat: partial DAG provides with Reprovider.Strategy=mfs|pinned+mfs #10754

feat: partial DAG provides with Reprovider.Strategy=mfs|pinned+mfs #10754

Conversation

hsanjuan commented Mar 13, 2025 • edited by lidel Loading

hsanjuan Mar 14, 2025

Choose a reason for hiding this comment

lidel Mar 14, 2025

Choose a reason for hiding this comment

hsanjuan Mar 17, 2025

Choose a reason for hiding this comment

hsanjuan Mar 17, 2025

Choose a reason for hiding this comment

lidel Mar 17, 2025 • edited Loading

Choose a reason for hiding this comment

hsanjuan Mar 17, 2025

Choose a reason for hiding this comment

lidel commented Mar 29, 2025

lidel commented Mar 29, 2025

hsanjuan commented Mar 31, 2025

hsanjuan commented Mar 31, 2025

lidel commented Mar 31, 2025 • edited Loading

hsanjuan commented Apr 1, 2025

lidel commented Apr 2, 2025 • edited Loading

lidel commented Apr 9, 2025

lidel left a comment • edited Loading

Choose a reason for hiding this comment

hsanjuan commented Mar 13, 2025 •

edited by lidel

Loading

lidel Mar 17, 2025 •

edited

Loading

lidel commented Mar 31, 2025 •

edited

Loading

lidel commented Apr 2, 2025 •

edited

Loading

lidel left a comment •

edited

Loading