Add block filter via BadBits

[hannahhoward]

Goals

Provide a mechanism for filtering out illegal CID requests in bitswap.

Implementation

• This implements a lightweight block filter module that fetches the bad bits list and checks bitswap requests against what is in the list.
  • The list downloads and updates every five minutes in the background -- currently, it's only 1meg large
  • The in place block check is O(1) against an in memory copy of the list (currently, again, not a serious memory overhead)
• integrates the block filter into the remote blockstore in booster bitswap

For discusion (that's why it's a draft)

• Currently, this lives inside of booster bitswap -- should it instead live inside of boost and possibly the block API calls?

• I didn't put optimization around passing If-Not-Modified as request header and checking for a 304 Not Modified response -- this may be prudent as the list updates a lot but not every five minutes

• There's no on disk cache -- this was done for simplicity but does mean you need to fetch at startup. Also, this is a blocking call currently -- maybe that's not ideal?

• What is the right way to handle fetching/parsing errors? Right now it's just log.Errorf

• blocked on LoadBalancer for bitswap (and later, more of libp2p) #786 (currently that branch is the base for this one)

[dirkmc]

Currently, this lives inside of booster bitswap -- should it instead live inside of boost and possibly the block API calls?

Bitswap has a built-in mechanism for deny-lists that we could use.

I didn't put optimization around passing If-Not-Modified as request header and checking for a 304 Not Modified response -- this may be prudent as the list updates a lot but not every five minutes

I think that's a good idea - we expect the size of the list to grow in future so it would be good to avoid unnecessary downloads

There's no on disk cache -- this was done for simplicity but does mean you need to fetch at startup. Also, this is a blocking call currently -- maybe that's not ideal?

I'd suggest we do use an on-disk cache. The size of the list is likely to increase in future. I'd suggest that on startup

• if the list hasn't ever been fetched, block on fetching it
• if there is already an on-disk cache, just do a new fetch in the background

What is the right way to handle fetching/parsing errors? Right now it's just log.Errorf

That sounds right to me - I think it makes sense for the denylist to be best-effort

blocked on #786 (currently that branch is the base for this one)

Merged ✅

[hannahhoward]

@dirkmc

PR updated:

• uses bitswap peer request block filter feature in bitswap for filtering
• supports if-not-modified when fetching deny list over http
• holds on disk cache of last fetched list

[dirkmc]

I've approved this PR, with a few suggestions for minor improvements. Nice work! 🙌

[dirkmc]

My one concern here is what happens as the badbits list grows large. Is there a way we can put a reasonable bound on the amount of memory reserved for the badbits filter? Would it make sense to use something like leveldb to keep it on disk instead?
We can experiment with solutions in a follow-up PR.

[hannahhoward]

@dirkmc yes, we can definitely do follow up experiments if the list grows large. One interesting thing to note: currently, the list is only about 1MB large, way smaller than I would have guessed given how long it's been in operation.