Skip to content
This repository has been archived by the owner on Aug 2, 2021. It is now read-only.

storage/localstore: Local Pinning fix for pinned chunks #2200

Merged
merged 2 commits into from
Jun 5, 2020

Conversation

santicomp2014
Copy link
Contributor

This PR closes issue #2195, to summarize GC was removing pinned chunks when it was not supposed to.

This is related to PR #2190 which was the incorrect fix for this bug, thanks to @jmozah for providing the correct fix for this situation.

The solution is to check if the chunk is in the pinIndex before inserting it into gcIndex, this avoids the GC removing the chunk in the first place.

Replaced in these files

  • mode_set.go
  • mode_get.go
  • mode_put.go
ok, err := db.pinIndex.Has(item)
	if err != nil {
		return 0, err
	}
	if !ok {
		err = db.gcIndex.PutInBatch(batch, item)
		if err != nil {
			return 0, err
		}
		gcSizeChange++
	}

Also, the test TestDBDebugIndexes was corrected due to the pinned chunks now being avoided correctly.

Test criteria were:

  • 500 chunk store capacity
  • no chunks in swarm storage (swarm folder empty)
  • pinning is enabled in the swarm node
  • Upload a chunk with --pin
  • Upload another chunk without pinning
  • Try to download it (should download correctly)
  • upload random chunks multiple times (20 rounds of 40000 chunks and 20 rounds of 2000 chunks)
dd if=/dev/urandom of=output.txt bs=4096 count=40000 && /home/git/go/src/github.com/ethersphere/swarm/build/bin/swarm --bzzapi http://localhost:8503 up output.txt
  • try to download upload/pinned chunk (should download correctly)
  • the other chunk uploaded without pinned flag should not be retrievable due to GC removing this from the store.

The bug is also in Bee and it is fixed in a similar fashion

Copy link
Collaborator

@jmozah jmozah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@santicomp2014 santicomp2014 merged commit ae39aaa into master Jun 5, 2020
@mortelli mortelli mentioned this pull request Jun 5, 2020
4 tasks
@mortelli mortelli deleted the local-pinning-fix-gcindex branch June 9, 2020 17:07
@mortelli mortelli added global pinning experimental implementation of global pinning and removed global pinning experimental implementation of global pinning labels Jun 12, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants