Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(file source)!: migrate checkpoints to single JSON file #4899

Merged
merged 7 commits into from
Nov 12, 2020

Conversation

lukesteensen
Copy link
Member

Closes #1427
Closes #1779

This adds functionality to migrate existing checkpoints to a new JSON-based file approach. It should be fully compatible, both backwards and forwards. This fixes a couple of outstanding issues and opens the door to storing more information with checkpoints, allowing for more intelligent fingerprinting algorithms.

Once this has been deployed in a new version, we should be able to remove a lot of the old code and clean things up, just noting that users will need to transition through this version if they want to migrate older-style checkpoints.

The first commit also fixes a bug introduced in #1499 and better testing that would have caught it.

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
@lukesteensen lukesteensen added the source: file Anything `file` source related label Nov 6, 2020
Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
@@ -62,7 +106,7 @@ impl Checkpointer {
}
'i' => {
let (dev, ino, pos) =
scan_fmt!(file_name, "i{x}.{y}.{}", [hex u64], [hex u64], FilePosition)
scan_fmt!(file_name, "i{x}.{x}.{}", [hex u64], [hex u64], FilePosition)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this intentional?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is the bug fix mentioned in the description. I can pull this commit out to its own PR if that'd be helpful, as I think it needs to land before 0.11 /cc @binarylogic

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind it here.

@Hoverbear Hoverbear changed the title chore(file source): migrate checkpoints to single JSON file chore(file source)!: migrate checkpoints to single JSON file Nov 6, 2020
Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
@lukesteensen
Copy link
Member Author

@Hoverbear can you explain why you changed the title of this PR? The change is specifically designed to be backward-compatible.

@Hoverbear
Copy link
Contributor

@lukesteensen I agree with you but it's important we note that this release has to be transitioned through if skipping releases.

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
@lukesteensen
Copy link
Member Author

The mac failures look like they could be related to the change but they're passing for me locally. Rerunning to see if they're transient.

@lukesteensen lukesteensen merged commit 5e5857c into master Nov 12, 2020
@lukesteensen lukesteensen deleted the fs-checkpointing branch November 12, 2020 02:53
mengesb pushed a commit to jacobbraaten/vector that referenced this pull request Dec 9, 2020
…otdev#4899)

* fix bug in decoding

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>

* transition to storing checkpoints in a json file

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>

* fix mysterious issue

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>

* clarify comment

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>

* add ignore_before handling

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>

* clippy fixes

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>

* clippy fixes for tests

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
Signed-off-by: Brian Menges <brian.menges@anaplan.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
source: file Anything `file` source related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reapproach file checkpointing strategy Cleanup checkpoints for file source
2 participants