Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(gatsby): purge jobsV2 results when cache is corrupt #27700

Merged
merged 1 commit into from
Oct 29, 2020

Conversation

vladar
Copy link
Contributor

@vladar vladar commented Oct 28, 2020

Description

We consider Gatsby cache to be corrupt when a user deletes the public folder but not the .cache folder. In this case we purge the cache for him (via #27549) and also call DELETE_CACHE to purge the redux state that is already in memory by this point.

But jobsV2 reducer doesn't clear job cache on DELETE_CACHE action. This makes sense because sharp jobs can survive .cache folder removal. But they can't survive the public folder removal (as that's where job results are physically stored).

This PR changes the behvaior of jobsV2 reducer and makes it purge the job cache when the cache is corrupt (so only when the public folder is deleted).

Related Issues

Fixes #27699

@vladar vladar requested a review from a team as a code owner October 28, 2020 13:53
@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Oct 28, 2020
@vladar vladar removed the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Oct 28, 2020
@wardpeet
Copy link
Contributor

Well, in the long run this isn't the right fix. We should be able to check if the output file still exists. If it does then we can keep it. If not we mark it as stale. Currently, we're not saving the files inside the cache so it would be more work than this PR.

Why is this relevant? Jobs API shouldn't just be used for files but can also be used for data like markdown transformations which never get stale as the contentDigest takes care of it.

@vladar
Copy link
Contributor Author

vladar commented Oct 29, 2020

Agreed, we need a better fix for this. But we don't know output artifacts of jobs to check if they exist. So that's just a workaround.

Also, it's a very edge case - when people delete their public folder but not .cache (see details in the linked issue).

In the past this would just throw a bunch of errors and wouldn't work at all. But now that we delete the .cache folder when public is not there - the build actually succeeds but transformed images are not there.

It happens because we load redux state from cache before we actually delete the .cache folder and so we emit this DELETE_CACHE action to make sure in-memory redux state is also purged:

// Tell reducers to delete their data (the store will already have
// been loaded from the file system cache).
store.dispatch({
type: `DELETE_CACHE`,
})

Far from pretty 😕

@vladar vladar added the bot: merge on green Gatsbot will merge these PRs automatically when all tests passes label Oct 29, 2020
@gatsbybot gatsbybot merged commit a6ecfb2 into master Oct 29, 2020
@delete-merged-branch delete-merged-branch bot deleted the vladar/gh-27699 branch October 29, 2020 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bot: merge on green Gatsbot will merge these PRs automatically when all tests passes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Job results are kept in cache even when the cache is corrupt
3 participants