fix(gatsby): invalidate queries if page context changes between runs

[pieh]

This is successor to #28316 which had potential of causing OOM crashes when persisting/serializing data in case there was huge amount of data passed via page context.

Instead of persisting entire page objects this persists just hash generated from page context which should minimize how much more data will be persisted (and hopefully avoid OOM crashes)

TODO:

• determine if

  gatsby/packages/gatsby/src/services/create-pages.ts

  Line 57 in c4978e9

  const deletedPages = deleteUntouchedPages(store.getState().pages, timestamp)

  needs to be adjusted - right now it uses pages slice which is not persisted - so it's entirely possible that this will keep stale page context hashes around. Potential solution here would be to add timestamp to newly added reducer (maybe even just move current ones we have in pages to new one) and switch to using new (persisted) slice of redux state for finding pages that were not re-created.

Related Issues

fixes #28281
fixes #26520
[ch19791]

[vladar]

Looks good to me 👍 At least I don't see any pitfalls with this approach.

[vladar]

🎉

[pieh]

Ok, this is not sufficent change. As I noted in the description

determine if

gatsby/packages/gatsby/src/services/create-pages.ts

Line 57 in c4978e9

const deletedPages = deleteUntouchedPages(store.getState().pages, timestamp)

needs to be adjusted - right now it uses pages slice which is not persisted - so it's entirely possible that this will keep stale page context hashes around. Potential solution here would be to add timestamp to newly added reducer (maybe even just move current ones we have in pages to new one) and switch to using new (persisted) slice of redux state for finding pages that were not re-created.

Just did run some manual tests for it. For cases when we:

• run build 1 that creates "page A",
• run build 2 without "page A",
• run build 3 with "page A" re-added (with same context as in build 1)
  we don't rerun query in "build 3" and "build 2" deletes generated files - most importantly page-data ( to not show page that was "deleted")

This result in build error as follows:

failed Building static HTML for pages - 0.757s
 ERROR #95313 
Building static HTML failed for path "/context/b/"
See our docs page for more info on this error: https://gatsby.dev/debug-html
  Error: ENOENT: no such file or directory, open '/Users/misiek/dev/gatsby-starter-blog/public/page-data/context/b/page-data.json'
  - render-page.js:635 getPageData
    /Users/misiek/dev/gatsby-starter-blog/public/render-page.js:635:1022
  - render-page.js:640 Module../.cache/static-entry.js.__webpack_exports__.default
    /Users/misiek/dev/gatsby-starter-blog/public/render-page.js:640:1497
  - render-html.js:30 
    [gatsby-starter-blog]/[gatsby]/dist/utils/worker/render-html.js:30:36

So we do need to cover properly cleaning up reducer that we will persist

[vladar]

Just trying to discard my review after @pieh request

[pieh]

To add to my above comment - just adding timestamp and rest of fields that are used in findChangedPages / deleteUnotuchedPages doesn't seem sufficient, because after finding deleted pages we do emit DELETE_PAGE action which currently uses entire page object as payload. We can adjust internals to make sure things are fine. But various plugins might already subscribe to this event in some cases and while this technically wouldn't be breaking change, it could cause unforseen consequences for the ecosystem...

[pieh]

Ok, this is not feasible approach in the end due to above - I will actually use #28316 as a base and handle potential OOM issues by expanding persisting chunking to also chunk pages on top of nodes - #28590

Will be closing this PR