[gatsby-source-contentful] Gatsby unaware of change in Contentful unless delete cache folder

[hrisng]

Hi, I'm facing a lot of issues that relate directly to the cache folder.

Description

1. If I query a reverse item in one of my query, for example, content type Brand is linked to Category, and I have the following query:

contentfulCategory {
  id
  brand {
    id
  }
}

Gatsby would not aware if I publish a new Brand, and the Category page remain the same. If I query for brand directly instead in brand template, the brand page is generated successfully. When I delete the cache folder, and run again, all query and pages work as expected.

2. If I add a new field to content type Category (that Brand link to) and is queried as above, then run gatsby develop, I would get the error unknown field brand in contentfulCategory. (query for brand from Category also won't work in Graphiql IDE). Again, manually delete the cache folder and run again solve the problem.

3. When I delete a field successfully in Contentful, that field will still be queryable in Gatsby. Delete cache folder also solve this.

Step to reproduce

1. Create content type Category and Brand (or any A and B)
2. In Brand, add a reference to Category
3. Run gatsby develop, it compiles successfully
4. Try each case in the description
5. Run gastby develop, and see the result
6. Delete the cache folder and run again, it work correctly.

Environment

Gatsby version: 1.9.145
Node.js version: 6.9.1
Operating System: Windows 6.1.7601

More Info

I'm using gatsby-plugin-react-next.
Before I found out deleting the cache folder help with the issues. Running gatsby develop multiple times would occasionally solve the problem.
Edit: It worked because I had some change in gatsby-node.js, which triggers the deletion of cache folder.

Request

Also, it would be great if there's a way to tell Gatsby to completely delete the cache folder on new build in the meantime. I didn't find anything in the documentation. Appreciate if you can show me the direction.

[KyleAMathews]

Sorry you're running into troubles and thanks for creating a detailed bug report!

A cache that you delete all the time isn't very helpful :-)

This sounds like a bug perhaps in gatsby-source-contentful where it's not correctly incrementally updating data in Gatsby so Gatsby can't correctly make the new schema.

[hrisng]

I'd love to dig deeper in the package code when I have time to see if there's anything suspicious. Deleting the cache is not desirable for me also. But for now, it's a temporary solution.

The point is we're planning to use Contentful webhook to tell Gastby to automatically rebuild on new content, so that my non-tech teammate can push new content on their own. So yeah, having to manually delete the cache kinda defeat the purpose.

Thanks for quick response and for Gatsby!

[kpollich]

Running into this today as well. Removing .cache isn't resolving for us though (or at least not for me).

Scenario:

1. Create content type in Contentful with Link field
2. Create entry for content type
3. Add single link to entry
4. Run gatsby develop
5. Gatsby generates a GraphQL schema where it assumes since only one Content type appears in the Link field that the Link field is limited to one type
6. Add another entry to the entry's Link field of a different type
7. Contentful GraphQL schema doesn't update on cache removal, rebuild, etc.

In the example here, we have a fullSlot field that contains a single Entry (could be many types), and it's "stuck" as ContentfulPlainTextModuleTest even though we have an entry whose fullSlot contains a different type of Entry.

Full error output

"errors": [
    {
      "message": "Expected value of type \"ContentfulPlainTextModuleTest\" but got: [object Object].",
      "locations": [
        {
          "line": 28,
          "column": 5
        }
      ],
      "path": [
        "allContentfulHomePage",
        "edges",
        0,
        "node",
        "grid",
        0,
        "fullSlot"
      ]
    }
  ],

[KyleAMathews]

Hmmm Gatsby should be creating a Union Type when there's multiple types being linked to:

gatsby/packages/gatsby/src/schema/infer-graphql-type.js

Line 316 in 3945d01

// If there's more than one type, we'll create a union type.

Perhaps something is wrong with the reference maps gatsby-source-contentful creates

gatsby/packages/gatsby-source-contentful/src/normalize.js

Line 89 in 3945d01

exports.buildForeignReferenceMap = ({

[hrisng]

@KyleAMathews I think I find out what went wrong. I'm super new to the project and how it actually works under the hood, so I might be wrong though. So please bear with me.

1. The first problem occured I think because of this:
   

   gatsby/packages/gatsby-source-contentful/src/gatsby-node.js

   Line 134 in 3945d01

   .filter(n => _.includes(newOrUpdatedEntries, n.id))

   
   We are checking the id of existing node against the ids of entries that changed, and in case of creating a new entry it will result in an empty array. So new reverse linkage for old nodes was not getting added.

2. We are using the sync endpoint to get the entryList (which only contain entries that were changed or deleted after the initial build).
   

   gatsby/packages/gatsby-source-contentful/src/fetch.js

   Line 38 in 3945d01

   let currentSyncData

   
   And we are also building foreignReferenceMap base on that list.
   

   gatsby/packages/gatsby-source-contentful/src/gatsby-node.js

   Line 115 in 3945d01

   const foreignReferenceMap = normalize.buildForeignReferenceMap({

   
   So what happened in the second case is that when I add a new field and populate it in Category, and do nothing with the other entries in Brand, only the updated category entries get returned in the entryList. It then attempts to build the ref map, which only contain ref in Category entries. If it has ref in itself to other content type, there's no problem because we do check it against a resolvable list. However, if something else was pointing to it, like in this case, it has no way to know.
   I did a small test to confirm this. If I also update the brand entries along with the categories, the reverse mapping work.

3. The third case I believe also happened because of the sync endpoint. That endpoint only response with changes in entries, no? Old nodes are not aware about the change in data model.

(Sorry for typos and wrong use of words. Not a native speaker, but I'm trying to communicate better)

[KyleAMathews]

Cool! What you're suggesting sounds reasonable — would you like to put together a PR with your fixes?

[hrisng]

@KyleAMathews I'd love to. It might take some time though cause I'm still kinda new to JS. In the meantime, if anyone need to fix this issue asap, please feel free to pick it up.

[ro-savage]

Running into a similar issue. I delete my export const query = graphql and yet I still see the data, I had to manually delete the cache.

[hrisng]

Sorry, I've been quite busy lately and this issue was on low priority for our own project. Gonna put together a fix in the next few days.

[calcsam]

Great, thanks for tackling this!

…

On Sat, Feb 3, 2018 at 10:30 PM, Harris Nguyen ***@***.***> wrote: Sorry, I've been quite busy lately and this issue was on low priority for our own project. Gonna put together a fix in the next few days. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3495 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEL3W66L9SUV9XTsUIv2oLxTfNwtKkaoks5tRU6NgaJpZM4Rb4OQ> .

[alex-saunders]

How's the fix coming along @harrisnguyen94 ? I've ran into the same issue and would be interested to hear how close we are to fixing this

[hrisng]

@alex-saunders hi, updating the reverse mapping for existing nodes (which solve issue 1 is simple and pretty much done. Issue 2 needs some thoughts though, but we have located what went wrong and kinda figure out how to fix it.
Issue 3 however, is a little bit different (making existing nodes aware of change in data model), and maybe will be tackled later.
So yeah, it depends on what issues you're facing. If 1 and 2, I would say we're pretty close. But if it's 3, it may take longer; if this is the case, please feel free to pick it up.

It was Lunar New Year so I was off for the last 2 weeks. Sorry for the delay. Looking forward to seeing these problems solved asap.

[alex-saunders]

Hi @harrisnguyen94 , no worries about the delay, sounds like it's pretty much there anyway! I'm only really needing (1) so happy to hear that the fix should be coming asap.