🐛 [firestore-bigquery-export] Docs with @type keys throw error and aren't synced #1903
Description
[REQUIRED] Step 2: Describe your configuration
- Extension name: firestore-bigquery-export
- Extension version: latest
- Configuration values (redact info where appropriate): not relevant
[REQUIRED] Step 3: Describe the problem
We recently updated our extension and noticed a bunch of new Data cannot be decoded from JSON. errors. It's not clear to me if these are showing up because the function structure or logging changed or if the actual json parsing has changed since we upgraded.
The current version of the extension appears to be using the decode method from the firebase SDK https module, which throws an error if an object has a key of @type with a value that isn't LONG_TYPE or UNSIGNED_LONG_TYPE. We use the @type field in a ton of places and it's not feasible to change the structure of our data model. Is there a way to work around the issue or somehow bypass this decoding error?
Steps to reproduce:
Create a firestore document that has a key of @type and a string value. We typically have arrays of objects, so something like:
{
"id": 1234,
"views": [
{ "@type": "query", operator: "is", value: "some query" }
]
}
Expected result
The data gets synced to BigQuery with the right data.
Actual result
An error is thrown with the jsonPayload of:
jsonPayload: {
@type: "query"
message: "Data cannot be decoded from JSON."
operator: "is"
value: "some query"
}
Activity