source-salesforce/destination-bigquery: Normalization failed with SyncMode Incremental Deduped+History #28264
Description
Connector Name
source-salesforce
Connector Version
2.1.0
What step the error happened?
During the sync
Revelant information
Airbyte OSS version 0.50.7 running on GCP VM
source-salesforce version 2.1.0 using Salesforce Sandbox
destination-bigquery version 1.5.1 using GCS Staging
I am running a sync between Salesforce and BigQuery with SyncMode Incremental Deduped+History and the sync fails while normalization. The log states that "2023-07-12 12:18:14 source > Finished syncing SourceSalesforce" and the data arrives in GCS. The airbyte_raw tables are filled with data too but the normalized tables are empty in BigQuery. The sync with Salesforce, GCS and airbyte_raw data transfer seems to be working but the nomalization part fails.
I tested syncing in SyncMode full-refresh overwrite and the sync and normalization worked.
The log delivers another error "$.SystemModstamp: 2023-07-11T11:07:16.000+0000 is an invalid date-time". This is the cursor of SyncMode Incremental Deduped+History. I received this invalid date-time error for two streams (Account, Order). I tested removing these two streams but the sync still fails while normalization without the invalid date-time errors but with similar errors -> relevant log output Row 1294-1297.
Here are the failure reasons stated by the Airbyte UI and the log file:
- Failure Origin: normalization, Message: Normalization failed during the dbt run. This may indicate a problem with the data itself. (in Job History)
- Failure reason: Pickling client objects is explicitly not supported. (in View Logs)
- a22f16eb_8a67_473e_9530_d11ad11c60c2_job_12_attempt_1_txt.txt
Relevant log output
Rows 933, 934
2023-07-12 12:18:14 WARN i.a.w.i.FieldSelector(lambda$reportMetrics$1):124 - Schema validation errors found for stream _Order. Error messages: [$.LastModifiedDate: 2023-07-12T11:57:06.000+0000 is an invalid date-time,
2023-07-12 12:18:14 WARN i.a.w.i.FieldSelector(lambda$reportMetrics$1):124 - Schema validation errors found for stream _Account. Error messages: [$.SystemModstamp: 2023-07-12T05:30:10.000+0000 is an invalid date-time,
Rows 1294-1297. This error repeats for all streams.
2023-07-12 12:19:03 normalization > Unhandled error while executing model.airbyte_utils.Domain__c_stg
Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.
2023-07-12 12:19:03 normalization > 1 of 83 ERROR creating view model _airbyte_salesforce_staging.Domain__c_stg............................................. [ERROR in 1.39s]
Contribute
- Yes, I want to contribute