🐛 [firestore-bigquery-export] The import script fails when run on multiple threads with millions of records. #1852
Open
Description
opened on Nov 29, 2023
[READ] Step 1: Are you in the right place?
Issues filed here should be about bugs for a specific extension in this repository.
If you have a general question, need help debugging, or fall into some
other category use one of these other channels:
- For general technical questions, post a question on StackOverflow
with the firebase tag. - For general Firebase discussion, use the firebase-talk
google group. - To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to
Firebase support directly.
[REQUIRED] Step 2: Describe your configuration
- Extension name: firestore-bigquery-export
- Extension version:
- The extension itself
- 0.1.41
- The import script
- 0.1.19
- The extension itself
- Configuration values (redact info where appropriate):
[REQUIRED] Step 3: Describe the problem
Steps to reproduce:
fs-bq-import-collection \
--non-interactive \
--project ${MY_PROJECT_NAME} \
--source-collection-path chats/{chat_token}/messages/{message_unique_id}/reaction_groups \
--query-collection-group true \
--dataset firestore_export \
--table-name-prefix reaction_groups \
--batch-size 300 \
--dataset-location us \
--multi-threaded true \
--use-new-snapshot-query-syntax true \
--use-emulator false
What happened? How can we make the problem occur?
This could be a description, log/console output, etc.
Expected result
The script succeeds.
Actual result
An error has occurred on the following documents, please re-run or insert the following query documents manually... {"startAt":{"before":true,"values":[{"referenceValue":"projects/*redacted*/databases/(default)/documents/chats/nishi-chat-token/messages/026e99f6-dd02-42c0-9d14-e5cda422d768/reaction_groups/THUMBS_UP","valueType":"referenceValue"}]},"endAt":{"before":true,"values":[{"referenceValue":"projects/*redacted*/databases/(default)/documents/chats/nishi-chat-token/messages/05a5784c-2818-45cf-84f6-1e214dcc28da/reaction_groups/THUMBS_UP","valueType":"referenceValue"}]}}
Note:
- The collection group
reaction_groups
has around 2.5 million documents. - The script succeeds when I run it on a single thread.
Activity