Skip to content

🐛 [firestore-bigquery-export] The import script fails when run on multiple threads with millions of records. #1852

Open
@RyosukeNishi

Description

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository.
If you have a general question, need help debugging, or fall into some
other category use one of these other channels:

  • For general technical questions, post a question on StackOverflow
    with the firebase tag.
  • For general Firebase discussion, use the firebase-talk
    google group.
  • To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to
    Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

  • Extension name: firestore-bigquery-export
  • Extension version:
    • The extension itself
      • 0.1.41
    • The import script
      • 0.1.19
  • Configuration values (redact info where appropriate):
スクリーンショット 2023-11-29 12 02 15 スクリーンショット 2023-11-29 12 02 23

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

fs-bq-import-collection \
  --non-interactive \
  --project ${MY_PROJECT_NAME} \
  --source-collection-path chats/{chat_token}/messages/{message_unique_id}/reaction_groups \
  --query-collection-group true \
  --dataset firestore_export \
  --table-name-prefix reaction_groups \
  --batch-size 300 \
  --dataset-location us \
  --multi-threaded true \
  --use-new-snapshot-query-syntax true \
  --use-emulator false

What happened? How can we make the problem occur?
This could be a description, log/console output, etc.

Expected result

The script succeeds.

Actual result
An error has occurred on the following documents, please re-run or insert the following query documents manually... {"startAt":{"before":true,"values":[{"referenceValue":"projects/*redacted*/databases/(default)/documents/chats/nishi-chat-token/messages/026e99f6-dd02-42c0-9d14-e5cda422d768/reaction_groups/THUMBS_UP","valueType":"referenceValue"}]},"endAt":{"before":true,"values":[{"referenceValue":"projects/*redacted*/databases/(default)/documents/chats/nishi-chat-token/messages/05a5784c-2818-45cf-84f6-1e214dcc28da/reaction_groups/THUMBS_UP","valueType":"referenceValue"}]}}
Note:
  • The collection group reaction_groups has around 2.5 million documents.
  • The script succeeds when I run it on a single thread.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions