🐛 [Stream Firestore to BigQuery] fs-bq-import-collection doesn't recognize partitioned field

[ysak-y]

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export (storage-resize-images, firestore-send-email, etc)
• Extension version: 0.1.49

[REQUIRED] Step 3: Describe the problem

When I install extension with following configuration, it creates BigQuery table with createdAt partitioned field.

The important thing is its extension project id and BigQuery id is different (Cross-Platform BigQuery).

But When I run fs-bq-import-collection to backfill, it doesn't recognize any partitioned fields. The script shows following debug message.

{"severity":"WARNING","message":"Did not add partitioning to schema: Partitioning not enabled"}

Steps to reproduce:

Create Firestore table with createdAt TIMESTAMP field and Install extension with above configuration.

Expected result

fs-bq-import-collection recognize createdAt partition field and set it when importing.

Actual result

fs-bq-import-collection doesn't recognize createdAt partition field and set null to createAt field when importing.

[pr-Mais]

From the POSTINSATALL:

Note: To enable partitioning for a Big Query database, the following fields are required:

 - Time Partitioning option type
 - Time partitioning column name
 - Time partiitioning table schema
 - Firestore document field name

`Clustering` will not need to create or modify a table when adding clustering options, this will be updated automatically.

You're expecting the table to be created with partitioning on createdAt, however the configuration doesn't specify that as a Time partitioning column name, the one you have is timestamp.

[KristofSochan]

Having the same issue. The bigquery table is partitioned (see below) but the following warning is thrown at the start of fs-bq-import-collection.

{"severity":"INFO","message":"BigQuery table with name users_raw_changelog already exists in dataset ***!"}
{"severity":"INFO","message":"Clustering removed on users_raw_changelog"}
{"severity":"WARNING","message":"Did not add partitioning to schema: Partitioning not enabled"}

Table Type
Partitioned
Partitioned by
DAY
Partitioned on field
creation_timestamp
Partition expiration
Partitions do not expire
Partition filter
Not required

[cabljac]

I think the OP's issue is misconfiguration of the extension, as @pr-Mais mentions.

@KristofSochan are you also using the script cross-project, or from firestore to bigquery on the same GCP project?

Can you provide:

a) your extension configuration
b) the import script command you are using