Passing number of stream in case of using bqstorage client #1695
Labels
api: bigquery
Issues related to the googleapis/python-bigquery API.
priority: p3
Desirable enhancement or fix. May not be included in next release.
type: feature request
‘Nice-to-have’ improvement, new feature or different behavior or design.
Hi,
I am using GCP batch to download the data from the BigQuery table using the bqStorage library
The problem that I start to face is the number of threads spawned when
_preserver_order
is not set which causes it to spawn up to 128 threads in case using (e2-medium, e2-small) machine type hangs the program can we add an argument in the function to control the number of reads stream that we have.eg:
for page in iteratable.to_arrow_iterable(bqstorage_client=client, max_queue_size=num_consumers * 2, no_of_stream=<>):
The text was updated successfully, but these errors were encountered: