--use_rest_api still seems to use storage api

[wnojopra]

Environment details

Ubuntu 18.04.5 LTS

Python 3.7.10

pip 21.1.3

google-cloud-bigquery 2.20.0
google-cloud-bigquery-datatransfer 3.2.0
google-cloud-bigquery-storage 2.4.0

Steps to reproduce

1. Use %%bigquery magic with the --use_rest_api flag on a project where I don't bigquery.readsession permissions.
2. Observe that the query fails due to not having readsession permissions.

Code example

%%bigquery my_test --use_rest_api
SELECT
*
FROM `aaa-willyn-test.test_dataset.my_test`

Stack trace

PermissionDenied                          Traceback (most recent call last)
<ipython-input-9-f594b2ed9ec4> in <module>
----> 1 get_ipython().run_cell_magic('bigquery', 'my_test --use_rest_api', 'SELECT\n*\nFROM `aaa-willyn-test.test_dataset.abi_copy`\n')

/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2401             with self.builtin_trap:
   2402                 args = (magic_arg_s, cell)
-> 2403                 result = fn(*args, **kwargs)
   2404             return result
   2405 

...skipping several frames here...

/opt/conda/lib/python3.7/site-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
     67             return callable_(*args, **kwargs)
     68         except grpc.RpcError as exc:
---> 69             six.raise_from(exceptions.from_grpc_error(exc), exc)
     70 
     71     return error_remapped_callable

/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

PermissionDenied: 403 request failed: the user does not have 'bigquery.readsessions.create' permission for 'projects/fc-product-demo'

[plamut]

Hmm... at some point the BQ storage API was made a default (before that, BQ Storage API had to be explicitly requested).

I suspect that cell magic just omits that "use BQ Storage", still assuming that REST API will be used by default, but that has changed.

Thanks for reporting this!

[wnojopra]

Thanks for the context @plamut ! In the meantime, are there any workarounds to get cell magic working with rest API? It doesn't seem to help if I install an older version of google-cloud-bigquery, say 2.14.0

[plamut]

@wnojopra Sorry for missing your question, I went offline for a few days and in the meantime too many notifications have piled up and yours got lost. :/

I can't think of any workarounds from the top of my head other than monekypatching the buggy code, but I'll have a look at this now. Unless it's something really weird and unexpected, you expect the fix some time next week.

[plamut]

@wnojopra Ah, but there IS a workaround - you can pass --max-results 9999999999 as a cell magic argument. Since BQ Storage is a streaming API, it cannot be used if max_results is set and REST API is used instead.

Just make sure that --max-results is really big so that it does not silently cut off any result rows.