BigQuery: Client.list_rows should prefetch the first page and populate total_rows and other attributes

[bits01]

https://googlecloudplatform.github.io/google-cloud-python/latest/_modules/google/cloud/bigquery/table.html#Table.fetch_data

Return value doc:

:returns: Iterator of row data :class:`tuple`s. During each page, the
                  iterator will have the ``total_rows`` attribute set,
                  which counts the total number of rows **in the table**
                  (this is distinct from the total number of rows in the
                  current page: ``iterator.page.num_items``).

The iterator does not have the total_rows attribute or a page attribute either.

Can fetch_data() get a count of all rows or the only way is to iterate and count?

[bits01]

QueryResults.fetch_data() too.

https://googlecloudplatform.github.io/google-cloud-python/latest/_modules/google/cloud/bigquery/query.html#QueryResults.fetch_data

[lukesneeringer]

Summoning @tswast since he has been in this code a lot recently and probably knows the answer off hand. :-)

[tswast]

There's definitely a pages property. Not sure about page.

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/99c4498339e8cddb628962215e7fa7953a77c957/bigquery/tests/system.py#L296-L299

Looking at https://github.com/GoogleCloudPlatform/google-cloud-python/blob/99c4498339e8cddb628962215e7fa7953a77c957/core/google/api/core/page_iterator.py#L248, I think you're right that the docs need to be updated.

[dhermes]

@tswast The iterator docs are great, they just got unpublished when the module moved from google.cloud.iterator to google.api.core.page_iterator.

/cc @jonparrott

[bits01]

#4153 is related.

[tswast]

The iterator is using

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/e6cc4b4a6486971f910139d766c7807a588ea53b/bigquery/google/cloud/bigquery/_helpers.py#L507-L522

which appears to be populating total_rows. I'll investigate further and make sure our system tests are checking for this property.

[bits01]

It works on QueryResult, but not on the iterator returned by the destination table fetch_data():
AttributeError: 'HTTPIterator' object has no attribute 'total_rows'

[tswast]

My guess as to what is happening is that total_rows only gets populated after you start iterating through, because it gets that information from the first page of results.

I agree that this is confusing behavior.

@jonparrott Do you think we could prefetch the first page when we create the iterator?

[bits01]

Indeed. Once you start iterating total_rows shows up. Should be made consistent with QueryResult, assuming that QueryResult even needs to exist at all.

[theacodes]

@jonparrott Do you think we could prefetch the first page when we create the iterator?

That's a better question for @dhermes, who designed the iterator.

[dhermes]

Yes. The easiest way to do this would be to just subclass the relevant iterator class and over-ride the _next_page method.

Or you could monkey-patch the _next_page method on your instances. This way you wouldn't have to subclass both the HTTP and gRPC iterators, but then you have the problem of figuring out how to fetch the first page via HTTP or gRPC.