Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add startIndex to bigquery.table.Table.fetch_data #3081

Closed
kylemcdonald opened this issue Feb 28, 2017 · 2 comments
Closed

Add startIndex to bigquery.table.Table.fetch_data #3081

kylemcdonald opened this issue Feb 28, 2017 · 2 comments
Assignees
Labels
api: bigquery Issues related to the BigQuery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@kylemcdonald
Copy link

kylemcdonald commented Feb 28, 2017

The API describes a startIndex parameter for the "Tabledata: list" endpoint. But the implementation doesn't include this parameter. An example of a class does implement a startIndex parameter is bigquery.query.Query as described in the "Jobs: getQueryResults" endpoint.

I tried editing the code to submit a PR, but I'm not familiar with testing this kind of Python package locally (pip install -e . didn't install the packages from their local copy as I expected). In lieu of a PR I can share this patched class TableWithStartIndex that works for my use case:

from google.cloud.bigquery.table import Table, _item_to_row, _rows_page_start
from google.cloud.iterator import HTTPIterator
class TableWithStartIndex(Table):
    def fetch_data_with_start_index(self, max_results=None, page_token=None, client=None, start_index=None):
        client = self._require_client(client)
        path = '%s/data' % (self.path,)
        extra_params = {}
        if start_index is not None:
            extra_params['startIndex'] = start_index
        iterator = HTTPIterator(client=client, path=path,
                                item_to_value=_item_to_row, items_key='rows',
                                page_token=page_token, max_results=max_results,
                                page_start=_rows_page_start,
                                extra_params=extra_params)
        iterator.schema = self._schema
        # Over-ride the key used to retrieve the next page token.
        iterator._NEXT_TOKEN = 'pageToken'
        return iterator

I added three small chunks: the start_index=None argument, this block:

        extra_params = {}
        if start_index is not None:
            extra_params['startIndex'] = start_index

And the bit that passes the extra_params to the HTTPIterator: extra_params=extra_params.

@daspecster daspecster added the api: bigquery Issues related to the BigQuery API. label Feb 28, 2017
@lukesneeringer
Copy link
Contributor

Thank you for opening the issue. We will look into it and get back to you soon.

@danoscarmike danoscarmike added priority: p2 Moderately-important priority. Fix may not be included in next release. Status: Acknowledged type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. labels Feb 28, 2017
@lukesneeringer lukesneeringer removed the priority: p2 Moderately-important priority. Fix may not be included in next release. label Aug 9, 2017
@tswast
Copy link
Contributor

tswast commented Oct 24, 2017

Fixed by #4245.

The new Client.list_rows function takes an optional start_index parameter.

@tswast tswast closed this as completed Oct 24, 2017
parthea pushed a commit that referenced this issue Jun 4, 2023
* Data Labeling Beta samples [(#2096)](GoogleCloudPlatform/python-docs-samples#2096)

* add files

* upate create_annotation_spec_set and test

* add requirements.txt

* update create_instruction and test

* update import data and test

* add label image and test

* add label_text test

* add label_video_test

* add manage dataset and tests

* flake

* fix

* add README

* Adds updates including compute [(#2436)](GoogleCloudPlatform/python-docs-samples#2436)

* Adds updates including compute

* Python 2 compat pytest

* Fixing weird \r\n issue from GH merge

* Put asset tests back in

* Re-add pod operator test

* Hack parameter for k8s pod operator

* Update datalabeling samples to hit test endpoint. [(#2641)](GoogleCloudPlatform/python-docs-samples#2641)

* Auto-update dependencies. [(#2005)](GoogleCloudPlatform/python-docs-samples#2005)

* Auto-update dependencies.

* Revert update of appengine/flexible/datastore.

* revert update of appengine/flexible/scipy

* revert update of bigquery/bqml

* revert update of bigquery/cloud-client

* revert update of bigquery/datalab-migration

* revert update of bigtable/quickstart

* revert update of compute/api

* revert update of container_registry/container_analysis

* revert update of dataflow/run_template

* revert update of datastore/cloud-ndb

* revert update of dialogflow/cloud-client

* revert update of dlp

* revert update of functions/imagemagick

* revert update of functions/ocr/app

* revert update of healthcare/api-client/fhir

* revert update of iam/api-client

* revert update of iot/api-client/gcs_file_to_device

* revert update of iot/api-client/mqtt_example

* revert update of language/automl

* revert update of run/image-processing

* revert update of vision/automl

* revert update testing/requirements.txt

* revert update of vision/cloud-client/detect

* revert update of vision/cloud-client/product_search

* revert update of jobs/v2/api_client

* revert update of jobs/v3/api_client

* revert update of opencensus

* revert update of translate/cloud-client

* revert update to speech/cloud-client

Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Co-authored-by: Doug Mahugh <dmahugh@gmail.com>

* Update datalabeling to match lint. [(#2642)](GoogleCloudPlatform/python-docs-samples#2642)

* datalabeling: ensure all tests use test endpoint [(#2918)](GoogleCloudPlatform/python-docs-samples#2918)

* datalabeling: ensure all tests use test endpoint

* requires an input csv for text input, slight print statement cleanup

Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>

* chore(deps): update dependency google-cloud-datalabeling to v0.4.0 [(#3081)](GoogleCloudPlatform/python-docs-samples#3081)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [google-cloud-datalabeling](https://togithub.com/googleapis/python-datalabeling) | minor | `==0.3.0` -> `==0.4.0` |

---

### Release Notes

<details>
<summary>googleapis/python-datalabeling</summary>

### [`v0.4.0`](https://togithub.com/googleapis/python-datalabeling/blob/master/CHANGELOG.md#&#8203;040-httpswwwgithubcomgoogleapispython-datalabelingcomparev030v040-2020-01-31)

[Compare Source](https://togithub.com/googleapis/python-datalabeling/compare/v0.3.0...v0.4.0)

##### Features

-   **datalabeling:** undeprecate resource name helper methods (via synth) ([#&#8203;10039](https://www.github.com/googleapis/python-datalabeling/issues/10039)) ([88f8090](https://www.github.com/googleapis/python-datalabeling/commit/88f809008ee6a709c02c78b1d93af779fab19adb))

##### Bug Fixes

-   **datalabeling:** deprecate resource name helper methods (via synth) ([#&#8203;9832](https://www.github.com/googleapis/python-datalabeling/issues/9832)) ([e5f9021](https://www.github.com/googleapis/python-datalabeling/commit/e5f902154ebe7fcb139aa405babfe9993fd51319))

</details>

---

### Renovate configuration

:date: **Schedule**: At any time (no schedule defined).

:vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

:recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox.

:no_bell: **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples).

* Simplify noxfile setup. [(#2806)](GoogleCloudPlatform/python-docs-samples#2806)

* chore(deps): update dependency requests to v2.23.0

* Simplify noxfile and add version control.

* Configure appengine/standard to only test Python 2.7.

* Update Kokokro configs to match noxfile.

* Add requirements-test to each folder.

* Remove Py2 versions from everything execept appengine/standard.

* Remove conftest.py.

* Remove appengine/standard/conftest.py

* Remove 'no-sucess-flaky-report' from pytest.ini.

* Add GAE SDK back to appengine/standard tests.

* Fix typo.

* Roll pytest to python 2 version.

* Add a bunch of testing requirements.

* Remove typo.

* Add appengine lib directory back in.

* Add some additional requirements.

* Fix issue with flake8 args.

* Even more requirements.

* Readd appengine conftest.py.

* Add a few more requirements.

* Even more Appengine requirements.

* Add webtest for appengine/standard/mailgun.

* Add some additional requirements.

* Add workaround for issue with mailjet-rest.

* Add responses for appengine/standard/mailjet.

Co-authored-by: Renovate Bot <bot@renovateapp.com>

* testing: mark some tests as flaky [(#3288)](GoogleCloudPlatform/python-docs-samples#3288)

fixes #3138

* [datalabeling] testing: wrap rpcs with backoff [(#3443)](GoogleCloudPlatform/python-docs-samples#3443)

* wrap all the rpcs with backoff
* add a shared testing lib
* remove flaky

* [datalabeling] fix: clean up old datasets before the test [(#3707)](GoogleCloudPlatform/python-docs-samples#3707)

fixes #3710 
fixes #3711

* [datalabeling] testing: retry upon ServerError [(#3762)](GoogleCloudPlatform/python-docs-samples#3762)

fixes #3760

* Replace GCLOUD_PROJECT with GOOGLE_CLOUD_PROJECT. [(#4022)](GoogleCloudPlatform/python-docs-samples#4022)

* chore(deps): update dependency pytest to v5.4.3 [(#4279)](GoogleCloudPlatform/python-docs-samples#4279)

* chore(deps): update dependency pytest to v5.4.3

* specify pytest for python 2 in appengine

Co-authored-by: Leah Cole <coleleah@google.com>

* Update dependency pytest to v6 [(#4390)](GoogleCloudPlatform/python-docs-samples#4390)

* chore: update templates

* chore: fix docs error

* chore: skip unavailable samples

* chore: use staging endpoint for labeling tests

Co-authored-by: Rebecca Taylor <remilytaylor@gmail.com>
Co-authored-by: Gus Class <gguuss@gmail.com>
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Co-authored-by: DPEBot <dpebot@google.com>
Co-authored-by: Doug Mahugh <dmahugh@gmail.com>
Co-authored-by: Noah Negrey <nnegrey@users.noreply.github.com>
Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
Co-authored-by: WhiteSource Renovate <bot@renovateapp.com>
Co-authored-by: Takashi Matsuo <tmatsuo@google.com>
Co-authored-by: Leah Cole <coleleah@google.com>
Co-authored-by: Bu Sun Kim <busunkim@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

5 participants