Reducing limit over the life of a datastore query iterator. #1787

dhermes · 2016-05-11T18:22:26Z

dhermes · 2016-05-11T18:22:35Z

Sorry for the delay @pcostell!

gcloud/datastore/query.py

@@ -463,6 +463,8 @@ def __iter__(self):
                yield entity
            if not self._more_results:
                break
+            if self._limit is not None:
+                self._limit -= len(self._page)
            self.next_page()


dhermes · 2016-05-12T23:52:17Z

@pcostell PTAL, I also added logic for the offset

gcloud/datastore/query.py

+            if self._limit is not None:
+                self._limit -= num_results
+            if self._offset is not None:
+                self._offset += num_results


dhermes · 2016-05-13T00:31:39Z

@pcostell You just mentioned that offset shouldn't be used to continue a query. You mean offset on it's own? Like without the end cursor?

gcloud/datastore/query.py

+            if self._offset is not None:
+                # NOTE: The offset goes down relative to the location
+                #       because we are updating the cursor each time.
+                self._offset -= num_results


pcostell · 2016-05-13T00:35:40Z

Sorry: When a query is continued, it may contain an offset. But like the limit the offset should always be decreasing. What I really mean:

Always use a cursor to step to the next batch of a query, never an offset.

Sometimes you need to go through batches of offset, in which you'll issue multiple queries with decreasing offsets. This is ok.

dhermes · 2016-05-17T17:13:45Z

@pcostell PTAL.

gcloud/datastore/query.py

@@ -382,7 +381,7 @@ class Iterator(object):
        _query_pb2.QueryResultBatch.MORE_RESULTS_AFTER_LIMIT,


dhermes · 2016-05-17T17:56:56Z

@tseaver I've got the LGTM from @pcostell. Can you give this a final pass?

gcloud/datastore/test_query.py

@@ -374,7 +382,7 @@ def test_ctor_defaults(self):
        iterator = self._makeOne(query, connection)
        self.assertTrue(iterator._query is query)
        self.assertEqual(iterator._limit, None)
-        self.assertEqual(iterator._offset, 0)
+        self.assertEqual(iterator._offset, None)


tseaver · 2016-05-17T20:56:38Z

Tests for next_page should assert correct values for self._skipped_results.

dhermes · 2016-05-17T21:51:00Z

@tseaver Good call. Check out the latest commit.

tseaver · 2016-05-17T23:43:16Z

LGTM

In the process, also keeping tracking of the number of skipped results (so that we can update the offset). Fixes googleapis#1763.

dhermes · 2016-05-17T23:52:08Z

Squashed commits. Will merge once Travis passes.

…thon-docs-samples#1787)

@nnegrey

* Add new "quickstart" samples [(#547)](GoogleCloudPlatform/python-docs-samples#547) * Quickstart tests [(#569)](GoogleCloudPlatform/python-docs-samples#569) * Add tests for quickstarts * Update secrets * Add translate samples [(#574)](GoogleCloudPlatform/python-docs-samples#574) * Generate most non-appengine readmes Change-Id: I3779282126cdd05b047194d356932b9995484115 * Update requirements.txt * added model in the sample [(#650)](GoogleCloudPlatform/python-docs-samples#650) * added with and without model separately [(#657)](GoogleCloudPlatform/python-docs-samples#657) * Translate API no longer requires an API key. [(#659)](GoogleCloudPlatform/python-docs-samples#659) * Auto-update dependencies. [(#715)](GoogleCloudPlatform/python-docs-samples#715) * Changes default model to premium [(#749)](GoogleCloudPlatform/python-docs-samples#749) * Updates readme template to show credential auth instead of api key. [(#802)](GoogleCloudPlatform/python-docs-samples#802) * Updates translate version to match speech and language. [(#843)](GoogleCloudPlatform/python-docs-samples#843) * Translate update [(#886)](GoogleCloudPlatform/python-docs-samples#886) * Remove cloud config fixture [(#887)](GoogleCloudPlatform/python-docs-samples#887) * Remove cloud config fixture * Fix client secrets * Fix bigtable instance * Unicode for translation * Adds check for translated text * Unicode for translation * Unicode for translation * Unicode for translation * missing newline * Adds six for 2/3 compatibility * Changes encoding style. * Fix unicode * renaming of the product to Google Translation API [(#910)](GoogleCloudPlatform/python-docs-samples#910) * Update translate readme * Re-generate all readmes * Fix README rst links [(#962)](GoogleCloudPlatform/python-docs-samples#962) * Fix README rst links * Update all READMEs * Auto-update dependencies. [(#1004)](GoogleCloudPlatform/python-docs-samples#1004) * Auto-update dependencies. * Fix natural language samples * Fix pubsub iam samples * Fix language samples * Fix bigquery samples * Auto-update dependencies. [(#1055)](GoogleCloudPlatform/python-docs-samples#1055) * Auto-update dependencies. * Explicitly use latest bigtable client Change-Id: Id71e9e768f020730e4ca9514a0d7ebaa794e7d9e * Revert language update for now Change-Id: I8867f154e9a5aae00d0047c9caf880e5e8f50c53 * Remove pdb. smh Change-Id: I5ff905fadc026eebbcd45512d4e76e003e3b2b43 * Auto-update dependencies. [(#1093)](GoogleCloudPlatform/python-docs-samples#1093) * Auto-update dependencies. * Fix storage notification poll sample Change-Id: I6afbc79d15e050531555e4c8e51066996717a0f3 * Fix spanner samples Change-Id: I40069222c60d57e8f3d3878167591af9130895cb * Drop coverage because it's not useful Change-Id: Iae399a7083d7866c3c7b9162d0de244fbff8b522 * Try again to fix flaky logging test Change-Id: I6225c074701970c17c426677ef1935bb6d7e36b4 * Update all generated readme auth instructions [(#1121)](GoogleCloudPlatform/python-docs-samples#1121) Change-Id: I03b5eaef8b17ac3dc3c0339fd2c7447bd3e11bd2 * Added Link to Python Setup Guide [(#1158)](GoogleCloudPlatform/python-docs-samples#1158) * Update Readme.rst to add Python setup guide As requested in b/64770713. This sample is linked in documentation https://cloud.google.com/bigtable/docs/scaling, and it would make more sense to update the guide here than in the documentation. * Update README.rst * Update README.rst * Update README.rst * Update README.rst * Update README.rst * Update install_deps.tmpl.rst * Updated readmegen scripts and re-generated related README files * Fixed the lint error * Auto-update dependencies. [(#1186)](GoogleCloudPlatform/python-docs-samples#1186) * Fixed failed tests on Kokoro (Spanner + Translate) [(#1192)](GoogleCloudPlatform/python-docs-samples#1192) * Fixed failed tests on Kokoro (Spanner + Translate) * Update quickstart_test.py * Added "Open in Cloud Shell" buttons to README files [(#1254)](GoogleCloudPlatform/python-docs-samples#1254) * Auto-update dependencies. [(#1377)](GoogleCloudPlatform/python-docs-samples#1377) * Auto-update dependencies. * Update requirements.txt * Regenerate the README files and fix the Open in Cloud Shell link for some samples [(#1441)](GoogleCloudPlatform/python-docs-samples#1441) * Update READMEs to fix numbering and add git clone [(#1464)](GoogleCloudPlatform/python-docs-samples#1464) * Add translate region tags [(#1488)](GoogleCloudPlatform/python-docs-samples#1488) * Add region tags * Added end region tags * Linting errors fixed * Include the comma in the translation [(#1787)](GoogleCloudPlatform/python-docs-samples#1787) * Auto-update dependencies. [(#1980)](GoogleCloudPlatform/python-docs-samples#1980) * Auto-update dependencies. * Update requirements.txt * Update requirements.txt * Translation v3beta1 samples [(#2084)](GoogleCloudPlatform/python-docs-samples#2084) * Add in progress beta snippets Change-Id: I2cd8ddc2307a8e40d56ce7e493749dc05c34d164 * Add google-cloud-storage dependency Change-Id: Iff7bc9b2c82b1e829580a3d4ad628087dbeee8be * Non-'global' location required for BatchTranslateText Change-Id: I5198aa6368a088e8f5ee295dc55a5e9e4ca8f494 * Upgrade google-cloud-translate to 1.4.0 1.4.0 includes the new v3beta1 alongside V2 Change-Id: I5adfe78ea7e78d84678db343cd84516e3d05491f * Update Translate samples You can now provide your own glossary ID The tests now run within a randomly created bucket (deleted after each test) Change-Id: I5cb2680cd0e9e43c85932a6a0dc19e6fab5008c5 * pytest.fixture for random test bucket Change-Id: I8e816ed4c95a6235347a29849044b4cab02d40a8 * flake8 spec fixes Change-Id: I4932bcf856a9498b01d9661c90c6b45ee2958ee1 * Added pytest fixture for creating glossary (WIP) Change-Id: Iddb5ecbf0eefb9efd2243dc4bc56b585102e9351 * Add assertions, remove placeholder TODOs Change-Id: If1eb20bca5bfcc87dd0652d5488b2188afa626af * fixing translate-with-glossary bug [(#2323)](GoogleCloudPlatform/python-docs-samples#2323) * Translate beta samples fix [(#2327)](GoogleCloudPlatform/python-docs-samples#2327) * fixing translate-with-glossary bug * tests passing * reverting to python3 compatibility * snippets test fix * Using glossaries with tts and vision tutorial sample code [(#2325)](GoogleCloudPlatform/python-docs-samples#2325) * fixing translate-with-glossary bug * initial commit * adding resources * adding more resources * glossary accomodates upper case words * finished hybrid glossaries tutorial sample code * Revert "fixing translate-with-glossary bug" This reverts commit 6a9f7ca3f68239a862106fcbcd9c73649ce36c77. * lint fix for tests. TODO src lint fix * lint * it's the final lint-down * adding README * implementing @nnegrey's feedback * lint * lint * extracting files from cloud-client * lint comment test * fixing comments per @beccasaurus * removing redundant directory * implementing @nnegrey's feedback * lint * lint * handling glossary-already-exists exception * lint * adding ssml functionality * fixing imports per @nnegrey * fixed import comment * more specific exceptions import * removing period from copyright * fix: refactored MP3 file creation test for Hybrid glossaries samples [(#2379)](GoogleCloudPlatform/python-docs-samples#2379) * fix: refactored MP3 file creation test * fix: lint * Fix variable names in comments [(#2400)](GoogleCloudPlatform/python-docs-samples#2400) * Adds updates for samples profiler ... vision [(#2439)](GoogleCloudPlatform/python-docs-samples#2439) * Update v2 samples to explicitly use v2 library [(#2498)](GoogleCloudPlatform/python-docs-samples#2498) * fix: translate test [(#2671)](GoogleCloudPlatform/python-docs-samples#2671) * fix: translate test * Add unicode formatting * automl: add natural language sentiment analysis ga samples [(#2677)](GoogleCloudPlatform/python-docs-samples#2677) * automl: add natural language sentiment analysis ga samples * Add links to documentation * Update tests to use centralized project * Fix environment variable, make translate test less flaky * fix region tag typo [(#2731)](GoogleCloudPlatform/python-docs-samples#2731) * Migrate published samples [(#2759)](GoogleCloudPlatform/python-docs-samples#2759) Migrate from tmp-generated-samples branch 615c08e Remove boilerplate Update copyright date Blacken Remove unused imports Shorten docstrings Remove CLI Set defaults in function definition Add link to supported types guide Inline function arguments Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * translate v3 snippets [(#2745)](GoogleCloudPlatform/python-docs-samples#2745) * translate text v3 * added translate text with glossary snippets * finished glossary tests * removed overlapping files * added encoding tag * added more descriptive docs and broke down tests * Update translate/cloud-client/translate_v3_create_glossary.py Co-Authored-By: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> * Update translate/cloud-client/translate_v3_create_glossary.py Co-Authored-By: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> * fixed test for translate with glossary * fixed lint Co-authored-by: Michelle Casbon <texasmichelle@users.noreply.github.com> Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> * Translate: migrate published samples [(#2768)](GoogleCloudPlatform/python-docs-samples#2768) Migrate from tmp-generated-samples branch 615c08e Remove boilerplate Update copyright date Blacken Remove unused imports * Translate: migrate published glossaries samples [(#2769)](GoogleCloudPlatform/python-docs-samples#2769) Migrate from tmp-generated-samples branch fef998b Remove boilerplate Update copyright date Blacken Remove unused imports Shorten docstrings Remove CLI * translate: increase timeout [(#2937)](GoogleCloudPlatform/python-docs-samples#2937) * Translate: migrate published v3 translate batch samples [(#2914)](GoogleCloudPlatform/python-docs-samples#2914) * Translate: migrate published b v3 tch samples * added missing requirements * extended wait time * inlined some vals and specified input and output * added link to supported file types & modified default values of input uri * fixed small nit * chore(deps): update dependency google-cloud-storage to v1.26.0 [(#3046)](GoogleCloudPlatform/python-docs-samples#3046) * chore(deps): update dependency google-cloud-storage to v1.26.0 * chore(deps): specify dependencies by python version * chore: up other deps to try to remove errors Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Leah Cole <coleleah@google.com> * chore(deps): update dependency google-cloud-translate to v1.7.0 [(#3084)](GoogleCloudPlatform/python-docs-samples#3084) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-translate](https://togithub.com/googleapis/python-translate) | minor | `==1.6.0` -> `==1.7.0` | --- ### Release Notes <details> <summary>googleapis/python-translate</summary> ### [`v1.7.0`](https://togithub.com/googleapis/python-translate/blob/master/CHANGELOG.md#170) [Compare Source](https://togithub.com/googleapis/python-translate/compare/v1.6.0...v1.7.0) 10-07-2019 14:57 PDT ##### Implementation Changes - Update docstrings, client confg (via synth). ([#9411](https://togithub.com/googleapis/google-cloud-python/pull/9411)) - Remove send / receive message size limit (via synth). ([#8974](https://togithub.com/googleapis/google-cloud-python/pull/8974)) ##### New Features - Add support for V3 of the API. ([#9020](https://togithub.com/googleapis/google-cloud-python/pull/9020)) - Make `parent` argument required for all client methods in v3beta1; add `labels` argument (via synth). ([#9354](https://togithub.com/googleapis/google-cloud-python/pull/9354)) - Add client options to translate_v2. ([#8737](https://togithub.com/googleapis/google-cloud-python/pull/8737)) ##### Dependencies - Bump minimum version for google-api-core to 1.14.0. ([#8709](https://togithub.com/googleapis/google-cloud-python/pull/8709)) ##### Documentation - Fix links to reference documentation. ([#8884](https://togithub.com/googleapis/google-cloud-python/pull/8884)) - Link to googleapis.dev documentation in READMEs. ([#8705](https://togithub.com/googleapis/google-cloud-python/pull/8705)) ##### Internal / Testing Changes - Update `ListGlossaries` method annotation (via synth) ([#9385](https://togithub.com/googleapis/google-cloud-python/pull/9385)) - Fix intersphinx reference to requests ([#9294](https://togithub.com/googleapis/google-cloud-python/pull/9294)) - Remove CI for gh-pages, use googleapis.dev for api_core refs. ([#9085](https://togithub.com/googleapis/google-cloud-python/pull/9085)) - Update intersphinx mapping for requests. ([#8805](https://togithub.com/googleapis/google-cloud-python/pull/8805)) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * translate: fix tests [(#3087)](GoogleCloudPlatform/python-docs-samples#3087) Fix: GoogleCloudPlatform/python-docs-samples#3018 * translate-v3: samples [(#3034)](GoogleCloudPlatform/python-docs-samples#3034) * translate with custom model, get supported langs * inlined small nit * added encoding to model test * added missing region tags and link to supported langs * inlined text-to-translate * directly inlined contents * revert text-translate vars * reversed inlined text params * small nit Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency google-cloud-vision to v0.42.0 [(#3170)](GoogleCloudPlatform/python-docs-samples#3170) * chore(deps): update dependency google-cloud-texttospeech to v1 [(#3210)](GoogleCloudPlatform/python-docs-samples#3210) Co-authored-by: gcf-merge-on-green[bot] <60162190+gcf-merge-on-green[bot]@users.noreply.github.com> * chore(deps): update dependency google-cloud-translate to v2 [(#3211)](GoogleCloudPlatform/python-docs-samples#3211) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-translate](https://togithub.com/googleapis/python-translate) | major | `==1.7.0` -> `==2.0.1` | --- ### Release Notes <details> <summary>googleapis/python-translate</summary> ### [`v2.0.1`](https://togithub.com/googleapis/python-translate/blob/master/CHANGELOG.md#201-httpswwwgithubcomgoogleapispython-translatecomparev200v201-2020-01-31) [Compare Source](https://togithub.com/googleapis/python-translate/compare/v2.0.0...v2.0.1) ### [`v2.0.0`](https://togithub.com/googleapis/python-translate/blob/master/CHANGELOG.md#200) [Compare Source](https://togithub.com/googleapis/python-translate/compare/v1.7.0...v2.0.0) 10-23-2019 11:13 PDT ##### New Features - Make v3 the default client. ([#9498](https://togithub.com/googleapis/google-cloud-python/pull/9498)) ##### Internal / Testing Changes - Add VPC-SC system tests. ([#9272](https://togithub.com/googleapis/google-cloud-python/pull/9272)) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Simplify noxfile setup. [(#2806)](GoogleCloudPlatform/python-docs-samples#2806) * chore(deps): update dependency requests to v2.23.0 * Simplify noxfile and add version control. * Configure appengine/standard to only test Python 2.7. * Update Kokokro configs to match noxfile. * Add requirements-test to each folder. * Remove Py2 versions from everything execept appengine/standard. * Remove conftest.py. * Remove appengine/standard/conftest.py * Remove 'no-sucess-flaky-report' from pytest.ini. * Add GAE SDK back to appengine/standard tests. * Fix typo. * Roll pytest to python 2 version. * Add a bunch of testing requirements. * Remove typo. * Add appengine lib directory back in. * Add some additional requirements. * Fix issue with flake8 args. * Even more requirements. * Readd appengine conftest.py. * Add a few more requirements. * Even more Appengine requirements. * Add webtest for appengine/standard/mailgun. * Add some additional requirements. * Add workaround for issue with mailjet-rest. * Add responses for appengine/standard/mailjet. Co-authored-by: Renovate Bot <bot@renovateapp.com> * Update dependency google-cloud-vision to v1 [(#3227)](GoogleCloudPlatform/python-docs-samples#3227) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-vision](https://togithub.com/googleapis/python-vision) | major | `==0.42.0` -> `==1.0.0` | --- ### Release Notes <details> <summary>googleapis/python-vision</summary> ### [`v1.0.0`](https://togithub.com/googleapis/python-vision/blob/master/CHANGELOG.md#100-httpswwwgithubcomgoogleapispython-visioncomparev0420v100-2020-02-28) [Compare Source](https://togithub.com/googleapis/python-vision/compare/v0.42.0...v1.0.0) ##### Features - bump release status to GA ([#11](https://www.github.com/googleapis/python-vision/issues/11)) ([2129bde](https://www.github.com/googleapis/python-vision/commit/2129bdedfa0dca85c5adc5350bff10d4a485df77)) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * translate: make test generic [(#3386)](GoogleCloudPlatform/python-docs-samples#3386) Fix: GoogleCloudPlatform/python-docs-samples#3381 * [translate] fix: mark some tests as flaky [(#3495)](GoogleCloudPlatform/python-docs-samples#3495) tests which involve LRO. fixes #2875 * testing: replace @flaky with @pytest.mark.flaky [(#3496)](GoogleCloudPlatform/python-docs-samples#3496) * testing: replace @flaky with @pytest.mark.flaky * lint * mark few tests as flaky that involves LRO polling. * lint * chore(deps): update dependency google-cloud-storage to v1.28.0 [(#3260)](GoogleCloudPlatform/python-docs-samples#3260) Co-authored-by: Takashi Matsuo <tmatsuo@google.com> * translate: fix glossary leak issue [(#3572)](GoogleCloudPlatform/python-docs-samples#3572) * fix glossary leak issue * removed try/catch from teardown methods, removed sample_ prefix from all other methods * added specific exceptions to tests, added backoff tags to tests * fixed the lint issues * reordered imports * moved backoff inside methd and removed Retry * corrected import nit * chore: some lint fixes [(#3751)](GoogleCloudPlatform/python-docs-samples#3751) * chore: some lint fixes * longer timeout, more retries * disable detect_test.py::test_async_detect_document * chore(deps): update dependency google-cloud-storage to v1.28.1 [(#3785)](GoogleCloudPlatform/python-docs-samples#3785) * chore(deps): update dependency google-cloud-storage to v1.28.1 * [asset] testing: use uuid instead of time Co-authored-by: Takashi Matsuo <tmatsuo@google.com> * [translate] fix hybrid glossaries tests [(#3936)](GoogleCloudPlatform/python-docs-samples#3936) * testing: start using btlr [(#3959)](GoogleCloudPlatform/python-docs-samples#3959) * testing: start using btlr The binary is at gs://cloud-devrel-kokoro-resources/btlr/v0.0.1/btlr * add period after DIFF_FROM * use array for btlr args * fix websocket tests * add debug message * wait longer for the server to spin up * dlp: bump the wait timeout to 10 minutes * [run] copy noxfile.py to child directory to avoid gcloud issue * [iam] fix: only display description when the key exists * use uuid4 instead of uuid1 * [iot] testing: use the same format for registry id * Stop asserting Out of memory not in the output * fix missing imports * [dns] testing: more retries with delay * [dlp] testing: longer timeout * use the max-concurrency flag * use 30 workers * [monitoring] use multiple projects * [dlp] testing: longer timeout * Replace GCLOUD_PROJECT with GOOGLE_CLOUD_PROJECT. [(#4022)](GoogleCloudPlatform/python-docs-samples#4022) * remove whitelist replace with allowlist [(#4050)](GoogleCloudPlatform/python-docs-samples#4050) * chore(deps): update dependency google-cloud-storage to v1.29.0 [(#4040)](GoogleCloudPlatform/python-docs-samples#4040) * chore(deps): update dependency google-cloud-texttospeech to v2.1.0 [(#4147)](GoogleCloudPlatform/python-docs-samples#4147) * testing(translate): parameterize the timeout [(#4247)](GoogleCloudPlatform/python-docs-samples#4247) fixes #4239 (by specifying a longer timeout) * chore(deps): update dependency pytest to v5.4.3 [(#4279)](GoogleCloudPlatform/python-docs-samples#4279) * chore(deps): update dependency pytest to v5.4.3 * specify pytest for python 2 in appengine Co-authored-by: Leah Cole <coleleah@google.com> * Update dependency flaky to v3.7.0 [(#4300)](GoogleCloudPlatform/python-docs-samples#4300) * testing(translate): bump the timeout for operations [(#4258)](GoogleCloudPlatform/python-docs-samples#4258) fixes #4220 * chore: update templates * chore: narrows samples CODEOWNERS to .py only * chore: wip migration to microgenerator client, units, docs complete * feat!: move API to python microgenerator * docs: readmegen updates * chore: add build config for docs-presubmit * chore: rm protos * chore: uses PROJECT_ID env var in system test * chore: clarifies examples in migration guide * chore: adds explicit variable Co-authored-by: Jason Dobry <jmdobry@users.noreply.github.com> Co-authored-by: Jon Wayne Parrott <jonwayne@google.com> Co-authored-by: Puneith Kaul <puneith@users.noreply.github.com> Co-authored-by: DPE bot <dpebot@google.com> Co-authored-by: Gus Class <gguuss@gmail.com> Co-authored-by: florencep <florenceperot@google.com> Co-authored-by: Bill Prin <waprin@gmail.com> Co-authored-by: michaelawyu <chenyumic@google.com> Co-authored-by: Frank Natividad <frankyn@users.noreply.github.com> Co-authored-by: Averi Kitsch <akitsch@google.com> Co-authored-by: Charles Engelke <github@engelke.com> Co-authored-by: Rebecca Taylor <remilytaylor@gmail.com> Co-authored-by: Elizabeth Crowdus <elcrowdus@gmail.com> Co-authored-by: Noah Negrey <nnegrey@users.noreply.github.com> Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Michelle Casbon <texasmichelle@users.noreply.github.com> Co-authored-by: Mike <45373284+munkhuushmgl@users.noreply.github.com> Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: WhiteSource Renovate <bot@renovateapp.com> Co-authored-by: Leah Cole <coleleah@google.com> Co-authored-by: gcf-merge-on-green[bot] <60162190+gcf-merge-on-green[bot]@users.noreply.github.com> Co-authored-by: Takashi Matsuo <tmatsuo@google.com> Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>

dhermes assigned tseaver May 11, 2016

dhermes added the api: datastore Issues related to the Datastore API. label May 11, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label May 11, 2016

pcostell reviewed May 11, 2016
View reviewed changes

gcloud/datastore/query.py

@@ -463,6 +463,8 @@ def __iter__(self):

yield entity

if not self._more_results:

break

if self._limit is not None:

self._limit -= len(self._page)

self.next_page()

This comment was marked as spam.

Sign in to view

dhermes force-pushed the fix-1763 branch from 1ad2799 to 49d9503 Compare May 12, 2016 23:51

pcostell reviewed May 13, 2016
View reviewed changes

gcloud/datastore/query.py

if self._offset is not None:

# NOTE: The offset goes down relative to the location

# because we are updating the cursor each time.

self._offset -= num_results

This comment was marked as spam.

Sign in to view

dhermes force-pushed the fix-1763 branch from 4d853dc to 3c0b3e1 Compare May 17, 2016 17:13

pcostell reviewed May 17, 2016
View reviewed changes

gcloud/datastore/query.py

@@ -382,7 +381,7 @@ class Iterator(object):

_query_pb2.QueryResultBatch.MORE_RESULTS_AFTER_LIMIT,

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

tseaver reviewed May 17, 2016
View reviewed changes

dhermes force-pushed the fix-1763 branch from 5c389c9 to a7332dd Compare May 17, 2016 21:50

dhermes force-pushed the fix-1763 branch from a7332dd to a03a95e Compare May 17, 2016 23:49

Reducing limit and offset over the life of a datastore query iterator.

74026f4

In the process, also keeping tracking of the number of skipped results (so that we can update the offset). Fixes googleapis#1763.

dhermes force-pushed the fix-1763 branch from a03a95e to 74026f4 Compare May 17, 2016 23:51

dhermes merged commit d956136 into googleapis:master May 18, 2016

dhermes deleted the fix-1763 branch May 18, 2016 00:19

dhermes mentioned this pull request Jun 7, 2016

Cut 0.15.0 release. #1842

Merged

dhermes changed the title ~~Reducing limit or the life of a datastore query iterator.~~ Reducing limit over the life of a datastore query iterator. Jun 7, 2016

dhermes mentioned this pull request Oct 3, 2016

bucket.list_blobs(max_results=n) has no effect #1467

Closed

parthea pushed a commit that referenced this pull request Oct 21, 2023

Include the comma in the translation [(#1787)](GoogleCloudPlatform/py…

8a8fc16

…thon-docs-samples#1787)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reducing limit over the life of a datastore query iterator. #1787

Reducing limit over the life of a datastore query iterator. #1787

dhermes commented May 11, 2016

dhermes commented May 11, 2016

This comment was marked as spam.

dhermes commented May 12, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

dhermes commented May 13, 2016

This comment was marked as spam.

pcostell commented May 13, 2016

dhermes commented May 17, 2016

This comment was marked as spam.

This comment was marked as spam.

dhermes commented May 17, 2016

This comment was marked as spam.

tseaver commented May 17, 2016

dhermes commented May 17, 2016

tseaver commented May 17, 2016

dhermes commented May 17, 2016

		@@ -382,7 +381,7 @@ class Iterator(object):
		_query_pb2.QueryResultBatch.MORE_RESULTS_AFTER_LIMIT,

Reducing limit over the life of a datastore query iterator. #1787

Reducing limit over the life of a datastore query iterator. #1787

Conversation

dhermes commented May 11, 2016

dhermes commented May 11, 2016

This comment was marked as spam.

dhermes commented May 12, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

dhermes commented May 13, 2016

This comment was marked as spam.

pcostell commented May 13, 2016

dhermes commented May 17, 2016

This comment was marked as spam.

This comment was marked as spam.

dhermes commented May 17, 2016

This comment was marked as spam.

tseaver commented May 17, 2016

dhermes commented May 17, 2016

tseaver commented May 17, 2016

dhermes commented May 17, 2016