Skip to content

Conversation

@dhermes
Copy link
Contributor

@dhermes dhermes commented Dec 23, 2014

Addresses sixth part of #451.

NOTE: This has #461 as a diffbase.

@coveralls
Copy link

Coverage Status

Coverage remained the same when pulling 031b864 on dhermes:fix-451-part6 into b343acf on GoogleCloudPlatform:master.

@tseaver
Copy link
Contributor

tseaver commented Dec 30, 2014

031b864 comments inline on the changeset.

@dhermes
Copy link
Contributor Author

dhermes commented Dec 30, 2014

RE: "Why not pass self._data". That was creep from the killed #445. I rebased and got rid of it.

RE: "Calling Entity.key() twice in Entity.save()". I was trying to use the public API as often as possible. I think we should remove the key() / _key distinction and just make key an attribute.

RE: "Replicating code from helpers.key_from_protobuf". See my comment in the parent issue.

RE: "Server completion of a partial key". ISTM the server will always complete with an ID, hence the name auto_id. @pcostell can you confirm?

@coveralls
Copy link

Coverage Status

Coverage remained the same when pulling 1778430 on dhermes:fix-451-part6 into 8cb41f6 on GoogleCloudPlatform:master.

@dhermes
Copy link
Contributor Author

dhermes commented Dec 30, 2014

As I look at this more I wonder why Connection.save_entity returns the whole Key pb when the backend assigns an auto ID. Why not just return the assigned ID? (Then this whole PR changes, compare_to_proto doesn't need to be implemented.)

In some sense, we want to check that the entire returned key matches but in another sense, we should trust that the backend has preserved everything except the missing final ID.

@tseaver WDYT?

@tseaver
Copy link
Contributor

tseaver commented Dec 31, 2014

I'd prefer not to return the protobuf, and I really don't like the "type variant" return signature. Could we have Connection.save_entity return a tuple, (assigned, new_id). E.g.:

assigned, new_id = connection.save_entity(ds_id, key_pb, properties)
if assigned:
    entity.key = entity.key.complete_key(new_id)

@dhermes
Copy link
Contributor Author

dhermes commented Dec 31, 2014

Good call. On it.

@tseaver
Copy link
Contributor

tseaver commented Dec 31, 2014

Travis failure: rename to completed_key was incomplete. :)

Making Connection.save_entity return the auto allocated ID
instead of the entire PB.
@dhermes
Copy link
Contributor Author

dhermes commented Dec 31, 2014

Removed Key.compare_to_proto and made Connection.save_entity have a consistent return type (if you consider None to be valid as type int).

@tseaver PTAL

@coveralls
Copy link

Coverage Status

Coverage remained the same when pulling a257a9c on dhermes:fix-451-part6 into f0e26b8 on GoogleCloudPlatform:master.

tseaver added a commit that referenced this pull request Dec 31, 2014
Address sixth part of 451: Implement Key.compare_to_proto to check pb keys against existing.
@tseaver tseaver merged commit e86ba09 into googleapis:master Dec 31, 2014
@dhermes dhermes deleted the fix-451-part6 branch December 31, 2014 22:05
@pcostell
Copy link
Contributor

pcostell commented Jan 4, 2015

@dhermes the server will always complete keys when the entity is part of insert_with_auto_id. Otherwise, it will not complete it. If something bad happens the server will throw an error (rather than return an incomplete key).

@dhermes dhermes added the api: datastore Issues related to the Datastore API. label Dec 31, 2015
atulep pushed a commit that referenced this pull request Apr 6, 2023
Co-authored-by: AJ Morozoff <amorozoff@google.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
atulep pushed a commit that referenced this pull request Apr 6, 2023
Co-authored-by: AJ Morozoff <amorozoff@google.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
atulep pushed a commit that referenced this pull request Apr 18, 2023
Co-authored-by: AJ Morozoff <amorozoff@google.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea added a commit that referenced this pull request Aug 15, 2023
* fix(deps): allow protobuf 3.19.5

* explicitly exclude protobuf 4.21.0
vchudnov-g pushed a commit that referenced this pull request Sep 20, 2023
Source-Link: googleapis/synthtool@4760d8d
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:f0e4b51deef56bed74d3e2359c583fc104a8d6367da3984fc5c66938db738828

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Sep 22, 2023
Source-Link: https://togithub.com/googleapis/synthtool/commit/30bd01b4ab78bf1b2a425816e15b3e7e090993dd
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:9bc5fa3b62b091f60614c08a7fb4fd1d3e1678e326f34dd66ce1eefb5dc3267b
parthea pushed a commit that referenced this pull request Sep 22, 2023
…[autoapprove] (#462)

Source-Link: googleapis/synthtool@1f37ce7
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:8e84e0e0d71a0d681668461bba02c9e1394c785f31a10ae3470660235b673086
parthea added a commit that referenced this pull request Sep 22, 2023
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this pull request Oct 21, 2023
…eIamPolicies API (#462)

* feat: Add client library support for AssetService v1 BatchGetEffectiveIamPolicies API
Committer: haochunzhang@

PiperOrigin-RevId: 466134014

Source-Link: googleapis/googleapis@63c73fb

Source-Link: googleapis/googleapis-gen@2350945
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMjM1MDk0NWY3YTcwZWNhYWVjZjlhMWZkZDdkNmU3MGFjNTBlODYyZCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Oct 21, 2023
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
parthea added a commit that referenced this pull request Oct 22, 2023
Co-authored-by: AJ Morozoff <amorozoff@google.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this pull request Nov 24, 2025
* fix: Major refactoring and fix for Polling, Retry and Timeout logic

This is in response to https://freeman.vc/notes/aws-vs-gcp-reliability-is-wildly-different, which triggered an investigation of the whole Polling/Retry/Timeout behavior in Python GAPIC clients and revealed many fundamental flaws in its implementaiton.

To properly describe the refactoring this PR does we need to stick to a rigorous terminology, as vague definitions of retries, timeouts, polling and related concepts seems to be the main source of the present bugs and overal confusion among both groups: users of the library and creators of the library. Please check the documentation of the `google.api_core.retry.Retry` class the `google.api_core.future.polling.Polling.result()` method for the proper definitions and context.

Note, the overall semantics around Polling, Retry and Timeout remains quite confusing even after refactoring (although it is now more or less rigorously defined), but it was clean as I could make it while still maintaining backward compatibility of the whole library.

The quick summary of the changes in this PR:

1) Properly define and fix the application of Deadline and Timeout concepts. Please check the updated documentation for the `google.api_core.retry.Retry` class for the actual definitions. Originally the `deadline` has been used to represent timeouts conflating the two concepts. As result this PR replaces `deadline` arguments with `timeout` ones in as backward-compatible manner as possible (i.e. backward compatible in all practical applications).

2) Properly define RPC Timeout, Retry Timeout and Pollint Timeout and how a generic Timeout concept (aka Logical Timeout) is mapped to one of those depending on the context. Please check `google.api_core.retry.Retry` class documentation for details.

3) Properly define and fix the application of Retry and Polling concepts. Please check the updated documentation for `google.api_core.future.polling.PollingFuture.result()` for details.

4) Separate `retry` and `polling` configurations for Polling future, as these are two different concepts (although both operating on `Retry` class). Originally both retry and polling configurations were controlled by a single `retry` parameter, merging configuration regarding how "rpc error responses" and how "operation not completed" responses are supposed to be handled.

5) For the following config properties - `Retry (including `Retry Timeout`), `Polling` (including `Polling Timeout`) and `RPC Timeout` - fix and properly define how each of the above properties gets configured and which config gets precedence in case of a conflict (check `PollingFuture.result()` method documentation for details). Each of those properties can be specified as follows: directly provided by the user for each call, specified during gapic generation time from config values in `grpc_service_config.json` file (for Retry and RPC Timeout) and `gapic.yaml` file (for Polling), or be provided as a hard-coded basic default values in python-api-core library itself.

6) Fix the per-call polling config propagation logic (the polling/retry configs supplied to `PollingFuture.result()` used to be ignored for actual call).

7) Deprecate the usage of `deadline` terminology in the whole library and backward-compatibly replace it with timeout. This is essential as what has been called "deadline" in this library was actually "timeout" as it is defined in `google.api_core.retry.Retry` class documentation.

8) Deprecate `ExponentialTimeout`, `ConstantTimeout` and related logic as those are outdated concepts and are not consistent with the other GAPIC Languages. Replace it with `TimeToDeadlineTimeout` to be consistent with how the rest of the languages do it.

9) Deprecate `google.api_core.operations_v1.config` as it is an outdated concept and self-inconsistent (as all gapic clients provide configuraiton in code). The configs are directly provided in code instead.

10) Switch randomized delay calculation from `delay` being treated as expected value for randomized_delay to `delay` being treated as maximum value for `randomized_delay` (i.e. the new expected valud for `randomized_delay` is `delay / 2`). See the `exponential_sleep_generator()` method implementation for details. This is needed to make Python implementation of retries and polling exponential backoff consistent with the rest of GAPIC languages. Also fix the uncontrollable growth of `delay` value (since it is a subject of exponential growth, the `delay` value was quickly reaching "infinity" value, and the whole thing was not failing simply due to python being a very forgiving language which forgives multiplying "infinity" by a number (`inf * number = inf`) binstead of simply overflowing to a (most likely) negative number).

11) Fix url construction in `OperationsRestTransport`. Without this fix the polling logic for REST transport was completely broken (is not affecting Compute client, as that one has custom LRO).

12) Las but not least: change the default values for Polling logic to be the following: `initial=1.0` (same as before), `maximum=20.0` (was `60`), `multiplier=1.5` (was `2.0`), `timeout=900` (was `120`, but due to timeout resolution logic was actually None (i.e. infinity)). This, in conjunction with changed calculation of randomized delay (i.e. its expected value now being  `delay / 2`) overall makes polling logic much less aggressive in terms of increasing delays between each polling iteration, making LRO return much earlier for users on average, but still keeping a healthy balance between strain put on both client and server by polling and responsiveness of LROs for user.

*The design doc summarising all the changes and reasons for them is in progress.

* fix ci failures (mainly sphinx errors)

* remove unused code

* fix typo

* Pin pytest version to <7.2.0

* reformat code

* address pr feedback

* address PR feedback

* address pr feedback

* Update google/api_core/future/polling.py

Co-authored-by: Victor Chudnovsky <vchudnov@google.com>

* Apply documentation suggestions from code review

Co-authored-by: Victor Chudnovsky <vchudnov@google.com>

* Address PR feedback

Co-authored-by: Victor Chudnovsky <vchudnov@google.com>
parthea added a commit that referenced this pull request Nov 24, 2025
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea added a commit that referenced this pull request Nov 24, 2025
* chore: Update gapic-generator-python to v1.12.0

PiperOrigin-RevId: 586356061

Source-Link: googleapis/googleapis@72a1f55

Source-Link: googleapis/googleapis-gen@558a04b
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNTU4YTA0YmNkMWNjMDU3NmU4ZmFjMTA4OWU0OGU0OGIyN2FjMTYxYiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* update required checks

* feat: added Generator API
docs: updated doc for speech mode

PiperOrigin-RevId: 586469693

Source-Link: googleapis/googleapis@e8148d6

Source-Link: googleapis/googleapis-gen@85136bd
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiODUxMzZiZDA0MzgzZWQ3MTcyYmIxOGI3YjhkMjIwZGQ3ZmY2YjNhMCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this pull request Nov 26, 2025
feat: fetch id token from GCE metadata server
parthea pushed a commit that referenced this pull request Nov 26, 2025
🤖 I have created a release \*beep\* \*boop\* 
---
## [1.12.0](https://www.github.com/googleapis/google-auth-library-python/compare/v1.11.3...v1.12.0) (2020-03-25)


### Features

* add mTLS ADC support for HTTP ([#457](https://www.github.com/googleapis/google-auth-library-python/issues/457)) ([bb9215a](https://www.github.com/googleapis/google-auth-library-python/commit/bb9215ad6dee6c1dc7f255a2e4ed7011b85bd6cf))
* add SslCredentials class for mTLS ADC ([#448](https://www.github.com/googleapis/google-auth-library-python/issues/448)) ([dafb41f](https://www.github.com/googleapis/google-auth-library-python/commit/dafb41fae3f513ea9a4f93404f6148bee7dda202))
* fetch id token from GCE metadata server ([#462](https://www.github.com/googleapis/google-auth-library-python/issues/462)) ([97e7700](https://www.github.com/googleapis/google-auth-library-python/commit/97e7700da031bfd80b63b1a3d2abc29c500936ef))


### Bug Fixes

* don't use threads for gRPC AuthMetadataPlugin ([#467](https://www.github.com/googleapis/google-auth-library-python/issues/467)) ([ee373f8](https://www.github.com/googleapis/google-auth-library-python/commit/ee373f88b512a38e791a1c085452c6c6da501eb6))
* make ThreadPoolExecutor a class var ([#461](https://www.github.com/googleapis/google-auth-library-python/issues/461)) ([b526473](https://www.github.com/googleapis/google-auth-library-python/commit/b5264730603947295cc97ecff2f6aef84aa3d6e9))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: datastore Issues related to the Datastore API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants