Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Makes operator checking deterministic in Query.filter. #294

Merged

Conversation

dhermes
Copy link
Contributor

@dhermes dhermes commented Oct 24, 2014

The order of dictionary keys is not guaranteed between runs of
code so expressions ending in '>=' or '<=' were sometimes
matching for '='.

@pcostell the issue with flakiness was non-deterministic behavior of Python internals. Yay for finding the bug!

This was holding up #281, but it also forced us to use ancestor queries, which is a good thing.

@tseaver This is the quick-and-dirtiest way to fix this issue but I am a fan of changing from

query.filter('foo >=', 10)

to

query.filter('foo', '>=', 10)

WDYT?

Also @jgeewax WDYT since your original design. (Maybe you had reasons for combination property name and operator?)

@coveralls
Copy link

Coverage Status

Coverage remained the same when pulling af1a39f on dhermes:query-operator-fix-dict-random-order into b6d3e74 on GoogleCloudPlatform:master.

@jgeewax
Copy link
Contributor

jgeewax commented Oct 25, 2014

I'm -1 to making it filter(field, operator, value) -- the reason being that this is how google.appengine.ext.db/ndb handled filtering (I just modeled it after them).

Another option to fix this is to use a regex:

I'd also feel much better if there were failing test to reproduce this as it's kind of a tricky bug... Any chance you could toss one in? Or add a ticket to make sure we have one to avoid regressing?

@dhermes
Copy link
Contributor Author

dhermes commented Oct 25, 2014

@jgeewax RE: emulating db, this library is more low-level. If/when we build something more high-level, (for me) the "right" way to do this is the ndb way

qry = Account.query(Account.userid == 42)

I would also guess that they were like that in db for historical reasons (i.e. a different proto def).

As for a regression test, #281 is what made us discover this bug in the first place and I just commited fc23837.

@coveralls
Copy link

Coverage Status

Coverage remained the same when pulling fc23837 on dhermes:query-operator-fix-dict-random-order into b6d3e74 on GoogleCloudPlatform:master.

@jgeewax
Copy link
Contributor

jgeewax commented Oct 25, 2014

👍 -- I think for this library, filter('field >=' value) is the right abstraction.

For a higher-level library (maybe we'll even pull ndb over ...) I absolutely agree with you that the ndb way is ideal (for example, it's far nicer than the Django ORM way..)

@Alfus
Copy link

Alfus commented Oct 27, 2014

I think filter('field', op, value) is technically better but more cumbersome. As long as filter('field', value) is not valid and filter('field with spaces =', value) is valid I think this function signature is workable (though it likely does not support property names with trailing white spaces).

Another option that doesn't require a full ORM is just adding a Property class with operator overloading: filter(Property('field') >= value)

@dhermes
Copy link
Contributor Author

dhermes commented Oct 27, 2014

@Alfus that is on the roadmap, but as part of a higher-level abstraction than this one. Here we essentially want to mimic the API. In that vein, the .proto def has three fields rather than two, but I have no strong preference.

@dhermes dhermes added api: datastore Issues related to the Datastore API. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Oct 27, 2014
The order of dictionary keys is not guaranteed between runs of
code so `expression`s ending in '>=' or '<=' were sometimes
matching for '='.
@dhermes dhermes force-pushed the query-operator-fix-dict-random-order branch from fc23837 to 799b6b7 Compare October 27, 2014 18:05
@dhermes
Copy link
Contributor Author

dhermes commented Oct 27, 2014

Rebased after #292

@coveralls
Copy link

Coverage Status

Coverage remained the same when pulling 799b6b7 on dhermes:query-operator-fix-dict-random-order into f7d0441 on GoogleCloudPlatform:master.

@@ -132,10 +134,12 @@ def filter(self, expression, value):
property_name, operator = None, None
expression = expression.strip()

for operator_string in self.OPERATORS:
for operator_string, pb_op_enum in self.OPERATORS:
if expression.endswith(operator_string):

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

@tseaver
Copy link
Contributor

tseaver commented Oct 27, 2014

LGTM pending Travis.

I don't know if we want to default to assuming equality in the case that there is no operator, but that shouldn't be a blocker.

@coveralls
Copy link

Coverage Status

Coverage remained the same when pulling f47ffa1 on dhermes:query-operator-fix-dict-random-order into f7d0441 on GoogleCloudPlatform:master.

This was to get test coverage back to 100%.
@dhermes
Copy link
Contributor Author

dhermes commented Oct 27, 2014

Missing a few lines of test coverage so added 1fb43bd

Will wait out Travis again then merge. Somewhat surprised coveralls said coverage was unchanged.

@coveralls
Copy link

Coverage Status

Coverage remained the same when pulling 1fb43bd on dhermes:query-operator-fix-dict-random-order into f7d0441 on GoogleCloudPlatform:master.

dhermes added a commit that referenced this pull request Oct 27, 2014
…rder

Makes operator checking deterministic in Query.filter.
@dhermes dhermes merged commit 6f26e98 into googleapis:master Oct 27, 2014
@dhermes dhermes deleted the query-operator-fix-dict-random-order branch October 27, 2014 20:21
f_pb, = list(q_pb.filter.composite_filter.filter)
p_pb = f_pb.property_filter
self.assertEqual(p_pb.property.name, 'firstname')
self.assertEqual(p_pb.value.string_value, u'John')
self.assertEqual(p_pb.operator, datastore_pb.PropertyFilter.EQUAL)

def test_filter_w_all_operators(self):

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

urshala pushed a commit to urshala/google-cloud-python that referenced this pull request Jan 17, 2020
urshala pushed a commit to urshala/google-cloud-python that referenced this pull request Jan 17, 2020
urshala pushed a commit to urshala/google-cloud-python that referenced this pull request Jan 17, 2020
parthea added a commit that referenced this pull request Jun 4, 2023
* chore: Update gapic-generator-python to v1.8.5

PiperOrigin-RevId: 511892190

Source-Link: googleapis/googleapis@a45d9c0

Source-Link: googleapis/googleapis-gen@1907294
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMTkwNzI5NGIxZDgzNjVlYTI0ZjhjNWYyZTA1OWE2NDEyNGM0ZWQzYiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this pull request Jun 4, 2023
* feat: add `content` field in TextAnchor

PiperOrigin-RevId: 435142243

Source-Link: googleapis/googleapis@721b367

Source-Link: googleapis/googleapis-gen@b627aae
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjYyN2FhZWIzYWZhZmU3ZDk0ZDYwOTlhMmQxOTQ1YWE5YjRkYWRhZCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Jun 4, 2023
- [ ] Regenerate this pull request now.

fix: resolve DuplicateCredentialArgs error when using credentials_file

committer: parthea
PiperOrigin-RevId: 425964861

Source-Link: googleapis/googleapis@84b1a5a

Source-Link: googleapis/googleapis-gen@4fb761b
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNGZiNzYxYmJkODUwNmFjMTU2ZjQ5YmFjNWYxODMwNmFhOGViM2FhOCJ9
parthea added a commit that referenced this pull request Jun 4, 2023
* chore(deps): update all dependencies

* revert

Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea added a commit that referenced this pull request Jun 4, 2023
* chore: exclude requirements.txt file from renovate-bot

Source-Link: googleapis/synthtool@f58d313
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:7a40313731a7cb1454eef6b33d3446ebb121836738dc3ab3d2d3ded5268c35b6

* update constraints files

* fix(deps): require protobuf 3.20.2

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea added a commit that referenced this pull request Jun 4, 2023
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this pull request Jun 4, 2023
Source-Link: googleapis/synthtool@f15cc72
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:bc5eed3804aec2f05fad42aacf973821d9500c174015341f721a984a0825b6fd
parthea pushed a commit that referenced this pull request Aug 15, 2023
* docs: minor wording update

PiperOrigin-RevId: 442267451

Source-Link: googleapis/googleapis@4445d18

Source-Link: googleapis/googleapis-gen@4e175b7
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNGUxNzViNzllYzQxNThmMmU3ZmFiOWY0ZGVhMGViZjc2M2E1Y2FhNiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* docs: minor wording update

PiperOrigin-RevId: 442267541

Source-Link: googleapis/googleapis@740f072

Source-Link: googleapis/googleapis-gen@29eb892
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMjllYjg5MmNjYTViMWIwMTM0YjRlYzE4NWRkM2QyZWY1ZDJhZjFhNCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Sep 20, 2023
vchudnov-g pushed a commit that referenced this pull request Sep 20, 2023
parthea added a commit that referenced this pull request Sep 22, 2023
…ic enums (#294)

* feat: enable "rest" transport in Python for services supporting numeric enums

PiperOrigin-RevId: 508143576

Source-Link: googleapis/googleapis@7a702a9

Source-Link: googleapis/googleapis-gen@6ad1279
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNmFkMTI3OWMwZTdhYTc4N2FjNmI2NmM5ZmQ0YTIxMDY5MmVkZmZjZCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* use REST transport in samples/tests

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* revert changes to fixtures

* revert

* revert

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this pull request Sep 22, 2023
* chore(deps): update all dependencies

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Sep 22, 2023
- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 473833416

Source-Link: googleapis/googleapis@565a550

Source-Link: googleapis/googleapis-gen@1ee1a06
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMWVlMWEwNmM2ZGUzY2E4Yjg0MzU3MmMxZmRlMDU0OGY4NDIzNjk4OSJ9
parthea pushed a commit that referenced this pull request Sep 22, 2023
#294)

Source-Link: googleapis/synthtool@694118b
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:ec49167c606648a063d1222220b48119c912562849a0528f35bfb592a9f72737

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Sep 22, 2023
Source-Link: googleapis/synthtool@c4dd595
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:ce3c1686bc81145c81dd269bd12c4025c6b275b22d14641358827334fddb1d72
parthea added a commit that referenced this pull request Sep 22, 2023
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea added a commit that referenced this pull request Oct 21, 2023
* docs(samples): adding sample for local SSD disk

* Updating region tag

* Regenerating

* Updating snippets

Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this pull request Oct 21, 2023
* chore(deps): update dependency google-cloud-storage to v2

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this pull request Oct 21, 2023
* chore: update gapic-generator-python-1.4.4 with unit tests generation fixes

PiperOrigin-RevId: 475683078

Source-Link: googleapis/googleapis@df791ce

Source-Link: googleapis/googleapis-gen@ee0ce41
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZWUwY2U0MWNiOWJmN2JiNTEwMDVlOTkxMDIyYjAwY2UxYmM2NjlkOCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea added a commit that referenced this pull request Oct 21, 2023
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this pull request Oct 22, 2023
* feat: Add Cloud Trace v2 retry defaults for BatchWriteSpans

PiperOrigin-RevId: 504068544

Source-Link: googleapis/googleapis@685d359

Source-Link: googleapis/googleapis-gen@43aa9da
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDNhYTlkYWNmYzQxN2I2MjVhZDhiZTFiZWQyYTlmYzM2YzI3N2RiNiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: datastore Issues related to the Datastore API. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants