Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Make sure that we close cursors before returning from a query #6408

Merged
merged 2 commits into from
Nov 25, 2019
Merged

Make sure that we close cursors before returning from a query #6408

merged 2 commits into from
Nov 25, 2019

Conversation

richvdh
Copy link
Member

@richvdh richvdh commented Nov 25, 2019

There are lots of words in the comment as to why this is a good idea.

Fixes #6403.

@richvdh
Make sure that we close cursors before returning from a query
8079cc4 
There are lots of words in the comment as to why this is a good idea.
@richvdh richvdh requested a review from a team November 25, 2019 12:38
hawkowl
hawkowl reviewed Nov 25, 2019
View reviewed changes
synapse/storage/_base.py Outdated
# statements on the connection [1], which will make our cursor
# invalid [2].
#
# While the above probably doesn't apply to postgres, we still need
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't actually use cursors in postgres, since we don't name them, therefore we're just fetching all the data on a query. psycopg2's cursor when not named is just an abstraction.

fwiw postgres says that any cursor can see any changes from any other cursor on the same connection, named or not

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also calling .close() on a postgres cursor isn't required as it's implicitly closed at the end of a transaction anyway, as after that it loses semantic meaning. for "client-side cursors" that we use, there's nothing to actually 'close', since no portal was created -- in this case, psycopg2 just sets it as closed and yells if you try and reuse the same cursor.

hawkowl
hawkowl reviewed Nov 25, 2019
View reviewed changes
synapse/storage/data_stores/main/receipts.py Outdated
@@ -280,7 +280,7 @@ def get_all_updated_receipts_txn(txn):
args.append(limit)
txn.execute(sql, args)

return (r[0:5] + (json.loads(r[5]),) for r in txn)
return [r[0:5] + (json.loads(r[5]),) for r in txn]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we not wrap this in an explicit list(), to force it from being a generator?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't that the same as using [] ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, yes, the ()s you replaced keep it being a generator (because it doesn't cast it to a tuple, while [] does cast to a list, because python is an intuitive language)

I'd personally do list(r[0:5] + (json.loads(r[5]),) for r in txn) to make the casting more obvious, but as-written is fine

hawkowl
hawkowl reviewed Nov 25, 2019
View reviewed changes
synapse/storage/_base.py Outdated
@@ -409,16 +409,16 @@ def _new_transaction(
i = 0
N = 5
while True:
cursor = conn.cursor()
cursor = LoggingTransaction(
cursor,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we call conn.cursor() in here instead of redefining a variable? this will bite us when mypy gets to it eventually :)

hawkowl
hawkowl approved these changes Nov 25, 2019
View reviewed changes
Copy link
Contributor

@hawkowl hawkowl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some background information on the PostgreSQL behaviour, so you can simplify the comment down regarding behavior. Calling close when we're done with cursors is a good idea anyway, since if they're server side they can hold resources. Also added some minor code comments, fix as you see fit.

@richvdh richvdh merged commit c01d543 into develop Nov 25, 2019
@richvdh richvdh deleted the rav/close_cursors branch November 25, 2019 21:03
neilisfragile added a commit that referenced this pull request Dec 9, 2019
@neilisfragile
Merge tag 'v1.7.0rc1' into develop
0f3614f 
Synapse 1.7.0rc1 (2019-12-09)
=============================

Features
--------

- Implement per-room message retention policies. ([\#5815](#5815), [\#6436](#6436))
- Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](#5858))
- Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925))
- Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](#6119))
- Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](#6176))
- Configure privacy-preserving settings by default for the room directory. ([\#6354](#6354))
- Add ephemeral messages support by partially implementing [MSC2228](matrix-org/matrix-spec-proposals#2228). ([\#6409](#6409))
- Add support for [MSC 2367](matrix-org/matrix-spec-proposals#2367), which allows specifying a reason on all membership events. ([\#6434](#6434))

Bugfixes
--------

- Transfer non-standard power levels on room upgrade. ([\#6237](#6237))
- Fix error from the Pillow library when uploading RGBA images. ([\#6241](#6241))
- Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](#6329))
- Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](#6332))
- Prevent account data syncs getting lost across TCP replication. ([\#6333](#6333))
- Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](#6406))
- Fix an intermittent exception when handling read-receipts. ([\#6408](#6408))
- Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](#6420))
- Fix startup error when http proxy is defined. ([\#6421](#6421))
- Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](#6449))
- Fix uploading multiple cross signing signatures for the same user. ([\#6451](#6451))
- Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](#6462))
- Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](#6470))
- Improve sanity-checking when receiving events over federation. ([\#6472](#6472))
- Fix inaccurate per-block Prometheus metrics. ([\#6491](#6491))
- Fix small performance regression for sending invites. ([\#6493](#6493))
- Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](#6494))

Improved Documentation
----------------------

- Update documentation and variables in user contributed systemd reference file. ([\#6369](#6369), [\#6490](#6490))
- Fix link in the user directory documentation. ([\#6388](#6388))
- Add build instructions to the docker readme. ([\#6390](#6390))
- Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](#6443))
- Write some docs for the quarantine_media api. ([\#6458](#6458))
- Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](#6461))

Deprecations and Removals
-------------------------

- Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925))
- Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](#6488))

Internal Changes
----------------

- Add benchmarks for structured logging and improve output performance. ([\#6266](#6266))
- Improve the performance of outputting structured logging. ([\#6322](#6322))
- Refactor some code in the event authentication path for clarity. ([\#6343](#6343), [\#6468](#6468), [\#6480](#6480))
- Clean up some unnecessary quotation marks around the codebase. ([\#6362](#6362))
- Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](#6379))
- Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](#6392))
- Clarifications for the email configuration settings. ([\#6423](#6423))
- Add more tests to the blacklist when running in worker mode. ([\#6429](#6429))
- Refactor data store layer to support multiple databases in the future. ([\#6454](#6454), [\#6464](#6464), [\#6469](#6469), [\#6487](#6487))
- Port synapse.rest.client.v1 to async/await. ([\#6482](#6482))
- Port synapse.rest.client.v2_alpha to async/await. ([\#6483](#6483))
- Port SyncHandler to async/await. ([\#6484](#6484))
erikjohnston added a commit that referenced this pull request Dec 13, 2019
@erikjohnston
Merge tag 'v1.7.0'
bee1982 
Synapse 1.7.0 (2019-12-13)
==========================

This release changes the default settings so that only local authenticated users can query the server's room directory. See the [upgrade notes](UPGRADE.rst#upgrading-to-v170) for details.

Support for SQLite versions before 3.11 is now deprecated. A future release will refuse to start if used with an SQLite version before 3.11.

Administrators are reminded that SQLite should not be used for production instances. Instructions for migrating to Postgres are available [here](docs/postgres.md). A future release of synapse will, by default, disable federation for servers using SQLite.

No significant changes since 1.7.0rc2.

Synapse 1.7.0rc2 (2019-12-11)
=============================

Bugfixes
--------

- Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](#6497))
- Fix support for SQLite 3.7. ([\#6499](#6499))
- Fix regression where sending email push would not work when using a pusher worker. ([\#6507](#6507), [\#6509](#6509))

Synapse 1.7.0rc1 (2019-12-09)
=============================

Features
--------

- Implement per-room message retention policies. ([\#5815](#5815), [\#6436](#6436))
- Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](#5858))
- Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925))
- Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](#6119))
- Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](#6176))
- Configure privacy-preserving settings by default for the room directory. ([\#6355](#6355))
- Add ephemeral messages support by partially implementing [MSC2228](matrix-org/matrix-spec-proposals#2228). ([\#6409](#6409))
- Add support for [MSC 2367](matrix-org/matrix-spec-proposals#2367), which allows specifying a reason on all membership events. ([\#6434](#6434))

Bugfixes
--------

- Transfer non-standard power levels on room upgrade. ([\#6237](#6237))
- Fix error from the Pillow library when uploading RGBA images. ([\#6241](#6241))
- Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](#6329))
- Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](#6332))
- Prevent account data syncs getting lost across TCP replication. ([\#6333](#6333))
- Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](#6406))
- Fix an intermittent exception when handling read-receipts. ([\#6408](#6408))
- Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](#6420))
- Fix startup error when http proxy is defined. ([\#6421](#6421))
- Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](#6449))
- Fix uploading multiple cross signing signatures for the same user. ([\#6451](#6451))
- Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](#6462))
- Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](#6470))
- Improve sanity-checking when receiving events over federation. ([\#6472](#6472))
- Fix inaccurate per-block Prometheus metrics. ([\#6491](#6491))
- Fix small performance regression for sending invites. ([\#6493](#6493))
- Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](#6494))

Improved Documentation
----------------------

- Update documentation and variables in user contributed systemd reference file. ([\#6369](#6369), [\#6490](#6490))
- Fix link in the user directory documentation. ([\#6388](#6388))
- Add build instructions to the docker readme. ([\#6390](#6390))
- Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](#6443))
- Write some docs for the quarantine_media api. ([\#6458](#6458))
- Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](#6461))

Deprecations and Removals
-------------------------

- Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925))
- Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](#6488))

Internal Changes
----------------

- Add benchmarks for structured logging and improve output performance. ([\#6266](#6266))
- Improve the performance of outputting structured logging. ([\#6322](#6322))
- Refactor some code in the event authentication path for clarity. ([\#6343](#6343), [\#6468](#6468), [\#6480](#6480))
- Clean up some unnecessary quotation marks around the codebase. ([\#6362](#6362))
- Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](#6379))
- Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](#6392))
- Clarifications for the email configuration settings. ([\#6423](#6423))
- Add more tests to the blacklist when running in worker mode. ([\#6429](#6429))
- Refactor data store layer to support multiple databases in the future. ([\#6454](#6454), [\#6464](#6464), [\#6469](#6469), [\#6487](#6487))
- Port synapse.rest.client.v1 to async/await. ([\#6482](#6482))
- Port synapse.rest.client.v2_alpha to async/await. ([\#6483](#6483))
- Port SyncHandler to async/await. ([\#6484](#6484))
babolivier pushed a commit that referenced this pull request Sep 1, 2021
@anoadragon453
Make sure that we close cursors before returning from a query (#6408)
70c5926 
* commit 'c01d54358':
  Make sure that we close cursors before returning from a query (#6408)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@hawkowl hawkowl hawkowl approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@richvdh @hawkowl