-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Allow setting transaction limit for db connections #10440
Conversation
Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool. Signed-off-by: Toni Spets <toni.spets@iki.fi>
Copy-paste from some discussion in #-dev:
a potential optimization |
It's quite a bit of more work to do that. I have doubts the connection time is actually meaningful enough to warrant that (when adding up all the other delays) unless it can be proven that this has more negative effects than positive given the reconnection itself is reliable. Testing this in scale would need to be done with good metrics anyway. I only have access to a small instance. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this seems like a sensible improvement; I'd be happy to merge this after a bit of clean-up.
I'm worried that the new code doesn't have any test coverage at all, and is therefore very likely to break in the future. Maybe you could add something to tests.storage.test_database
? There is some vaguely related stuff in tests.storage.test_base
, though I think it mocks things out at the wrong level.
Did all requested changes and added a changelog file. |
Missing import and formatting.
The code here looks good, though you'll need to run I'm still a bit dubious about merging this without the new code being used in CI at all. Did you get anywhere with looking into writing some tests for this? Maybe we could at least run a couple of |
I tried a few different methods but nothing seems to go between the right stuff without mocking the whole adbapi ConnectionPool including connections. |
Added trivial test to see it doesn't break. |
@@ -0,0 +1,36 @@ | |||
# Copyright 2014-2021 The Matrix.org Foundation C.I.C. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we don't ask you to assign copyright, so this is yours:
# Copyright 2014-2021 The Matrix.org Foundation C.I.C. | |
# Copyright 2021 Toni Spets |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's fine 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks very much for your work on this!
Synapse 1.40.0 (2021-08-10) =========================== No significant changes. Synapse 1.40.0rc3 (2021-08-09) ============================== Features -------- - Support [MSC3289: room version 8](matrix-org/matrix-spec-proposals#3289). ([\matrix-org#10449](matrix-org#10449)) Bugfixes -------- - Mark the experimental room version from [MSC2716](matrix-org/matrix-spec-proposals#2716) as unstable. ([\matrix-org#10449](matrix-org#10449)) Improved Documentation ---------------------- - Fix broken links in `upgrade.md`. Contributed by @dklimpel. ([\matrix-org#10543](matrix-org#10543)) Synapse 1.40.0rc2 (2021-08-04) ============================== Bugfixes -------- - Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\matrix-org#10517](matrix-org#10517)) - Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\matrix-org#10531](matrix-org#10531)) Internal Changes ---------------- - Fix release script to open the correct URL for the release. ([\matrix-org#10516](matrix-org#10516)) Synapse 1.40.0rc1 (2021-08-03) ============================== Features -------- - Add support for [MSC2033](matrix-org/matrix-spec-proposals#2033): `device_id` on `/account/whoami`. ([\matrix-org#9918](matrix-org#9918)) - Update support for [MSC2716 - Incrementally importing history into existing rooms](matrix-org/matrix-spec-proposals#2716). ([\matrix-org#10245](matrix-org#10245), [\matrix-org#10432](matrix-org#10432), [\matrix-org#10463](matrix-org#10463)) - Update support for [MSC3083](matrix-org/matrix-spec-proposals#3083) to consider changes in the MSC around which servers can issue join events. ([\matrix-org#10254](matrix-org#10254), [\matrix-org#10447](matrix-org#10447), [\matrix-org#10489](matrix-org#10489)) - Initial support for [MSC3244](matrix-org/matrix-spec-proposals#3244), Room version capabilities over the /capabilities API. ([\matrix-org#10283](matrix-org#10283)) - Add a buffered logging handler which periodically flushes itself. ([\matrix-org#10407](matrix-org#10407), [\matrix-org#10515](matrix-org#10515)) - Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\matrix-org#10411](matrix-org#10411)) - Support for [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\matrix-org#10413](matrix-org#10413)) - Email notifications now state whether an invitation is to a room or a space. ([\matrix-org#10426](matrix-org#10426)) - Allow setting transaction limit for database connections. ([\matrix-org#10440](matrix-org#10440), [\matrix-org#10511](matrix-org#10511)) - Add `creation_ts` to "list users" admin API. ([\matrix-org#10448](matrix-org#10448)) Bugfixes -------- - Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\matrix-org#10410](matrix-org#10410)) - Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\matrix-org#10439](matrix-org#10439)) - Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\matrix-org#10455](matrix-org#10455)) - Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\matrix-org#10499](matrix-org#10499)) Improved Documentation ---------------------- - Fix hierarchy of providers on the OpenID page. ([\matrix-org#10445](matrix-org#10445)) - Consolidate development documentation to `docs/development/`. ([\matrix-org#10453](matrix-org#10453)) - Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\matrix-org#10464](matrix-org#10464)) - Document how to use Complement while developing a new Synapse feature. ([\matrix-org#10483](matrix-org#10483)) Internal Changes ---------------- - Prune inbound federation queues for a room if they get too large. ([\matrix-org#10390](matrix-org#10390)) - Add type hints to `synapse.federation.transport.client` module. ([\matrix-org#10408](matrix-org#10408)) - Remove shebang line from module files. ([\matrix-org#10415](matrix-org#10415)) - Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\matrix-org#10429](matrix-org#10429)) - Use a docker image cache for the prerequisites for the debian package build. ([\matrix-org#10431](matrix-org#10431)) - Improve servlet type hints. ([\matrix-org#10437](matrix-org#10437), [\matrix-org#10438](matrix-org#10438)) - Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\matrix-org#10442](matrix-org#10442)) - Update the `tests-done` Github Actions status. ([\matrix-org#10444](matrix-org#10444), [\matrix-org#10512](matrix-org#10512)) - Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\matrix-org#10446](matrix-org#10446), [\matrix-org#10450](matrix-org#10450)) - Cancel redundant GHA workflows when a new commit is pushed. ([\matrix-org#10451](matrix-org#10451)) - Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\matrix-org#10468](matrix-org#10468)) - Additional type hints in the state handler. ([\matrix-org#10482](matrix-org#10482)) - Update syntax used to run complement tests. ([\matrix-org#10488](matrix-org#10488)) - Fix up type annotations to work with Twisted 21.7. ([\matrix-org#10490](matrix-org#10490)) - Improve type annotations for `ObservableDeferred`. ([\matrix-org#10491](matrix-org#10491)) - Extend release script to also tag and create GitHub releases. ([\matrix-org#10496](matrix-org#10496)) - Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\matrix-org#10500](matrix-org#10500))
Synapse 1.40.0 (2021-08-10) =========================== No significant changes. Synapse 1.40.0rc3 (2021-08-09) ============================== Features -------- - Support [MSC3289: room version 8](matrix-org/matrix-spec-proposals#3289). ([\#10449](matrix-org/synapse#10449)) Bugfixes -------- - Mark the experimental room version from [MSC2716](matrix-org/matrix-spec-proposals#2716) as unstable. ([\#10449](matrix-org/synapse#10449)) Improved Documentation ---------------------- - Fix broken links in `upgrade.md`. Contributed by @dklimpel. ([\#10543](matrix-org/synapse#10543)) Synapse 1.40.0rc2 (2021-08-04) ============================== Bugfixes -------- - Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\#10517](matrix-org/synapse#10517)) - Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\#10531](matrix-org/synapse#10531)) Internal Changes ---------------- - Fix release script to open the correct URL for the release. ([\#10516](matrix-org/synapse#10516)) Synapse 1.40.0rc1 (2021-08-03) ============================== Features -------- - Add support for [MSC2033](matrix-org/matrix-spec-proposals#2033): `device_id` on `/account/whoami`. ([\#9918](matrix-org/synapse#9918)) - Update support for [MSC2716 - Incrementally importing history into existing rooms](matrix-org/matrix-spec-proposals#2716). ([\#10245](matrix-org/synapse#10245), [\#10432](matrix-org/synapse#10432), [\#10463](matrix-org/synapse#10463)) - Update support for [MSC3083](matrix-org/matrix-spec-proposals#3083) to consider changes in the MSC around which servers can issue join events. ([\#10254](matrix-org/synapse#10254), [\#10447](matrix-org/synapse#10447), [\#10489](matrix-org/synapse#10489)) - Initial support for [MSC3244](matrix-org/matrix-spec-proposals#3244), Room version capabilities over the /capabilities API. ([\#10283](matrix-org/synapse#10283)) - Add a buffered logging handler which periodically flushes itself. ([\#10407](matrix-org/synapse#10407), [\#10515](matrix-org/synapse#10515)) - Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\#10411](matrix-org/synapse#10411)) - Support for [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\#10413](matrix-org/synapse#10413)) - Email notifications now state whether an invitation is to a room or a space. ([\#10426](matrix-org/synapse#10426)) - Allow setting transaction limit for database connections. ([\#10440](matrix-org/synapse#10440), [\#10511](matrix-org/synapse#10511)) - Add `creation_ts` to "list users" admin API. ([\#10448](matrix-org/synapse#10448)) Bugfixes -------- - Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\#10410](matrix-org/synapse#10410)) - Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\#10439](matrix-org/synapse#10439)) - Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\#10455](matrix-org/synapse#10455)) - Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\#10499](matrix-org/synapse#10499)) Improved Documentation ---------------------- - Fix hierarchy of providers on the OpenID page. ([\#10445](matrix-org/synapse#10445)) - Consolidate development documentation to `docs/development/`. ([\#10453](matrix-org/synapse#10453)) - Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\#10464](matrix-org/synapse#10464)) - Document how to use Complement while developing a new Synapse feature. ([\#10483](matrix-org/synapse#10483)) Internal Changes ---------------- - Prune inbound federation queues for a room if they get too large. ([\#10390](matrix-org/synapse#10390)) - Add type hints to `synapse.federation.transport.client` module. ([\#10408](matrix-org/synapse#10408)) - Remove shebang line from module files. ([\#10415](matrix-org/synapse#10415)) - Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\#10429](matrix-org/synapse#10429)) - Use a docker image cache for the prerequisites for the debian package build. ([\#10431](matrix-org/synapse#10431)) - Improve servlet type hints. ([\#10437](matrix-org/synapse#10437), [\#10438](matrix-org/synapse#10438)) - Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\#10442](matrix-org/synapse#10442)) - Update the `tests-done` Github Actions status. ([\#10444](matrix-org/synapse#10444), [\#10512](matrix-org/synapse#10512)) - Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\#10446](matrix-org/synapse#10446), [\#10450](matrix-org/synapse#10450)) - Cancel redundant GHA workflows when a new commit is pushed. ([\#10451](matrix-org/synapse#10451)) - Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\#10468](matrix-org/synapse#10468)) - Additional type hints in the state handler. ([\#10482](matrix-org/synapse#10482)) - Update syntax used to run complement tests. ([\#10488](matrix-org/synapse#10488)) - Fix up type annotations to work with Twisted 21.7. ([\#10490](matrix-org/synapse#10490)) - Improve type annotations for `ObservableDeferred`. ([\#10491](matrix-org/synapse#10491)) - Extend release script to also tag and create GitHub releases. ([\#10496](matrix-org/synapse#10496)) - Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\#10500](matrix-org/synapse#10500))
Synapse 1.40.0 (2021-08-10) =========================== No significant changes. Synapse 1.40.0rc3 (2021-08-09) ============================== Features -------- - Support [MSC3289: room version 8](matrix-org/matrix-spec-proposals#3289). ([\matrix-org#10449](matrix-org#10449)) Bugfixes -------- - Mark the experimental room version from [MSC2716](matrix-org/matrix-spec-proposals#2716) as unstable. ([\matrix-org#10449](matrix-org#10449)) Improved Documentation ---------------------- - Fix broken links in `upgrade.md`. Contributed by @dklimpel. ([\matrix-org#10543](matrix-org#10543)) Synapse 1.40.0rc2 (2021-08-04) ============================== Bugfixes -------- - Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\matrix-org#10517](matrix-org#10517)) - Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\matrix-org#10531](matrix-org#10531)) Internal Changes ---------------- - Fix release script to open the correct URL for the release. ([\matrix-org#10516](matrix-org#10516)) Synapse 1.40.0rc1 (2021-08-03) ============================== Features -------- - Add support for [MSC2033](matrix-org/matrix-spec-proposals#2033): `device_id` on `/account/whoami`. ([\matrix-org#9918](matrix-org#9918)) - Update support for [MSC2716 - Incrementally importing history into existing rooms](matrix-org/matrix-spec-proposals#2716). ([\matrix-org#10245](matrix-org#10245), [\matrix-org#10432](matrix-org#10432), [\matrix-org#10463](matrix-org#10463)) - Update support for [MSC3083](matrix-org/matrix-spec-proposals#3083) to consider changes in the MSC around which servers can issue join events. ([\matrix-org#10254](matrix-org#10254), [\matrix-org#10447](matrix-org#10447), [\matrix-org#10489](matrix-org#10489)) - Initial support for [MSC3244](matrix-org/matrix-spec-proposals#3244), Room version capabilities over the /capabilities API. ([\matrix-org#10283](matrix-org#10283)) - Add a buffered logging handler which periodically flushes itself. ([\matrix-org#10407](matrix-org#10407), [\matrix-org#10515](matrix-org#10515)) - Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\matrix-org#10411](matrix-org#10411)) - Support for [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\matrix-org#10413](matrix-org#10413)) - Email notifications now state whether an invitation is to a room or a space. ([\matrix-org#10426](matrix-org#10426)) - Allow setting transaction limit for database connections. ([\matrix-org#10440](matrix-org#10440), [\matrix-org#10511](matrix-org#10511)) - Add `creation_ts` to "list users" admin API. ([\matrix-org#10448](matrix-org#10448)) Bugfixes -------- - Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\matrix-org#10410](matrix-org#10410)) - Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\matrix-org#10439](matrix-org#10439)) - Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\matrix-org#10455](matrix-org#10455)) - Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\matrix-org#10499](matrix-org#10499)) Improved Documentation ---------------------- - Fix hierarchy of providers on the OpenID page. ([\matrix-org#10445](matrix-org#10445)) - Consolidate development documentation to `docs/development/`. ([\matrix-org#10453](matrix-org#10453)) - Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\matrix-org#10464](matrix-org#10464)) - Document how to use Complement while developing a new Synapse feature. ([\matrix-org#10483](matrix-org#10483)) Internal Changes ---------------- - Prune inbound federation queues for a room if they get too large. ([\matrix-org#10390](matrix-org#10390)) - Add type hints to `synapse.federation.transport.client` module. ([\matrix-org#10408](matrix-org#10408)) - Remove shebang line from module files. ([\matrix-org#10415](matrix-org#10415)) - Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\matrix-org#10429](matrix-org#10429)) - Use a docker image cache for the prerequisites for the debian package build. ([\matrix-org#10431](matrix-org#10431)) - Improve servlet type hints. ([\matrix-org#10437](matrix-org#10437), [\matrix-org#10438](matrix-org#10438)) - Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\matrix-org#10442](matrix-org#10442)) - Update the `tests-done` Github Actions status. ([\matrix-org#10444](matrix-org#10444), [\matrix-org#10512](matrix-org#10512)) - Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\matrix-org#10446](matrix-org#10446), [\matrix-org#10450](matrix-org#10450)) - Cancel redundant GHA workflows when a new commit is pushed. ([\matrix-org#10451](matrix-org#10451)) - Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\matrix-org#10468](matrix-org#10468)) - Additional type hints in the state handler. ([\matrix-org#10482](matrix-org#10482)) - Update syntax used to run complement tests. ([\matrix-org#10488](matrix-org#10488)) - Fix up type annotations to work with Twisted 21.7. ([\matrix-org#10490](matrix-org#10490)) - Improve type annotations for `ObservableDeferred`. ([\matrix-org#10491](matrix-org#10491)) - Extend release script to also tag and create GitHub releases. ([\matrix-org#10496](matrix-org#10496)) - Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\matrix-org#10500](matrix-org#10500))
Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool.
I would not set a default value for now but allow testing this as an experimental feature.
Possibly fixes #9182
Pull Request Checklist
EventStore
toEventWorkerStore
.".code blocks
.