When prepared statements are disabled, avoid relying on them harder #1065
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It appears that PgBouncer's `transaction` pooling mode does not consider implicit transactions properly, and so in a [`Parse`, `Flush`, `Bind`, `Execute`, `Sync`] sequence, `Flush` would be (incorrectly) considered by PgBouncer as a transaction boundary and it will happily send the following `Bind` / `Execute` messages to a different backend process. This makes it so that when `statement_cache_size` is set to `0`, asyncpg assumes a pessimistic stance on prepared statement persistence and does not rely on them even in implicit transactions. The above message sequence thus becomes `Parse`, `Flush`, `Parse` (a second time), `Bind`, `Execute`, `Sync`. This obviously has negative performance impact due to the extraneous `Parse`. Fixes: #1058 Fixes: #1041
fantix
approved these changes
Aug 17, 2023
elprans
added a commit
that referenced
this pull request
Nov 5, 2023
Minor fixes and improvements. Improvements ============ * Python 3.12 and PostgreSQL 16 support (#1084) (by @elprans in deea86c) * Add support for tuple-format custom codecs on composite types (#1061) (by @elprans in 922fcd1) * Support `target_session_attrs` in URL format, add tests (#1073) (by @elprans in 7cb4e70) * Infinity numeric support (#1067) (by @krokoziabla in 0c3bf60 for #1020) * Add support for the `WHERE` clause in `copy_to` methods (#941) (by @kaylynn234 in b7ffab6) * Add query logging callbacks and context manager (#1043) (by @dcwatson in b2697ff) Fixes ===== * When prepared statements are disabled, avoid relying on them harder (#1065) (by @elprans in cbf64e1) * Handle environments with HOME set to a not-a-directory (#1063) (by @elprans in af922bc) * Fix handling of non-ASCII passwords (#1062) (by @elprans in 89d5bd0) * Disable JIT while doing type introspection (#1082) (by @elprans in f21ebf6) * Remove connection parameter caching in `Pool` (#1053) (by @ermakov-oleg in 4ddb039) * Switch to Python 3.12-style `wait_for` (#1086) (by @elprans in 4bdd8a7) * Update automatic PostGIS type conversion for Shapely 2.0 (#1085) (by @ChimneySwift in 8b45beb) * Use the `timeout` context manager in the connection path (#1087) (by @elprans in 313b2b2) * Small fix for documentation on using SSL in Connection (#995) (by @ScottFred in ccc7baf) * Use cleanup_ctx in pool usage doc (#878) (by @ir4y in 70c8bd8) * Close cursor portals once the iterator is exhausted (#1088) (by @elprans in ca9f03b) * Cut BaseProtocol circular reference on close. (#1049) (by @pteromys in 93a6f79) * Allow passing hosts as tuples to `connect()` (in addition to lists) (#1021) (by @lezram in d7faaff) Other ===== * Drop support for Python 3.7 (#1064) (by @bryanforbes in 87ab143)
Merged
elprans
added a commit
that referenced
this pull request
Nov 5, 2023
Minor fixes and improvements. Improvements ============ * Python 3.12 and PostgreSQL 16 support (#1084) (by @elprans in deea86c) * Add support for tuple-format custom codecs on composite types (#1061) (by @elprans in 922fcd1) * Support `target_session_attrs` in URL format, add tests (#1073) (by @elprans in 7cb4e70) * Infinity numeric support (#1067) (by @krokoziabla in 0c3bf60 for #1020) * Add support for the `WHERE` clause in `copy_to` methods (#941) (by @kaylynn234 in b7ffab6) * Add query logging callbacks and context manager (#1043) (by @dcwatson in b2697ff) Fixes ===== * When prepared statements are disabled, avoid relying on them harder (#1065) (by @elprans in cbf64e1) * Handle environments with HOME set to a not-a-directory (#1063) (by @elprans in af922bc) * Fix handling of non-ASCII passwords (#1062) (by @elprans in 89d5bd0) * Disable JIT while doing type introspection (#1082) (by @elprans in f21ebf6) * Remove connection parameter caching in `Pool` (#1053) (by @ermakov-oleg in 4ddb039) * Switch to Python 3.12-style `wait_for` (#1086) (by @elprans in 4bdd8a7) * Update automatic PostGIS type conversion for Shapely 2.0 (#1085) (by @ChimneySwift in 8b45beb) * Use the `timeout` context manager in the connection path (#1087) (by @elprans in 313b2b2) * Small fix for documentation on using SSL in Connection (#995) (by @ScottFred in ccc7baf) * Use cleanup_ctx in pool usage doc (#878) (by @ir4y in 70c8bd8) * Close cursor portals once the iterator is exhausted (#1088) (by @elprans in ca9f03b) * Cut BaseProtocol circular reference on close. (#1049) (by @pteromys in 93a6f79) * Allow passing hosts as tuples to `connect()` (in addition to lists) (#1021) (by @lezram in d7faaff) Other ===== * Drop support for Python 3.7 (#1064) (by @bryanforbes in 87ab143)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It appears that PgBouncer's
transactionpooling mode does not considerimplicit transactions properly, and so in a [
Parse,Flush,Bind,Execute,Sync] sequence,Flushwould be (incorrectly) considered byPgBouncer as a transaction boundary and it will happily send the
following
Bind/Executemessages to a different backend process.This makes it so that when
statement_cache_sizeis set to0, asyncpgassumes a pessimistic stance on prepared statement persistence and does
not rely on them even in implicit transactions. The above message
sequence thus becomes
Parse,Flush,Parse(a second time),Bind,Execute,Sync.This obviously has negative performance impact due to the extraneous
Parse.Fixes: #1058
Fixes: #1041