-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Inline SQL queries using boolean parameters #15525
Conversation
Fixing that SQLite query according to 3.27.0
Updated Inline SQL queries according so SQLite 3.27
@anshulm333 - I guess there are multiple places where the SQL queries are not inline. You have addressed one of the query. Isn't it better to address all of them in this PR? Thoughts? @H-Shay |
Thanks for taking this on!
If there are not a huge number of queries to reformat then I think it makes sense to do it all in one PR, but if there are a significant number I think it would make sense to do a couple of PRs, but you are correct that doing them just one at a time doesn't really make sense. |
The current changes look fine, but I'm going to remove this from the review queue since it sounds like more work is planned. @anshulm333 Also please be sure to add a changelog and sign-off on your changes as the description says. |
…in sqlite 3.27 Signed-off-by: Anshul Madnawat <100751856+anshulm333@users.noreply.github.com>
…in sqlite 3.27 Signed-off-by: Anshul Madnawat <100751856+anshulm333@users.noreply.github.com>
I think I've updated most of the queries, I hardly found any other than these. There're not much SQL queries that needs to be inline. |
@@ -1455,8 +1455,8 @@ def _update_outliers_txn( | |||
}, | |||
) | |||
|
|||
sql = "UPDATE events SET outlier = ? WHERE event_id = ?" | |||
txn.execute(sql, (False, event.event_id)) | |||
sql = "UPDATE events SET outlier = False WHERE event_id = ?" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I've updated most of the queries, I hardly found any other than these. There're not much SQL queries that needs to be inline.
I think there are a lot more of these instances. If I just search for outlier = ?
, I see 5 things and only 3 places changed in this PR.
We probably need to go through any boolean
column in synapse/storage/schema/main/full_schemas/72/full.sql.postgres
and do the same search across the codebase.
And figure out a plan for how you want to tackle and split things if there are too many.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably need to go through any
boolean
column insynapse/storage/schema/main/full_schemas/72/full.sql.postgres
and do the same search across the codebase.
I think that would be a good way to check all the inline queries in the codebase.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Note that the schema dump is a little out of date; there will likely be new boolean columns added since then.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that the synapse_port_db
script might have the most up-to-date list of boolean columns to check:
synapse/synapse/_scripts/synapse_port_db.py
Lines 107 to 129 in 7c9b917
BOOLEAN_COLUMNS = { | |
"access_tokens": ["used"], | |
"account_validity": ["email_sent"], | |
"device_lists_changes_in_room": ["converted_to_destinations"], | |
"device_lists_outbound_pokes": ["sent"], | |
"devices": ["hidden"], | |
"e2e_fallback_keys_json": ["used"], | |
"e2e_room_keys": ["is_verified"], | |
"event_edges": ["is_state"], | |
"events": ["processed", "outlier", "contains_url"], | |
"local_media_repository": ["safe_from_quarantine"], | |
"presence_list": ["accepted"], | |
"presence_stream": ["currently_active"], | |
"public_room_list_stream": ["visibility"], | |
"pushers": ["enabled"], | |
"redactions": ["have_censored"], | |
"room_stats_state": ["is_federatable"], | |
"rooms": ["is_public", "has_auth_chain_index"], | |
"users": ["shadow_banned", "approved"], | |
"un_partial_stated_event_stream": ["rejection_status_changed"], | |
"users_who_share_rooms": ["share_private"], | |
"per_user_experimental_features": ["enabled"], | |
} |
Hello. My guess is that this issue is still open and we have to fix the issues in the codebase altogether. Can I contribute to this PR? |
I am already working on it, will commit all/most of the changes within next 3-4 days. |
…QLite 3.27 Signed-off-by: Anshul Madnawat <100751856+anshulm333@users.noreply.github.com>
@clokep can you please ask the team to review it, I've done changes as recommended. |
Signed-off-by: Anshul Madnawat <100751856+anshulm333@users.noreply.github.com>
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Signed-off-by: Anshul Madnawat <100751856+anshulm333@users.noreply.github.com>
Signed-off-by: Anshul Madnawat <100751856+anshulm333@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! Thanks for working through this with us.
@anshulm333 There's no reason to keep merging in develop -- the failed build was due to a flake and we can just kick it off. Now we have to wait for all the builds to complete again. |
Woo, thanks so much for your first contribution @anshulm333! 🎉 |
No significant changes since 1.90.0rc1. - Scope transaction IDs to devices (implement [MSC3970](matrix-org/matrix-spec-proposals#3970)). ([\matrix-org#15629](matrix-org#15629)) - Remove old rows from the `cache_invalidation_stream_by_instance` table automatically (this table is unused in SQLite). ([\matrix-org#15868](matrix-org#15868)) - Fix a long-standing bug where purging history and paginating simultaneously could lead to database corruption when using workers. ([\matrix-org#15791](matrix-org#15791)) - Fix a long-standing bug where profile endpoint returned a 404 when the user's display name was empty. ([\matrix-org#16012](matrix-org#16012)) - Fix a long-standing bug where the `synapse_port_db` failed to configure sequences for application services and partial stated rooms. ([\matrix-org#16043](matrix-org#16043)) - Fix long-standing bug with deletion in dehydrated devices v2. ([\matrix-org#16046](matrix-org#16046)) - Add `org.opencontainers.image.version` labels to Docker containers [published by Matrix.org](https://hub.docker.com/r/matrixdotorg/synapse). Contributed by Mo Balaa. ([\matrix-org#15972](matrix-org#15972), [\matrix-org#16009](matrix-org#16009)) - Add a internal documentation page describing the ["streams" used within Synapse](https://matrix-org.github.io/synapse/v1.90/development/synapse_architecture/streams.html). ([\matrix-org#16015](matrix-org#16015)) - Clarify comment on the keys/upload over replication enpoint. ([\matrix-org#16016](matrix-org#16016)) - Do not expose Admin API in caddy reverse proxy example. Contributed by @NilsIrl. ([\matrix-org#16027](matrix-org#16027)) - Remove support for legacy application service paths. ([\matrix-org#15964](matrix-org#15964)) - Move support for application service query parameter authorization behind a configuration option. ([\matrix-org#16017](matrix-org#16017)) - Update SQL queries to inline boolean parameters as supported in SQLite 3.27. ([\matrix-org#15525](matrix-org#15525)) - Allow for the configuration of the backoff algorithm for federation destinations. ([\matrix-org#15754](matrix-org#15754)) - Allow modules to check whether the current worker is configured to run background tasks. ([\matrix-org#15991](matrix-org#15991)) - Update support for [MSC3958](matrix-org/matrix-spec-proposals#3958) to match the latest revision of the MSC. ([\matrix-org#15992](matrix-org#15992)) - Allow modules to schedule delayed background calls. ([\matrix-org#15993](matrix-org#15993)) - Properly overwrite the `redacts` content-property for forwards-compatibility with room versions 1 through 10. ([\matrix-org#16013](matrix-org#16013)) - Fix building the nix development environment on MacOS systems. ([\matrix-org#16019](matrix-org#16019)) - Remove leading and trailing spaces when setting a display name. ([\matrix-org#16031](matrix-org#16031)) - Combine duplicated code. ([\matrix-org#16023](matrix-org#16023)) - Collect additional metrics from `ResponseCache` for eviction. ([\matrix-org#16028](matrix-org#16028)) - Fix endpoint improperly declaring support for MSC3814. ([\matrix-org#16068](matrix-org#16068)) - Drop backwards compat hack for event serialization. ([\matrix-org#16069](matrix-org#16069)) * Update PyYAML to 6.0.1. ([\matrix-org#16011](matrix-org#16011)) * Bump cryptography from 41.0.2 to 41.0.3. ([\matrix-org#16048](matrix-org#16048)) * Bump furo from 2023.5.20 to 2023.7.26. ([\matrix-org#16077](matrix-org#16077)) * Bump immutabledict from 2.2.4 to 3.0.0. ([\matrix-org#16034](matrix-org#16034)) * Update certifi to 2023.7.22 and pygments to 2.15.1. ([\matrix-org#16044](matrix-org#16044)) * Bump jsonschema from 4.18.3 to 4.19.0. ([\matrix-org#16081](matrix-org#16081)) * Bump phonenumbers from 8.13.14 to 8.13.18. ([\matrix-org#16076](matrix-org#16076)) * Bump regex from 1.9.1 to 1.9.3. ([\matrix-org#16073](matrix-org#16073)) * Bump serde from 1.0.171 to 1.0.175. ([\matrix-org#15982](matrix-org#15982)) * Bump serde from 1.0.175 to 1.0.179. ([\matrix-org#16033](matrix-org#16033)) * Bump serde from 1.0.179 to 1.0.183. ([\matrix-org#16074](matrix-org#16074)) * Bump serde_json from 1.0.103 to 1.0.104. ([\matrix-org#16032](matrix-org#16032)) * Bump service-identity from 21.1.0 to 23.1.0. ([\matrix-org#16038](matrix-org#16038)) * Bump types-commonmark from 0.9.2.3 to 0.9.2.4. ([\matrix-org#16037](matrix-org#16037)) * Bump types-jsonschema from 4.17.0.8 to 4.17.0.10. ([\matrix-org#16036](matrix-org#16036)) * Bump types-netaddr from 0.8.0.8 to 0.8.0.9. ([\matrix-org#16035](matrix-org#16035)) * Bump types-opentracing from 2.4.10.5 to 2.4.10.6. ([\matrix-org#16078](matrix-org#16078)) * Bump types-setuptools from 68.0.0.0 to 68.0.0.3. ([\matrix-org#16079](matrix-org#16079))
No significant changes since 1.90.0rc1. - Scope transaction IDs to devices (implement [MSC3970](matrix-org/matrix-spec-proposals#3970)). ([\matrix-org#15629](matrix-org#15629)) - Remove old rows from the `cache_invalidation_stream_by_instance` table automatically (this table is unused in SQLite). ([\matrix-org#15868](matrix-org#15868)) - Fix a long-standing bug where purging history and paginating simultaneously could lead to database corruption when using workers. ([\matrix-org#15791](matrix-org#15791)) - Fix a long-standing bug where profile endpoint returned a 404 when the user's display name was empty. ([\matrix-org#16012](matrix-org#16012)) - Fix a long-standing bug where the `synapse_port_db` failed to configure sequences for application services and partial stated rooms. ([\matrix-org#16043](matrix-org#16043)) - Fix long-standing bug with deletion in dehydrated devices v2. ([\matrix-org#16046](matrix-org#16046)) - Add `org.opencontainers.image.version` labels to Docker containers [published by Matrix.org](https://hub.docker.com/r/matrixdotorg/synapse). Contributed by Mo Balaa. ([\matrix-org#15972](matrix-org#15972), [\matrix-org#16009](matrix-org#16009)) - Add a internal documentation page describing the ["streams" used within Synapse](https://matrix-org.github.io/synapse/v1.90/development/synapse_architecture/streams.html). ([\matrix-org#16015](matrix-org#16015)) - Clarify comment on the keys/upload over replication enpoint. ([\matrix-org#16016](matrix-org#16016)) - Do not expose Admin API in caddy reverse proxy example. Contributed by @NilsIrl. ([\matrix-org#16027](matrix-org#16027)) - Remove support for legacy application service paths. ([\matrix-org#15964](matrix-org#15964)) - Move support for application service query parameter authorization behind a configuration option. ([\matrix-org#16017](matrix-org#16017)) - Update SQL queries to inline boolean parameters as supported in SQLite 3.27. ([\matrix-org#15525](matrix-org#15525)) - Allow for the configuration of the backoff algorithm for federation destinations. ([\matrix-org#15754](matrix-org#15754)) - Allow modules to check whether the current worker is configured to run background tasks. ([\matrix-org#15991](matrix-org#15991)) - Update support for [MSC3958](matrix-org/matrix-spec-proposals#3958) to match the latest revision of the MSC. ([\matrix-org#15992](matrix-org#15992)) - Allow modules to schedule delayed background calls. ([\matrix-org#15993](matrix-org#15993)) - Properly overwrite the `redacts` content-property for forwards-compatibility with room versions 1 through 10. ([\matrix-org#16013](matrix-org#16013)) - Fix building the nix development environment on MacOS systems. ([\matrix-org#16019](matrix-org#16019)) - Remove leading and trailing spaces when setting a display name. ([\matrix-org#16031](matrix-org#16031)) - Combine duplicated code. ([\matrix-org#16023](matrix-org#16023)) - Collect additional metrics from `ResponseCache` for eviction. ([\matrix-org#16028](matrix-org#16028)) - Fix endpoint improperly declaring support for MSC3814. ([\matrix-org#16068](matrix-org#16068)) - Drop backwards compat hack for event serialization. ([\matrix-org#16069](matrix-org#16069)) * Update PyYAML to 6.0.1. ([\matrix-org#16011](matrix-org#16011)) * Bump cryptography from 41.0.2 to 41.0.3. ([\matrix-org#16048](matrix-org#16048)) * Bump furo from 2023.5.20 to 2023.7.26. ([\matrix-org#16077](matrix-org#16077)) * Bump immutabledict from 2.2.4 to 3.0.0. ([\matrix-org#16034](matrix-org#16034)) * Update certifi to 2023.7.22 and pygments to 2.15.1. ([\matrix-org#16044](matrix-org#16044)) * Bump jsonschema from 4.18.3 to 4.19.0. ([\matrix-org#16081](matrix-org#16081)) * Bump phonenumbers from 8.13.14 to 8.13.18. ([\matrix-org#16076](matrix-org#16076)) * Bump regex from 1.9.1 to 1.9.3. ([\matrix-org#16073](matrix-org#16073)) * Bump serde from 1.0.171 to 1.0.175. ([\matrix-org#15982](matrix-org#15982)) * Bump serde from 1.0.175 to 1.0.179. ([\matrix-org#16033](matrix-org#16033)) * Bump serde from 1.0.179 to 1.0.183. ([\matrix-org#16074](matrix-org#16074)) * Bump serde_json from 1.0.103 to 1.0.104. ([\matrix-org#16032](matrix-org#16032)) * Bump service-identity from 21.1.0 to 23.1.0. ([\matrix-org#16038](matrix-org#16038)) * Bump types-commonmark from 0.9.2.3 to 0.9.2.4. ([\matrix-org#16037](matrix-org#16037)) * Bump types-jsonschema from 4.17.0.8 to 4.17.0.10. ([\matrix-org#16036](matrix-org#16036)) * Bump types-netaddr from 0.8.0.8 to 0.8.0.9. ([\matrix-org#16035](matrix-org#16035)) * Bump types-opentracing from 2.4.10.5 to 2.4.10.6. ([\matrix-org#16078](matrix-org#16078)) * Bump types-setuptools from 68.0.0.0 to 68.0.0.3. ([\matrix-org#16079](matrix-org#16079)) # -----BEGIN PGP SIGNATURE----- # # iQIzBAABCAAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmTbUOEACgkQLS76LzL7 # 4Efskw/+J4X30PoqSvBWbilr8kTzwNSDXrkefYXR2sLburgowCyuAtKtCdbvnZUX # 3KRwii5/GDsduXiNY836oRxO/KWE43b1ce9C9qJM7V6NmgkJBgHRvnh69wdlmBqt # 6b6TQHoEByYS7yK90+QsRm1Bqrw7eoVO9oxcZ+4lb7Mjswf491Pga8kFJqdvjtTX # UXo4vAqYyP6Yn7sUrQmXy0N8gZ5ZFHhZEvZZ8+iEsNjPO468cSVGq8/iPB1EwBm2 # nbfZWMDnD2p7plJezXOPEBxnVR3RrWbCbK08SiiNMcQynCvBgAUfkd3GnsO726jb # 19i8p6tjuj2r41UgqYCTG2i2ij6uJquA/qq3rIiVNQVKG9aPHQ8hJfu9XOdEvaJe # Je9H6QFrU/hR640tFvb5Hdc/4ThabvtC5xgl4ZGT6y6I0s5LNwk8fJiz3sDFt0i1 # XKsqGVemBGopZnwjQIPFJaHjPT7of33PXLE/hf1vX+oXU/6MNbFYkDLY9nnnQeOx # 0GEbiYaxrj8SfxNmEykMLNCfxwJ719cSR1q8vPYn6r6TOS1pJMV0SgciXoaQ/VW6 # WlRpZolvXYSye34JW8Rg4ojAz0oYfJ2IiUpwY7eSEq4DtuosTjEECKrgB8DLqKlM # +qDd4/yeqVN5/sui5oGsR71aTMy/jdnzqmdmuFvsSwz9/7PfMEU= # =FWp5 # -----END PGP SIGNATURE----- # gpg: Signature made Tue Aug 15 11:18:09 2023 BST # gpg: using RSA key F124520CEEE062448FE1C8442D2EFA2F32FBE047 # gpg: Can't check signature: No public key # Conflicts: # .github/workflows/docker.yml # CHANGES.md # Cargo.lock # debian/changelog # docs/upgrade.md # flake.lock # poetry.lock # pyproject.toml # rust/src/push/base_rules.rs # synapse/events/utils.py # synapse/handlers/pagination.py # synapse/http/site.py # synapse/rest/client/devices.py # synapse/storage/databases/main/roommember.py # synapse/storage/schema/__init__.py # tests/rest/client/test_devices.py # tests/rest/client/test_redactions.py
Updated Inline SQL queries using boolean parameters according to SQLite 3.27.
Fix #15515
Pull Request Checklist
EventStore
toEventWorkerStore
.".code blocks
.(run the linters)