Add {_snowflake_id} wildcard support to object storage #789
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Cheap alternative to #697 The |
arthurpassos
changed the title
arthurpassos
commented
May 21, 2025
src/Core/Settings.cpp
Outdated
| @@ -5951,6 +5951,9 @@ This only affects operations performed on the client side, in particular parsing | |||
| Normally this setting should be set in user profile (users.xml or queries like `ALTER USER`), not through the client (client command line arguments, `SET` query, or `SETTINGS` section of `SELECT` query). Through the client it can be changed to false, but can't be changed to true (because the server won't send the settings if user profile has `apply_settings_from_server = false`). | |||
|
|
|||
| Note that initially (24.12) there was a server setting (`send_settings_to_client`), but latter it got replaced with this client setting, for better usability. | |||
| )", 0) \ | |||
| DECLARE(Bool, object_storage_treat_key_wildcard_as_star, false, R"( | |||
There was a problem hiding this comment.
Three options here:
- Off by default
- On by default
- No setting at all, default behavior
ianton-ru
reviewed
May 22, 2025
| @@ -373,21 +374,35 @@ void StorageObjectStorage::read( | |||
| if (update_configuration_on_read) | |||
| configuration->update(object_storage, local_context); | |||
|
|
|||
| if (partition_by && configuration->withPartitionWildcard()) | |||
| auto config_clone = configuration->clone(); | |||
There was a problem hiding this comment.
We make a clone every time, but actually change only in specific cases.
May be make here a smart_ptr on original config, and make a clone only when required?
| @@ -659,13 +674,29 @@ bool StorageObjectStorage::Configuration::withPartitionWildcard() const | |||
| || getNamespace().find(PARTITION_ID_WILDCARD) != String::npos; | |||
| } | |||
|
|
|||
| bool StorageObjectStorage::Configuration::withSnowflakeIdWildcard() const | |||
| { | |||
| static const String PARTITION_ID_WILDCARD = "{_snowflake_id}"; | |||
arthurpassos
commented
May 22, 2025
| @@ -5951,6 +5951,9 @@ This only affects operations performed on the client side, in particular parsing | |||
| Normally this setting should be set in user profile (users.xml or queries like `ALTER USER`), not through the client (client command line arguments, `SET` query, or `SETTINGS` section of `SELECT` query). Through the client it can be changed to false, but can't be changed to true (because the server won't send the settings if user profile has `apply_settings_from_server = false`). | |||
|
|
|||
| Note that initially (24.12) there was a server setting (`send_settings_to_client`), but latter it got replaced with this client setting, for better usability. | |||
| )", 0) \ | |||
| DECLARE(Bool, object_storage_treat_key_related_wildcards_as_star, false, R"( | |||
There was a problem hiding this comment.
Three options here:
- make it the default behavior, not behind a setting
- make the setting on by default
- keep it off by default
36 tasks
|
From the usability point of view ChatGPT suggest to consider using |
That sounds like a terrible idea |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add {_snowflake_id} wildcard support to object storage paths. Upon writing, ClickHouse will generate a snowflakeid on the fly and replace the wildcard. This will help us with parallel and concurrent writes to object storage.
Also introduce a new setting
object_storage_treat_key_related_wildcards_as_starto allow symmetrical reads & writes using a single table. Why is it needed? Consider the following:CREATE TABLE ... s3('path_to_table_root/**.parquet')Ok, we can select from it, but how do we write? How do we name the files? In which directory?
Therefore, we introduced the snowflake id.
CREATE TABLE ... s3('path_to_table_root/{_snowflake_id}.parquet')- we can now write to it because we know the file location and we know how to name it.But how do we read now? The path isn't globbed anymore. That's what the setting is for.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add {_snowflake_id} wildcard support to object storage paths. Also add a new setting
object_storage_treat_key_related_wildcards_as_starto allow symmetrical reads & writes using a single table.Documentation entry for user-facing changes