Skip to content

Move encryption and decryption configuration options into a separate crypto namespace #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

adamreeve
Copy link

This moves the file_encryption_properties and file_decryption_properties configuration options out of the global Parquet options and into a new config field inside TableParquetOptions.

This allows using :: as the separator for column specific keys and metadata by avoiding the conflict with the logic for TableParquetOptions::column_specific_options, and means we can make use of the config_namespace_with_hashmap macro to simplify the ConfigField implementations.

It also simplifies the reading configuration by allowing encryption options to be set in the ParquetReadOptions rather than needing to modify the session state.

I think this also makes more sense conceptually, as these options are likely to be table/file specific rather than something you'd want to set for a whole session.

@@ -252,6 +253,8 @@ pub struct ParquetReadOptions<'a> {
pub schema: Option<&'a Schema>,
/// Indicates how the file is sorted
pub file_sort_order: Vec<Vec<SortExpr>>,
/// Properties for decryption of Parquet files that use modular encryption
pub file_decryption_properties: Option<FileDecryptionProperties>,
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is something I wasn't 100% sure about. Should this use FileDecryptionProperties or ConfigFileDecryptionProperties? Given this doesn't need to be serializable here it seemed nicer to just use FileDecryptionProperties from the Parquet crate. Later in to_listing_options these get converted to ConfigFileDecryptionProperties though.

I'm concerned that this might be a bit misleading to users who could expect to be able to use any FileDecryptionProperties instance here, but this won't work if the decryption properties use a key retriever as that will be lost when converting to ConfigFileDecryptionProperties.

I think that's probably OK as long as this is well documented and results in a helpful error message if misused.

@corwinjoy corwinjoy merged commit 219d0b3 into corwinjoy:parquet_encryption Jun 18, 2025
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants