Allow setting non-string typed values in set_properties#504
Allow setting non-string typed values in set_properties#504Fokko merged 7 commits intoapache:mainfrom
set_properties#504Conversation
set_properties
HonahX
left a comment
There was a problem hiding this comment.
This looks great! @kevinjqliu . It adds lots of convenience when setting properties through transaction.
pyiceberg/table/__init__.py
Outdated
| updates: Dict[str, str] | ||
| updates: Properties | ||
| # validators | ||
| transform_properties_dict_value_to_str = field_validator('updates', mode='before')(transform_dict_value_to_str) |
There was a problem hiding this comment.
| transform_properties_dict_value_to_str = field_validator('updates', mode='before')(transform_dict_value_to_str) | |
| @field_validator('updates', mode='before') | |
| def transform_properties_dict_value_to_str(cls, properties: Properties) -> Dict[str, str]: | |
| return transform_dict_value_to_str(properties) |
Shall we reformat the validator to use decorator? This aligns with the way we define model validators in metadata.py and also the example in Pydantic doc. WDYT?
There was a problem hiding this comment.
looks much cleaner, thank you!
bda7fdb to
79a5954
Compare
79a5954 to
9fe9cff
Compare
|
rebased after #503 |
HonahX
left a comment
There was a problem hiding this comment.
Sorry for being late. Overall LGTM. Just have one small comment regarding the type.
BTW, could you please update the same field validator in metadata to use the decorator @field_validator?
https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/metadata.py#L224 Thanks!
pyiceberg/table/__init__.py
Outdated
| class SetPropertiesUpdate(TableUpdate): | ||
| action: TableUpdateAction = TableUpdateAction.set_properties | ||
| updates: Dict[str, str] | ||
| updates: Properties |
There was a problem hiding this comment.
I think this one should be Dict[str, str] because at the time when the model is constructed, the before validator has converted all values to str
Fokko
left a comment
There was a problem hiding this comment.
Thanks @kevinjqliu for working on this, and @HonahX for the review!
Requires #503 to be merged first, extension of #502.
This PR modifies
Transaction'sset_propertiesAPI to accept table properties as typeDict[str, Any], either as dictionary or kwargs.For example,
This is done by reusing the field validator defined in #469. Setting
Noneproperty value is disallowed since the transformation will change None to str "None" which is unintuitive.Note
The Iceberg spec stores table properties as strings. That is why when setting a property value as int, reading it back produces a string.
Use
PropertyUtil.property_as_intto return property value as int