iceberg table properties are saved in table metadata's properties field

[kevinjqliu]

Apache Iceberg version

None

Please describe the bug 🐞

Context #2013 (comment)

Because Table.properties returns the table metadata's properties field

iceberg-python/pyiceberg/table/__init__.py

Lines 1131 to 1134 in a67c559

	@property
	def properties(self) -> Dict[str, str]:
	"""Properties of the table."""
	return self.metadata.properties

adding/removing the table properties is modifying the table metadata's properties field.

This is not the recommended usage of the table metadata's properties field

properties	A string to string map of table properties. This is used to control settings that affect reading and writing and is not intended to be used for arbitrary metadata. For example, commit.retry.num-retries is used to control the number of commit retries.

Catalogs should handle figuring out where to properly save the table's properties. For example, Hive catalog should save the table's properties in the hive table parameters.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[kevinjqliu]

Transaction.set_properties creates a SetPropertiesUpdate table update which is then applied to the table metadata here

iceberg-python/pyiceberg/table/update/__init__.py

Lines 304 to 313 in a67c559

	@_apply_table_update.register(SetPropertiesUpdate)
	def _(update: SetPropertiesUpdate, base_metadata: TableMetadata, context: _TableMetadataUpdateContext) -> TableMetadata:
	if len(update.updates) == 0:
	return base_metadata
	properties = dict(base_metadata.properties)
	properties.update(update.updates)
	context.add_update(update)
	return base_metadata.model_copy(update={"properties": properties})

[kevinjqliu]

We should have a way to override the behavior for applying SetPropertiesUpdate

[kevinjqliu]

cc @Fokko wdyt?

[summermousa-vendia]

This would be a nice thing to fix because I am using pyiceberg with glue and I would want to update the table properties so that I can update the glue catalog's metadata file pointer manually.

[Fokko]

@kevinjqliu It makes sense to me to keep those properties in sync 👍