[Feature] Lance's snapshot should support store summary property

[xx789633]

While integrating Lance with Fluss, we want to store some summary property to record the current Fluss tiering status when Fluss commit a snapshot in Lance . But Lance does not support this feature, I think we can introduce a summary property into snapshot just like iceberg does.

paimon: apache/paimon#5649
iceberg: apache/iceberg#4956

[jackye1995]

We discussed about this in #4078 (comment)

should be a quick feature to add if you are interested, or I don't know if @chenghao-guo is planning to include that in his original PR

[xx789633]

Cool. Please expose this feature in Java API.

[chenghao-guo]

We discussed about this in #4078 (comment)

should be a quick feature to add if you are interested, or I don't know if @chenghao-guo is planning to include that in his original PR

Hi @jackye1995 thanks for the suggestion! Sorry @cxxwang789 , I'm tied up with other priorities and won't be able to dedicate time to this feature immediately. Assuming we can hold off until late July, I’ll have capacity to add this and progress the work.

Previously I was working on a simple design by setting a property in struct Transaction.

pub struct Transaction {
     ...............
    ///intended to add an extra_metadata to store the properties for a transaction
    **pub extra_metadata: Option<HashMap<String, String>>,**
}

Then bypass the write_dataset/add_columns optional parameter to transaction and stored in txn file. At that previous PR, I was only considering adding a simple commit_message, which is not general.

To meet your @cxxwang789 requirement, I believe the below works need to be done.

1. add the extra_metadata in Transaction and allows for parameter bypass as mentioned, currently a lot of rust methods has used Transaction, so there are still some works to be done to prevent making breaking changes.
2. expose in Java API. Regarding this , what kind of java API we need? for example, do we call Dataset commit() in Java for this integration? If so, I can directly expose using existing commit() method. Feel free to discuss it here if it doesn't meet the requirement.

I’ll circle back when bandwidth allows by the end of this month. I hope that don’t let this block the progress of other PRs. Feel free to add it directly if it’s urgent!

[leonardBang]

We discussed about this in #4078 (comment)

should be a quick feature to add if you are interested, or I don't know if @chenghao-guo is planning to include that in his original PR

Cool, looking forward to this feature.

[xx789633]

Hi @chenghao-guo , I think the paimon code and iceberg code set a good example:
paimon: https://github.com/apache/paimon/blob/master/paimon-core/src/test/java/org/apache/paimon/operation/FileStoreCommitTest.java#L1027

CommitMetadata in iceberg: https://iceberg.apache.org/docs/latest/spark-configuration/#write-options

[chenghao-guo]

Hi @chenghao-guo , I think the paimon code and iceberg code set a good example: paimon: https://github.com/apache/paimon/blob/master/paimon-core/src/test/java/org/apache/paimon/operation/FileStoreCommitTest.java#L1027

CommitMetadata in iceberg: https://iceberg.apache.org/docs/latest/spark-configuration/#write-options

Got it, thanks. These two examples are convincing! I will expose this transaction extra property to Java write and commit Transaction API. (btw, for ManifestConfig, lance already has this in java Dataset updateConfig(), that manifest config was designed to be inherited) .

For your property information that only describe a specific transaction and intends not to be inherited, I will add this extra_metadata in Transaction. The estimated completion will be the end of July once I have bandwidth. I will update here once done. I hope that won't block your too late. Feel free to add it directly if it’s urgent.

[jackye1995]

@chenghao-guo thank you for continuing the effort!

there are still some works to be done to prevent making breaking changes.

I think we don't need to worry too much about preventing making breaking changes, since the release process already has breaking changes policy in place you just need to follow when you add the extra input.

extra_metadata: Option<HashMap<String, String>>

just a nit, I think we don't need to call it extra_. Either properties or summary or metadata should be good.

[chenghao-guo]

got it , I will use properties as suggested

[leonardBang]

@chenghao-guo thank you for continuing the effort!

there are still some works to be done to prevent making breaking changes.

I think we don't need to worry too much about preventing making breaking changes, since the release process already has breaking changes policy in place you just need to follow when you add the extra input.

extra_metadata: Option<HashMap<String, String>>

just a nit, I think we don't need to call it extra_. Either properties or summary or metadata should be good.

+1 for properties naming

[majin1102]

Hi @cxxwang789, I have a question about this scenario.

If there are other engines writing to the underlying lance dataset, let's say:

tx1: fluss flush —> s0
tx2: fluss flush —> s1
tx3: spark write —> s2
tx4: spark write —> s3

How would Fluss read the tiering status when the latest snapshot reaches s3. Does it need to track snapshot histories and find s1 to read it?

[xx789633]

It needs to trace back to the latest snapshot committed by Fluss itself. https://github.com/apache/fluss/blob/main/fluss-lake/fluss-lake-paimon/src/main/java/com/alibaba/fluss/lake/paimon/tiering/PaimonLakeCommitter.java#L160

[chenghao-guo]

The current design for Transaction properties would be
tx1: properties( hashmap<string, string>) stored in txn1 file
tx2: properties( hashmap<string, string>) stored in txn2 file
tx3: properties( hashmap<string, string>) stored in txn3 file
tx4: properties( hashmap<string, string>) stored in txn4 file
Simply listing and filtering could be slow if we need to extract every hashmap. (Initially I used it to store commit messages, simply list without filtering)

I have a plan for this scenerio. Lance already has manifestConfig, which is inherited for every version. So I guess there could be a workaround, by additionally putting the latest fluss snapshot version number in the manifestConfig.

Then, when tracing back the latest snapshot committed by Fluss itself, the process could be

1. look into latest manifestConfig, get the latest version number commited by Fluss
2. based on the fluss version number, navigate to the Transaction properties, and get the properties that fluss needs. How do you think so. I will provide this API in the Transaction properties.

[majin1102]

I have a plan for this scenerio. Lance already has manifestConfig, which is inherited for every version. So I guess there could be a workaround, by additionally putting the latest fluss snapshot version number in the manifestConfig.

Do you mean table_config in manifest file?

[jackye1995]

I have a plan for this scenerio. Lance already has manifestConfig, which is inherited for every version. So I guess there could be a workaround, by additionally putting the latest fluss snapshot version number in the manifestConfig.

Based on this description, it seems like you are basically storing a high watermark. In that case, why not just store this info directly in the config, so you don't need to do the 2 step read? @cxxwang789