[#2695] feat(doc): Add docs for fileset catalog #2781
Conversation
|
@coolderli @xloya would you please help to review? Thanks. |
|
@jerryshao Do we need to introduce how to use the Fileset in the Spark engine? In addition, I have already tested the Tensorflow and submitted an MR: tensorflow/io#1970. After #2779 is resolved, we can support tensorflow. I think we can add a doc like https://help.aliyun.com/zh/hdfs/using-tensorflow-on?spm=a2c4g.11186623.0.i6. What do you think? |
I would suggest to have another doc about gvfs and add Spark, TF related things there. |
|
|
||
| FilesetCatalog filesetCatalog = catalog.asFilesetCatalog(); | ||
| NameIdentifier[] identifiers = | ||
| filesetCatalog.listFilesets(Namespace.ofFileset("metalake", "catalog", "schema")); |
There was a problem hiding this comment.
I think there's another issue. The metalake in Namespace seems redundant. The new GravitinoClient, we have declared the name of the current metalake. It is not related to this MR. Never mind.
There was a problem hiding this comment.
Yeah, it is unrelated, and we are working on the client refactoring things.
|
@jerryshao Left some comments. Overall, it looks good to me. |
|
@shaofengshi would you please also check the java client part? Thanks. |
| tabular data and others in Gravitino with a unified way. | ||
|
|
||
| After fileset is created, users can easily access, manage the files/directories through | ||
| Fileset's identifier, without needing to know the physical path of the managed datasets. Also, with |
There was a problem hiding this comment.
Maybe of the managed datasets is not necessary.
There was a problem hiding this comment.
I feel like it is still necessary, it means that the dataset is managed by Gravitino, so users don't need to know the physical path. Fro unmanaged dataset, users still need to know the physical path before visiting the dataset.
What changes were proposed in this pull request?
This PR proposes to add docs for fileset catalog.
Why are the changes needed?
Fix: #2695
Does this PR introduce any user-facing change?
No.
How was this patch tested?
No.