[#1244] feat(API, meta): Add basic API and metadata design for file catalog #1487

jerryshao · 2024-01-15T06:40:24Z

What changes were proposed in this pull request?

This PR proposes to add the Fileset metadata support in Gravitino. Include the API, metadata design and implementation.

Why are the changes needed?

This is the first work of support file-based non-tabular data management in Gravitino.

Fix: #1244

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

Add some UTs to test metadata, API test will be added when the implementation is achieved.

yuqi1129 · 2024-01-15T07:02:07Z

api/src/main/java/com/datastrato/gravitino/file/File.java

+  }
+
+  /** @return Name of the file object. */
+  String name();


The name of the File entity is it's full path or just the name of it?

Just the name.

The virtual path is derived from the name identifier, and the physical path is a must-set property. About the physical path, I'm still thinking about using a property or a must-set field, I'm not so sure about which way is a good way.

I'd like to use a must-set field, just like the warehouse location. If we use a property to keep the physical path, where should the file storage if we forget to set it?

qqqttt123 · 2024-01-15T07:07:23Z

I think we can corporate with OpenDAL community after we implemented a file catalog. https://github.com/apache/incubator-opendal If we want to implement Rust client. It may be useful, too.

qqqttt123 · 2024-01-15T07:20:06Z

In general, we have three points to ensure.

Do we need the concept of directory? Do you need the api of directories?
For Hadoop compatible file apis, do we need the truncate api? Do we need to support soft-link? Do we need read/write api? It may be useful to record audit information for writing.
Do we need managed file system? If so, maybe we should support upload api. Databricks implement upload api. https://docs.databricks.com/api/workspace/dbfs/put

jerryshao · 2024-01-15T07:30:12Z

In general, we have three points to ensure.

Do we need the concept of directory? Do you need the api of directories?

File is a virtual concept from metadata perspective, it will refer to both physical file or directory, I don't want additional directory API, which seems not necessary.

For Hadoop compatible file apis, do we need the truncate api? Do we need to support soft-link? Do we need read/write api? It may be useful to record audit information for writing.

This is not the scope of this PR. And from my random thinking, I think truncate is not necessary as object store doesn't support it natively. Also for soft-link. We don't have to implement all the posix FS API once for all.

Do we need managed file system? If so, maybe we should support upload api. Databricks implement upload api. https://docs.databricks.com/api/workspace/dbfs/put

Upload is not an API, it is just a read and write, we can do it but it will be supported naturally when HCFS interface is implemented.

qqqttt123 · 2024-01-15T07:36:56Z

api/src/main/java/com/datastrato/gravitino/file/FileCatalog.java

+   * @param ident A file identifier.
+   * @return true If the file is dropped, false the file did not exist.
+   */
+  boolean dropFile(NameIdentifier ident);


Do we need the api purgeFile? It may be a little weird to use the name purge. You should get the point of mine. It means that we can delete the file of external system.

This API will drop all files underlying the path, right? Do we support the subdirectory? If not, how can we deal with the TTL? for example I want to drop the files in /metalake/catalog/schema/my_file/date=20201010

No. I don't plan to add purgeFile API, this seems not so necessary for now. We can add it when we really have this requirement.

This API will drop all files underlying the path, right? Do we support the subdirectory? If not, how can we deal with the TTL? for example I want to drop the files in /metalake/catalog/schema/my_file/date=20201010

Yes, we can support it using FS API as I mentioned in the design doc.

qqqttt123 · 2024-01-15T07:44:41Z

api/src/main/java/com/datastrato/gravitino/file/FileChange.java

+   * @param newName The new name of the file.
+   * @return The file change.
+   */
+  static FileChange rename(String newName) {


Does this api will call underlying filesystem api rename? Will it modify the physical path property?

No, it will not. It will only manage the file metadata, not the physical file path.

Is there an API to modify the physical path? We may want to change the cloud storage to another one but keep the virtual path unchanged. Then our user job will not need to recompile anymore.

I was thinking about it a bit. Yearh, it is useful, but we can add it later, this PR is mainly for adding the skeleton, we can always add it later on.

qqqttt123 · 2024-01-15T07:51:56Z

meta/src/main/proto/gravitino_meta.proto

+message File {
+  enum Format {
+    CSV = 0;


UNKNOWN = 0 may be better. You can see https://blog.51cto.com/u_16162321/6496255

jerryshao · 2024-01-15T11:58:47Z

@coolderli Can you please take a review when you have time? Thanks.

coolderli · 2024-01-15T11:20:37Z

api/src/main/java/com/datastrato/gravitino/file/File.java

+  }
+
+  /** @return Name of the file object. */
+  String name();


I'd like to use a must-set field, just like the warehouse location. If we use a property to keep the physical path, where should the file storage if we forget to set it?

coolderli · 2024-01-15T11:38:07Z

api/src/main/java/com/datastrato/gravitino/file/File.java

+ * of a file object. A catalog implementation with {@link FileCatalog} should implement this
+ * interface.
+ */
+public interface File extends Auditable {


I'm not very clear about the File. Is the File like a Volume on Databricks?
How should I understand a file in a schema? It only one real file or one directory

I think the File is ambiguous. There may be some misunderstandings during communication

The File metadata concept is similar to a Volume in UC, it can be referred to a physical file or directory.

Yes, it is ambiguous, but I cannot figure out a better name, do you have a better suggestion?

How about using fileset?

yeah, I think the fileset is much better than file

coolderli · 2024-01-15T11:39:18Z

api/src/main/java/com/datastrato/gravitino/file/FileCatalog.java

+   * @param ident A file identifier.
+   * @return true If the file is dropped, false the file did not exist.
+   */
+  boolean dropFile(NameIdentifier ident);


This API will drop all files underlying the path, right? Do we support the subdirectory? If not, how can we deal with the TTL? for example I want to drop the files in /metalake/catalog/schema/my_file/date=20201010

coolderli · 2024-01-15T11:46:35Z

api/src/main/java/com/datastrato/gravitino/file/FileChange.java

+   * @param newName The new name of the file.
+   * @return The file change.
+   */
+  static FileChange rename(String newName) {


Is there an API to modify the physical path? We may want to change the cloud storage to another one but keep the virtual path unchanged. Then our user job will not need to recompile anymore.

coolderli · 2024-01-15T12:02:20Z

api/src/main/java/com/datastrato/gravitino/file/File.java

+  }
+
+  /** @return the format of the underlying filesets that is managed by this file object. */
+  Format format();


Should we set the format for the File? If we did, it will assume all files in a File must be the same format.
is there possible different file format on different directory?

For example, we use orc as our current file format, but one day we want to change it to parquet. What should I do with the history files?

I delete the format, seems it is not so useful and brings inconvenience for users.

xloya · 2024-01-16T02:04:46Z

api/src/main/java/com/datastrato/gravitino/file/Fileset.java

+   *
+   * @return The storage location of the fileset object.
+   */
+  String storageLocation();


Does need a virtual path property here? I saw that there is a virtual path in the volume of Databricks, and resources can be accessed through this virtual path:

Does need a virtual path property here? I saw that there is a virtual path in the volume of Databricks, and resources can be accessed through this virtual path:

After thinking about it again, if the conversion from virtual path to metalake.catalog.schema.fileset can be realized in Proxy FS, it seems that it is not necessary.

Virtual path is generated from NameIdentifier, which is "metalake.catalog.schema.fileset", so the virtual path will be "/metadata/catalog/schema/fileset".

qqqttt123 · 2024-01-16T07:25:45Z

Another discussion. Do we want to have some another data optimization in the file catalog? For example, we can transfer some cold data from the HDFS to S3. we can use more effective compression codec. We can clean some temporary files.

jerryshao · 2024-01-16T07:28:06Z

Another discussion. Do we want to have some another data optimization in the file catalog? For example, we can transfer some cold data from the HDFS to S3. we can use more effective compression codec. We can clean some temporary files.

This seems out of the scope of current file catalog. But I think it is doable with file catalog. What we need is to have several tools around file catalog.

jerryshao · 2024-01-16T11:00:53Z

@qqqttt123 @yuqi1129 @xloya @coolderli Can you please review again?

xloya · 2024-01-16T11:25:13Z

Overall looks nice to me, can not wait to see next pr :)

coolderli · 2024-01-16T11:29:15Z

LGTM!

coolderli · 2024-01-16T11:33:39Z

Another discussion. Do we want to have some another data optimization in the file catalog? For example, we can transfer some cold data from the HDFS to S3. we can use more effective compression codec. We can clean some temporary files.

This seems out of the scope of current file catalog. But I think it is doable with file catalog. What we need is to have several tools around file catalog.

Maybe later we can provide some services to govern tables and files on gravitino, for example, manage compaction、expire snapshots, etc.

qqqttt123 · 2024-01-16T11:58:04Z

meta/src/main/proto/gravitino_meta.proto

+  optional string comment = 3;
+  Type type = 4;
+  optional string storage_location = 5;


Should we provide a default storage_location if storage_location is absent?

We will use the catalog/schema's location to figure out a location for fileset if it is not set.

qqqttt123

LGTM. thanks @jerryshao @coolderli @xloya

jerryshao · 2024-01-16T12:12:06Z

Thanks all for your comments, greatly appreciated.

jerryshao requested review from yuqi1129 and mchades January 15, 2024 06:40

jerryshao self-assigned this Jan 15, 2024

jerryshao mentioned this pull request Jan 15, 2024

[Subtask] Investigate on how non-tabular data is managed #1284

Closed

yuqi1129 reviewed Jan 15, 2024

View reviewed changes

qqqttt123 reviewed Jan 15, 2024

View reviewed changes

jerryshao requested a review from diqiu50 January 15, 2024 11:58

coolderli reviewed Jan 15, 2024

View reviewed changes

xloya reviewed Jan 16, 2024

View reviewed changes

jerryshao added 6 commits January 16, 2024 14:14

Add basic API and metadata design for file catalog

0428f55

Address the comments

8e2d153

Improve the doc

8712484

change to use fileset

27e9f19

Style changes

457c11d

Slightly change the comment

59d978a

jerryshao force-pushed the issue-1244 branch from fc550e4 to 59d978a Compare January 16, 2024 06:15

qqqttt123 reviewed Jan 16, 2024

View reviewed changes

qqqttt123 approved these changes Jan 16, 2024

View reviewed changes

jerryshao merged commit f0129d4 into apache:main Jan 16, 2024
12 checks passed

[#1244] feat(API, meta): Add basic API and metadata design for file catalog #1487

[#1244] feat(API, meta): Add basic API and metadata design for file catalog #1487

Conversation

jerryshao commented Jan 15, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qqqttt123 commented Jan 15, 2024 • edited Loading

qqqttt123 commented Jan 15, 2024

jerryshao commented Jan 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryshao Jan 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryshao commented Jan 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qqqttt123 commented Jan 16, 2024

jerryshao commented Jan 16, 2024

jerryshao commented Jan 16, 2024

xloya commented Jan 16, 2024

coolderli commented Jan 16, 2024

coolderli commented Jan 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qqqttt123 left a comment

Choose a reason for hiding this comment

jerryshao commented Jan 16, 2024

jerryshao commented Jan 15, 2024 •

edited

Loading

qqqttt123 commented Jan 15, 2024 •

edited

Loading

jerryshao Jan 15, 2024 •

edited

Loading

coolderli commented Jan 16, 2024 •

edited

Loading