You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In Kafka and Fileset, we may need a table schema to deserialize data. We can manage the external schema registry in Gravitino.
Motivation
Manage the table schema in Gravitino.
Using SQL to read the Kafka topic or Fileset when binding the table schema to the topic or fileset.
Describe the solution
We can introduce a TableSchemaCatalog to manage the TableSchema.
We can bind a table schema such as catalog.schema.table-schema to a topic or fileset when needed. So we can get the table schema from the external schema registry. We can also add a schema registry managed by gravitino, so we can directory save the table schema to the gravitino metastore.
Additional context
No response
The text was updated successfully, but these errors were encountered:
I think it is a good idea to manage Schema as a resource at the same level as Table/Fileset/Messaging. In this way, we can distinguish between Managed Schema (data type is based on Gravitino) and External Schema (data type is based on the existing external Schema Registry or other systems). Then, in resources that require a specific Schema (such as some Filesets), we can bind a Schema to it. When obtaining Fileset metadata, we will also obtain the corresponding Schema and use it in some clients.
It's a bit strange that "schema" is an entity. Theoretically, the entity maps a data object, whereas "schema" is binding to the entity. We should think more about how to support this scenario.
@jerryshao Yes. But we can manage the 'schema' in gravitino for better binding. We can locate the schema with catalog.schema.table-schema, it's easier to bind it to a data object.
Describe the feature
In Kafka and Fileset, we may need a table schema to deserialize data. We can manage the external schema registry in Gravitino.
Motivation
Describe the solution
We can introduce a TableSchemaCatalog to manage the TableSchema.
We can bind a table schema such as
catalog.schema.table-schema
to a topic or fileset when needed. So we can get the table schema from the external schema registry. We can also add a schema registry managed by gravitino, so we can directory save the table schema to the gravitino metastore.Additional context
No response
The text was updated successfully, but these errors were encountered: