Remove TableInfo from server MetadataCache

[swuferhong]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, all the TableInfo is passed to TabletServer by UpdateMetadata RPC which is a big overhead during starting and occupies a lot memory of TabletServer. However, the TableInfo is not necessary for metadata cache and can be avoid for network transporting and memory storing. See comment: #890 (comment)

Solution

No response

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[LiebingYu]

Hi @swuferhong. I'd like to try. Could you please assign it to me?

I took a look at the code. My thought is to change the table_json field in PbTableMetadata from required to optional, and add fields like tableId and schemaId to TableMeta. This is because the metadata RPC call also relies on PbTableMetadata and needs TableInfo. Additionally, this makes it easier to maintain compatibility across different versions.

Do you have any advice about this?

[swuferhong]

Hi @swuferhong. I'd like to try. Could you please assign it to me?

I took a look at the code. My thought is to change the table_json field in PbTableMetadata from required to optional, and add fields like tableId and schemaId to TableMeta. This is because the metadata RPC call also relies on PbTableMetadata and needs TableInfo. Additionally, this makes it easier to maintain compatibility across different versions.

Do you have any advice about this?

I think you cannot directly change the PbTableMetadata from required to optional. Even though this appears to be a change from a strong constraint to a weak one, it would cause incompatibility when lower-version clients parse responses from higher-version servers. Personally, I think we can introduce PbTableMetadataV2, which could be optional. As do that, the json_field in MetadataResponse may looks like:

'''
message MetadataResponse {
optional PbServerNode coordinator_server = 1;
repeated PbServerNode tablet_servers = 2;
repeated PbTableMetadata table_metadata = 3;
repeated PbPartitionMetadata partition_metadata = 4;
repeated PbPartitionMetadataV2 table_metadata_v2 = 5;
}

message PbTableMetadataV2 {
required PbTablePath table_path = 1;
required int64 table_id = 2;
required int32 schema_id = 3;
optional bytes table_json = 4; // mark as optional
repeated PbBucketMetadata bucket_metadata = 5;
required int64 created_time = 6;
required int64 modified_time = 7;
}
'''