[HUDI-7069] Optimize metaclient construction and include table config…

[majian1998]

In the current implementation of run multi tables services, the clustering task and compaction task both build metaclient repeatedly for each table, causing additional overhead. To reduce this overhead, we extract the construction of metaclient and only construct it once for each table, passing it as a parameter to the corresponding task.

At the same time, when running multi tables services, the write config lacks some information from the table config, such as the table name. This leads to empty strings when retrieving the table name in certain situations. For example, when configuring the prefix for metrics, if not specified, the table name is used as the prefix. However, in the current situation, without the table config, it's impossible to differentiate the metrics of different tables, resulting in an empty prefix. By adding the table config to the write config beforehand, we can obtain all the configuration information in the subsequent write config step.
Taking a metrics example, it would be displayed as follows before the fix:
".replacecommit.totalFilesInsert":"2"
After the fix, it would be displayed as follows:
"test_tb.replacecommit.totalFilesInsert":"2"

Additionally, we made a small modification by removing the redundant construction of metaclient in the clusteringJob's constructor.

Change Logs

We now construct metaclient only once per table and add table config to the write config to obtain all necessary information. Furthermore, unnecessary construction of metaclient in the clusteringJob's constructor has been removed.

Impact

None

Risk level (write none, low medium or high below)

None

Documentation Update

None

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

Be caution that the timeline should be refreshed each time for compaction when metaClient is reused.

[majian1998]

This‘s right. In the existing code, the metaclient is reloaded during clustering, and a new metaclient is created during compaction. A better implementation would be to have consistent behavior for compaction as well. The metaclient should be reloaded at the point of creation.

private int doScheduleAndCluster(JavaSparkContext jsc) throws Exception {
LOG.info("Step 1: Do schedule");
metaClient = HoodieTableMetaClient.reload(metaClient);
...
}

� private int doCompact(JavaSparkContext jsc) throws Exception {
 ...
         if (StringUtils.isNullOrEmpty(cfg.compactionInstantTime)) {
        HoodieTableMetaClient metaClient = UtilHelpers.createMetaClient(jsc, cfg.basePath, true);...
}

I will make modifications here.

[danny0405]

both build metaclient repeatedly for each table, causing additional overhead. To reduce this overhead, we extract the construction of metaclient and only construct it once for each table

In my opinion, the main overhead of instantiating meta client is the loading and decoding of the hoodie instants which include file listing and deciphering of the commit metadata. The refreshing of hoodie instants for each round of table service scheduling is necessary becase the latest metadata is required. So I don't think we gains much by resuing the instance.

At the same time, when running multi tables services, the write config lacks some information from the table config, such as the table name. This leads to empty strings when retrieving the table name in certain situations.

If there are some configuration inconsistencies, we can just fix it.

[majian1998]

@danny0405 : You are right. The original intention of passing the meta client was not actually for performance optimization, but rather to obtain the table config when unifying the table config and write config early on. However, during the coding process, it was discovered that there were cases of redundant meta client construction in this code. Therefore, this part was also modified, simplifying the code and bringing about a small performance benefit.

[stream2000]

Should we use table config to overwrite those configs already set in write configs by user? Not sure which one have a higher priority here. @danny0405 What do you think?

[majian1998]

This can indeed lead to a priority issue here. A simple solution is to use addNecessaryTableConfigToWriteConfig to add the necessary parameters.

[danny0405]

Are you refering to the per-table options? We should consider write config options with higher priority always.

[hudi-bot]

CI report:

• 4a35dfe Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build