fix statistics

[xuanyu66]

What problem does this PR solve?

close #2573
In #2259, TiResolutionRule was deleted, which cause statistics would not be collected.
Thus in https://github.com/pingcap/tispark/pull/2300/files, ENABLE_AUTO_LOAD_STATISTICS was considered to be useless and deprecated.

What is changed and how it works?

• add a new ResolutionRule to collect statistics if ENABLE_AUTO_LOAD_STATISTICS is true.
• add documents to explain ENABLE_AUTO_LOAD_STATISTICS.
• fix the logicalPlan result.

Check List

Tests

• Unit test

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation
• Need to be included in the release note

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• Daemonxiao
• shiyuhang0

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[github-actions]

license-eye has totally checked 485 files.

Valid	Invalid	Ignored	Fixed
484	1	0	0

Click to see the invalid file list

• core/src/main/scala/org/apache/spark/sql/catalyst/rule/TiStatisticsRuleFactory.scala

[xuanyu66]

/run-all-tests

[xuanyu66]

/run-all-tests

[xuanyu66]

/run-all-tests

[xuanyu66]

/run-all-tests

[xuanyu66]

/run-all-tests

[xuanyu66]

/run-all-tests tidb=v6.1.0 tikv=v6.1.0

[xuanyu66]

/run-all-tests

[xuanyu66]

/run-all-tests

[xuanyu66]

/run-all-tests tidb=v6.1.0 tikv=v6.1.0

[xuanyu66]

/run-all-tests tidb=v6.1.0 tikv=v6.1.0

[wfxxh]

Is there any cache in tispark ? When i use this PR to run my process,the tikv read task number is correct,same as table region number.If I only process one table it is success.But if I loop process several tables it cause executor OOM。

loop process several tables:

process the oom table when the loop is processed separately

my process

[xuanyu66]

@wfxxh Yes, the cache is in com.pingcap.tispark.statistics.StatisticsManager

[wfxxh]

@xuanyu66 But the reasion when loop read tikv cause OOM is not find,It is may be a bug

[xuanyu66]

@wfxxh Can you check if OOM will happen when using the released version(v3.1.1) and the master version?
If it only happens in this PR, I will hold on.

[wfxxh]

@xuanyu66 I have checked the v2.5.1 ,v3.1.1 and the master version.
The v2.5.1 version read task number is the same as this PR and equal to the table region number,but will cause OOM when loop read several table.
The v3.1.1 and the master version read task number is not correct,and equal to the table index region number.
The v2.5.1 version and the PR version is fast than v3.1.1 and master.
I think it is very strange,why read single table and exit spark job not OOM,loop several to read cause OOM.

[xuanyu66]

@wfxxh So 2.5.1 and this PR will OOM while 3.1.1 and master don't have the OOM problem, am I right?

[wfxxh]

@xuanyu66 Yes,but even it cause OOM ,there will start a new executor pod,the spark job final status is success,and it was faster than the 3.1.1 version

[xuanyu66]

@wfxxh Can we dump the memory snapshot to analyze it?

[wfxxh]

@xuanyu66 Yes I am trying to do it ,but I am run spark on k8s , I should mount a storage.

I am so sorry , I can not do it ,when the executor pod deleted ,the volumn deleted too.

[xuanyu66]

@wfxxh Can you increase your memory, and hold on for 10 minutes when the task is running. And dump the memory to analyze

[wfxxh]

@xuanyu66 How to hold on the process? sleep the thread? But the time when i sleep it may be not oom

[xuanyu66]

@wfxxh use "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="c:\temp\dump2.hprof" "
and use the hostpath

[wfxxh]

@xuanyu66 Yes,I have try this method ,but when the executor pod OOM and deleted,it will start a new pod ,and this will clear the hostpath dir.

[wfxxh]

@xuanyu66 I think you can merge this PR,because even it may be cause the executor oom ,but the final status is success,and it is faster

[xuanyu66]

@wfxxh You can simply use "-XX:HeapDumpPath="c:\temp-${randomstring}\dump2.hprof"

[wfxxh]

@xuanyu66 I do not konw what is your meaning. I say when the pod oom deleted ,and restart a new one ,the hostpath in host machine will be deleted too.
I will run the process on spark standalone cluster to test whether it is the k8s reasion

[xuanyu66]

@wfxxh it's strange, normally hostpath will not be deleted.
please check persistentVolumeReclaimPolicy, maybe you are using recycle

[xuanyu66]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 5176f13

[ti-srebot]

cherry pick to release-release failed

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #2588.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #2589.

[xuanyu66]

@wfxxh I have merged this PR and we will discuss the OOM problem here #2587