*: Build TiDB with PGO

[crazycs520]

What problem does this PR solve?

Issue Number: close #57502

Problem Summary: Build TiDB with PGO

What changed and how does it work?

This PR support builds TiDB with PGO. According to the doc of PGO, go build will detect default.pgo files automatically and enable PGO, so you need to put default.pgo to cmd/tidb-server/ dir to enable PGO.

Watch out, there's a potential performance regression risk here, see golang/go#71400

As you can see from the tests below, the performance of TiDB is improved after compiling with PGO, even with the old (v7.0.0) profile, but the improvement is less, so we need to update the default.pgo profile regularly in the future.

Sysbench Benchmark

Topology

Component	Num	GCP instance
TiKV	3	c2-standard-8
PD	1	c2-standard-8
TiDB	1	c2-standard-16
workload	1	c2-standard-8

• Master: git hash is 5934515
• Master with PGO point-select: build master with PGO, use oltp_point_select's profile
• Master with PGO read-only: build master with PGO, use oltp_read_only's profile
• Master with PGO read-write: build master with PGO, use oltp_read_write's profile
• Master with PGO read-write-v7.0.0: build master with PGO, use profile of v7.0.0 TiDB running oltp_read_write workload.

workload	thread	version	QPS	diff
oltp_point_select	100	Master	83110
oltp_point_select	100	Master with PGO point-select	88691	6.72%
oltp_point_select	100	Master with PGO read-only	88675	6.70%
oltp_point_select	100	Master with PGO read-write	88658	6.68%
oltp_point_select	100	Master with PGO read-write-v7.0.0	85221	2.54%
oltp_read_only	100	Master	60680
oltp_read_only	100	Master with PGO point-select	64110	5.65%
oltp_read_only	100	Master with PGO read-only	64964	7.06%
oltp_read_only	100	Master with PGO read-write	63881	5.28%
oltp_read_only	100	Master with PGO read-write-v7.0.0	61939	2.07%
oltp_read_write	100	Master	45343
oltp_read_write	100	Master with PGO point-select	46933	3.51%
oltp_read_write	100	Master with PGO read-only	47281	4.27%
oltp_read_write	100	Master with PGO read-write	47229	4.16%
oltp_read_write	100	Master with PGO read-write-v7.0.0	46365	2.25%

Benchmark from Benchbot

Following benchmark use a merged cpu_profile (merge profile files from build/profile folder) to build TiDB with PGO. As you can see from the results, the performance of TiDB compiled with pgo is generally better.

go tool pprof -proto build/profile/*.proto > default.pgo

Sysbench

workload-sub-type	thread	Version	QPS	Diff
oltp_point_select	100	master	86538
oltp_point_select	100	master-pgo	89037	2.89%
oltp_read_only	100	master	43055
oltp_read_only	100	master-pgo	44215	2.69%
select_random	100	master	21591
select_random	100	master-pgo	22840	5.78%
oltp_read_write	100	master	34852
oltp_read_write	100	master-pgo	35875	2.94%

Sysbench on ARM CPU

TPCC

workload	thread	Version	QPS	Diff
tpcc	100	master	41414
tpcc	100	master-pgo	43234	4.39%

YCSB

workload	workload-sub-type	thread	Version	QPS	Diff
ycsb	workloada	100	master	26238
ycsb	workloada	100	master-pgo	26909	2.56%
ycsb	workloadb	100	master	41910
ycsb	workloadb	100	master-pgo	43413	3.59%

Bank

workload	thread	Version	QPS	Diff
bank	100	master	36672
bank	100	master-pgo	37576	2.47%

benchmarksql

workload	thread	Version	QPS	Diff
benchmarksql	100	master	37619
benchmarksql	100	master-pgo	39395	4.7%

tpcds

query_28.sql,query_88.sql

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[tiprow]

Hi @crazycs520. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.5414%. Comparing base (ade745f) to head (168cc68).
Report is 16 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #58630        +/-   ##
================================================
+ Coverage   73.0824%   73.5414%   +0.4590%     
================================================
  Files          1684       1689         +5     
  Lines        466409     478991     +12582     
================================================
+ Hits         340863     352257     +11394     
- Misses       104588     105095       +507     
- Partials      20958      21639       +681     

Flag	Coverage Δ
integration	42.8251% <ø> (?)
unit	72.2378% <ø> (-0.0376%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.6910% <ø> (ø)
parser	∅ <ø> (∅)
br	44.7972% <ø> (-0.5117%)	⬇️

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lance6716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [lance6716]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-01-02 14:46:43.793903233 +0000 UTC m=+620939.149907772: ☑️ agreed by lance6716.

[crazycs520]

/retest-required

[tiprow]

@crazycs520: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tiancaiamao]

We can test more workloads like using benchbot to see its overall effect.
Especially watch out the regression on AP workload, and insert heavy workload etc, and also the result on other arch not limit to arm / x86

[crazycs520]

Have we asked QA to test overall performance?

I've uploaded some test results from other tests inside the PR description.

[zyguan]

Here are some other suggestions:

• It would be better to show the PGO info in tidb-server -V.
• Add a README.md to build/profile to explain the standard way to generate those pprof files.

BTW, do we need to update the build flags for bazel as well?

[zyguan]

Since this file will be generated by make, is it necessary to include it in git? IMO, we only need to keep either default.pgo or build/profile/*.proto.

[cfzjywxk]

+1

@crazycs520 Better to also add some introduction for the choice of default.pgo like how it is generated and why it is chosen as default, which options are recommonded in different scenarios.

[crazycs520]

default.pgo need to keep since the build plugin will use it, otherwise I may need to change other ci/build yaml of plugin.

as build/profile/*.proto files, I also want to keep it, since one look at this folder will tell us which CPU profiles default.pgo is currently using.

[crazycs520]

I write a test about git with large binary file, from the following test result, It wasn't really appropriate to put the profile files inside the git repo, so I've deleted them.

Modified binary file, iteration 1, size: 1 MB, dir_size: 1 MB, git cost: 2.901416ms
Modified binary file, iteration 2, size: 1 MB, dir_size: 2 MB, git cost: 1.016208ms
Modified binary file, iteration 3, size: 1 MB, dir_size: 3 MB, git cost: 1.069875ms
...
Modified binary file, iteration 1000, size: 1 MB, dir_size: 1001 MB, git cost: 35.69275ms

[crazycs520]

Here are some other suggestions:

• It would be better to show the PGO info in tidb-server -V.

We can use the following command to check the PGO info of a binary.

go version -m bin/tidb-server | grep pgo

• Add a README.md to build/profile to explain the standard way to generate those pprof files.

Great, I already add a README.md.

BTW, do we need to update the build flags for bazel as well?

@hawkingrei PTAL

[zyguan]

IMO, there is no need to update Makefile. According to the doc of PGO, go build will detect default.pgo files automatically and enable PGO. Thus, we only need to put a default.pgo to cmd/tidb-server when we want to build with PGO and this can be done in CI/CD.

[crazycs520]

Great, I used to put default.pgo in tidb repo root dir, but it doesn't work, then I add this into Makefile. Put default.pgo to cmd/tidb-server is work.

[crazycs520]

done.