[feature](debug) support debug point used in debug code #24502

yujun777 · 2023-09-17T11:02:48Z

Proposed changes

Debug point is a code switch used for debug development.When a debug point is opened, the corresponding code can be executed.

Both fe and be support debug points.

For example， we want to debug publish version with TOO_MANY_VERSIONS error. Then we can add debug code in BE:

Status   StorageEngine::do_publish_version() {
            .....

           //"test_too_many_versions" is a debug point,
           // if it's enabled,  then it will return error TOO_MANY_VERSIONS
          DBUG_EXECUTE_IF("test_too_many_versions",   {  return Err(TOO_MANY_VERSIONS); });  

          ......
}

for fe, no macro DBUG_EXECUTE_IF, but just use function DebugPointUtil.isEnable(debug_point) to test if enable a debug point;

To enable a debug point, we need two steps:

start fe or be with config.enable_debug_points = true;
post a http request /api/debug_point/add/{debug_point_name} to the fe or be;

Http API:

add a debug point: post /api/debug_point/add/{debug_point_name}[?execute=times&timeout=seconds]. execute is optional, indicates the maximum number of times the debug point can take effect. timeout is optional, indicates the maximum number of seconds the debug point takes effect;
remove a debug point: post /api/debug_point/remove/{debug_point_name};
clear all the debug points: post /api/debug_point/clear.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

yujun777 · 2023-09-17T11:03:02Z

run buildall

github-actions · 2023-09-17T11:10:37Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-17T12:21:31Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (7981/21667)
Line Coverage: 28.85% (64032/221911)
Region Coverage: 27.74% (33225/119770)
Branch Coverage: 24.37% (17035/69904)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d17abc68282c852deec5f23926d3d42c4d432382_d17abc68282c852deec5f23926d3d42c4d432382/report/index.html

doris-robot · 2023-09-17T12:29:42Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.65 seconds
stream load tsv: 598 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162152973 Bytes

yujun777 · 2023-09-17T13:16:40Z

run buildall

github-actions · 2023-09-17T13:23:54Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-17T13:47:20Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.2 seconds
stream load tsv: 599 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162479058 Bytes

doris-robot · 2023-09-17T13:50:44Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (7979/21667)
Line Coverage: 28.85% (64022/221911)
Region Coverage: 27.74% (33224/119770)
Branch Coverage: 24.37% (17033/69904)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4523ce3001cad221c2f04c839a78a939e08847b9_4523ce3001cad221c2f04c839a78a939e08847b9/report/index.html

yujun777 · 2023-09-21T15:51:50Z

run buildall

yujun777 · 2023-09-21T15:52:24Z

run buildall

github-actions · 2023-09-21T15:59:32Z

clang-tidy review says "All clean, LGTM! 👍"

yujun777 · 2023-09-22T01:36:47Z

run buildall

github-actions · 2023-09-22T01:43:06Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-22T02:42:15Z

TeamCity be ut coverage result:
Function Coverage: 36.41% (8023/22037)
Line Coverage: 28.70% (64430/224523)
Region Coverage: 27.61% (33467/121213)
Branch Coverage: 24.27% (17121/70540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1895700a028108e1dda2a19fabbb8080c43ac6f6_1895700a028108e1dda2a19fabbb8080c43ac6f6/report/index.html

doris-robot · 2023-09-22T03:00:53Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.95 seconds
stream load tsv: 598 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162421789 Bytes

yujun777 · 2023-09-22T03:08:45Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/http/action/debug_point_action.cpp

be/src/service/http_service.cpp

be/src/util/debug_points.cpp

be/src/util/debug_points.h

doris-robot · 2023-09-22T03:29:42Z

TeamCity be ut coverage result:
Function Coverage: 36.40% (8021/22037)
Line Coverage: 28.69% (64416/224523)
Region Coverage: 27.59% (33448/121213)
Branch Coverage: 24.27% (17119/70540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b69d412649f56d92bfb8c2cb569a15da93228895_b69d412649f56d92bfb8c2cb569a15da93228895/report/index.html

doris-robot · 2023-09-22T03:41:30Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.79 seconds
stream load tsv: 594 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162380971 Bytes

yujun777 · 2023-09-22T03:44:13Z

run buildall

doris-robot · 2023-09-22T04:21:15Z

TeamCity be ut coverage result:
Function Coverage: 36.41% (8023/22038)
Line Coverage: 28.69% (64430/224555)
Region Coverage: 27.60% (33457/121231)
Branch Coverage: 24.27% (17123/70552)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b69d412649f56d92bfb8c2cb569a15da93228895_b69d412649f56d92bfb8c2cb569a15da93228895/report/index.html

doris-robot · 2023-09-22T04:43:28Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.59 seconds
stream load tsv: 599 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162378503 Bytes

github-actions

clang-tidy made some suggestions

be/src/util/debug_points.h

github-actions

clang-tidy made some suggestions

be/test/testutil/http_utils.cpp

be/test/testutil/http_utils.h

doris-robot · 2023-09-24T07:30:37Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.7 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17162475422 Bytes

yujun777 · 2023-09-24T08:34:32Z

run buildall

yujun777 · 2023-09-24T08:35:44Z

run buildall

github-actions

clang-tidy made some suggestions

be/test/util/debug_points_test.cpp

yujun777 · 2023-09-24T08:44:40Z

run buildall

github-actions

clang-tidy made some suggestions

be/test/util/debug_points_test.cpp

github-actions

clang-tidy made some suggestions

be/test/http/http_auth_test.cpp

doris-robot · 2023-09-24T09:05:45Z

TeamCity be ut coverage result:
Function Coverage: 36.37% (8112/22307)
Line Coverage: 28.48% (64846/227693)
Region Coverage: 27.38% (33638/122853)
Branch Coverage: 24.03% (17170/71466)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3307c73b30f70b64a2cac1241368cd058accb792_3307c73b30f70b64a2cac1241368cd058accb792/report/index.html

doris-robot · 2023-09-24T09:05:50Z

TeamCity be ut coverage result:
Function Coverage: 36.36% (8110/22307)
Line Coverage: 28.49% (64872/227693)
Region Coverage: 27.38% (33642/122853)
Branch Coverage: 24.02% (17168/71466)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3307c73b30f70b64a2cac1241368cd058accb792_3307c73b30f70b64a2cac1241368cd058accb792/report/index.html

doris-robot · 2023-09-24T09:18:37Z

TeamCity be ut coverage result:
Function Coverage: 36.36% (8110/22307)
Line Coverage: 28.49% (64875/227693)
Region Coverage: 27.38% (33637/122853)
Branch Coverage: 24.03% (17174/71466)
Coverage Report: http://coverage.selectdb-in.cc/coverage/db69935fbfdf9ec67ae7c988a272cfb0b283e9c2_db69935fbfdf9ec67ae7c988a272cfb0b283e9c2/report/index.html

doris-robot · 2023-09-24T09:19:31Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.74 seconds
stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17162397032 Bytes

doris-robot · 2023-09-24T09:33:01Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.2 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162412920 Bytes

dataroaring

LGTM

github-actions · 2023-09-25T09:17:38Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-25T09:17:43Z

PR approved by anyone and no changes requested.

yujun777 force-pushed the feat-debug-point branch from 4523ce3 to e3369ba Compare September 21, 2023 15:50

yujun777 added 2 commits September 22, 2023 09:36

support debug point

26ea7aa

impove check is_enable

1895700

yujun777 force-pushed the feat-debug-point branch from e3369ba to 1895700 Compare September 22, 2023 01:36

add docs

b69d412

github-actions bot reviewed Sep 22, 2023

View reviewed changes

remove mutex

1cc2420

github-actions bot reviewed Sep 23, 2023

View reviewed changes

be/src/util/debug_points.h Show resolved Hide resolved

update be test

d23de55

github-actions bot reviewed Sep 24, 2023

View reviewed changes

be/test/testutil/http_utils.cpp Show resolved Hide resolved

be/test/testutil/http_utils.h Show resolved Hide resolved

update http

8ad0ab6

remove temp file

3307c73

github-actions bot reviewed Sep 24, 2023

View reviewed changes

be/test/util/debug_points_test.cpp Outdated Show resolved Hide resolved

update http

db69935

github-actions bot reviewed Sep 24, 2023

View reviewed changes

be/test/util/debug_points_test.cpp Show resolved Hide resolved

github-actions bot reviewed Sep 24, 2023

View reviewed changes

be/test/http/http_auth_test.cpp Show resolved Hide resolved

dataroaring approved these changes Sep 25, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 25, 2023

github-actions bot added the reviewed label Sep 25, 2023

Yukang-Lian approved these changes Sep 25, 2023

View reviewed changes

dataroaring merged commit 8679095 into apache:master Sep 25, 2023

xiaokang added dev/2.0.3 merge_conflict dev/2.0.3-merged and removed dev/2.0.3 labels Sep 26, 2023

xiaokang pushed a commit to xiaokang/doris that referenced this pull request Sep 27, 2023

[feature](debug) support debug point used in debug code (apache#24502)

790cfe6

xiaokang pushed a commit that referenced this pull request Oct 3, 2023

[feature](debug) support debug point used in debug code (#24502)

07a9830

vinlee19 pushed a commit to vinlee19/doris that referenced this pull request Oct 7, 2023

[feature](debug) support debug point used in debug code (apache#24502)

e927ff7

[feature](debug) support debug point used in debug code #24502

[feature](debug) support debug point used in debug code #24502

Conversation

yujun777 commented Sep 17, 2023

Proposed changes

Further comments

yujun777 commented Sep 17, 2023

github-actions bot commented Sep 17, 2023

doris-robot commented Sep 17, 2023

doris-robot commented Sep 17, 2023

yujun777 commented Sep 17, 2023

github-actions bot commented Sep 17, 2023

doris-robot commented Sep 17, 2023

doris-robot commented Sep 17, 2023

yujun777 commented Sep 21, 2023

yujun777 commented Sep 21, 2023

github-actions bot commented Sep 21, 2023

yujun777 commented Sep 22, 2023

github-actions bot commented Sep 22, 2023

doris-robot commented Sep 22, 2023

doris-robot commented Sep 22, 2023

yujun777 commented Sep 22, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Sep 22, 2023

doris-robot commented Sep 22, 2023

yujun777 commented Sep 22, 2023

doris-robot commented Sep 22, 2023

doris-robot commented Sep 22, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Sep 24, 2023

yujun777 commented Sep 24, 2023

yujun777 commented Sep 24, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

yujun777 commented Sep 24, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Sep 24, 2023

doris-robot commented Sep 24, 2023

doris-robot commented Sep 24, 2023

doris-robot commented Sep 24, 2023

doris-robot commented Sep 24, 2023

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 25, 2023

github-actions bot commented Sep 25, 2023