*: encode the returned data using the Chunk format in mocktikv #12023

wshwsh12 · 2019-09-04T06:23:47Z

What problem does this PR solve?

Return chunk-encoded data when the requested EncodeType is TypeArrow.

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change

Side effects

Increased code complexity

BenchMark

Now I only test the time cost in TPCH-Q5. (10g data)
The reuslt: (Default concurrency)

Default  37.11 sec
Arrow    37.67 sec

The reuslt: (set tidb_index_lookup_join_concurrency=16;)

Default  22.80 sec
Arrow    16.68 sec

Default

+--------------------------------------------------+-------------+------+-------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
| id                                               | count       | task | operator info                                                                             | execution info                                                                                                                              | memory                |
+--------------------------------------------------+-------------+------+-------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
| Sort_23                                          | 5.00        | root | Column#51:desc                                                                            | time:22.770147694s, loops:2, rows:5                                                                                                         | 1.91796875 KB         |
| └─Projection_25                                  | 5.00        | root | Column#43, Column#49                                                                      | time:22.770131654s, loops:2, rows:5                                                                                                         | N/A                   |
|   └─HashAgg_28                                   | 5.00        | root | group by:Column#52, funcs:sum(Column#0), firstrow(Column#0)                               | time:22.770128546s, loops:2, rows:5                                                                                                         | N/A                   |
|     └─Projection_80                              | 2236833.33  | root | mul(Column#23, minus(1, Column#24)), Column#43, Column#43                                 | time:22.76982236s, loops:76, rows:72985                                                                                                     | N/A                   |
|       └─HashLeftJoin_36                          | 2236833.33  | root | inner join, inner:TableReader_78, equal:[eq(Column#38, Column#4) eq(Column#10, Column#1)] | time:22.769594411s, loops:76, rows:72985                                                                                                    | 23.561248779296875 MB |
|         ├─IndexMergeJoin_47                      | 2236833.33  | root | inner join, inner:TableReader_45, outer key:Column#18, inner key:Column#9                 | time:22.459100187s, loops:1790, rows:1823514                                                                                                | 2.2937463484704494 GB |
|         │ ├─HashRightJoin_51                     | 12006815.85 | root | inner join, inner:HashRightJoin_53, equal:[eq(Column#35, Column#20)]                      | time:19.661978759s, loops:11745, rows:12022687                                                                                              | 1.5376052856445312 MB |
|         │ │ ├─HashRightJoin_53                   | 20000.00    | root | inner join, inner:HashRightJoin_64, equal:[eq(Column#42, Column#38)]                      | time:142.30407ms, loops:23, rows:20042                                                                                                      | 2.05859375 KB         |
|         │ │ │ ├─HashRightJoin_64                 | 5.00        | root | inner join, inner:TableReader_69, equal:[eq(Column#46, Column#44)]                        | time:106.428µs, loops:2, rows:5                                                                                                             | 20.46484375 KB        |
|         │ │ │ │ ├─TableReader_69                 | 1.00        | root | data:Selection_68                                                                         | time:13.526µs, loops:2, rows:1, rpc time:1.092075ms                                                                                         | 169 Bytes             |
|         │ │ │ │ │ └─Selection_68                 | 1.00        | cop  | eq(Column#47, "MIDDLE EAST")                                                              | time:0s, loops:1, rows:1                                                                                                                    | N/A                   |
|         │ │ │ │ │   └─TableScan_67               | 5.00        | cop  | table:region, range:[-inf,+inf], keep order:false                                         | time:0s, loops:1, rows:5                                                                                                                    | N/A                   |
|         │ │ │ │ └─TableReader_66                 | 25.00       | root | data:TableScan_65                                                                         | time:27.713µs, loops:2, rows:25, rpc time:1.274225ms                                                                                        | 688 Bytes             |
|         │ │ │ │   └─TableScan_65                 | 25.00       | cop  | table:nation, range:[-inf,+inf], keep order:false                                         | time:1ms, loops:1, rows:25                                                                                                                  | N/A                   |
|         │ │ │ └─TableReader_71                   | 100000.00   | root | data:TableScan_70                                                                         | time:142.651099ms, loops:99, rows:100000, rpc time:157.204305ms                                                                             | 1.050323486328125 MB  |
|         │ │ │   └─TableScan_70                   | 100000.00   | cop  | table:supplier, range:[-inf,+inf], keep order:false                                       | time:146ms, loops:102, rows:100000                                                                                                          | N/A                   |
|         │ │ └─TableReader_73                     | 59986052.00 | root | data:TableScan_72                                                                         | time:22.233397247s, loops:58582, rows:59986052, rpc max:1.160607429s, min:79.726388ms, avg:661.569041ms, p80:851.602895ms, p95:1.026335118s | 489.6857976913452 MB  |
|         │ │   └─TableScan_72                     | 59986052.00 | cop  | table:lineitem, range:[-inf,+inf], keep order:false                                       | proc max:1.07s, min:73ms, p80:767ms, p95:949ms, rows:59986052, iters:59231, tasks:153                                                       | N/A                   |
|         │ └─TableReader_45                       | 0.15        | root | data:Selection_44                                                                         | time:2m28.076393464s, loops:1893, rows:1249432, rpc max:646.915632ms, min:1.048567ms, avg:274.856929ms, p80:318.035459ms, p95:424.753119ms  | N/A                   |
|         │   └─Selection_44                       | 0.15        | cop  | ge(Column#13, 1994-01-01 00:00:00.000000), lt(Column#13, 1995-01-01)                      | proc max:595ms, min:1ms, p80:312ms, p95:411ms, rows:1249432, iters:10709, tasks:564                                                         | N/A                   |
|         │     └─TableScan_43                     | 0.19        | cop  | table:orders, range: decided by [Column#18], keep order:true                              | proc max:585ms, min:1ms, p80:308ms, p95:401ms, rows:8236741, iters:10709, tasks:564                                                         | N/A                   |
|         └─TableReader_78                         | 1500000.00  | root | data:TableScan_77                                                                         | time:612.626797ms, loops:1466, rows:1500000, rpc max:640.497105ms, min:404.915325ms, avg:534.187743ms, p80:640.497105ms, p95:640.497105ms   | 12.281054496765137 MB |
|           └─TableScan_77                         | 1500000.00  | cop  | table:customer, range:[-inf,+inf], keep order:false                                       | proc max:598ms, min:381ms, p80:598ms, p95:598ms, rows:1500000, iters:1482, tasks:4                                                          | N/A                   |
+--------------------------------------------------+-------------+------+-------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
23 rows in set (22.80 sec)

Arrow

+--------------------------------------------------+-------------+------+-------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
| id                                               | count       | task | operator info                                                                             | execution info                                                                                                                              | memory                |
+--------------------------------------------------+-------------+------+-------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
| Sort_23                                          | 5.00        | root | Column#51:desc                                                                            | time:16.672479712s, loops:2, rows:5                                                                                                         | 1.91796875 KB         |
| └─Projection_25                                  | 5.00        | root | Column#43, Column#49                                                                      | time:16.672467204s, loops:2, rows:5                                                                                                         | N/A                   |
|   └─HashAgg_28                                   | 5.00        | root | group by:Column#52, funcs:sum(Column#0), firstrow(Column#0)                               | time:16.672464934s, loops:2, rows:5                                                                                                         | N/A                   |
|     └─Projection_80                              | 2236833.33  | root | mul(Column#23, minus(1, Column#24)), Column#43, Column#43                                 | time:16.671955743s, loops:75, rows:72985                                                                                                    | N/A                   |
|       └─HashLeftJoin_36                          | 2236833.33  | root | inner join, inner:TableReader_78, equal:[eq(Column#38, Column#4) eq(Column#10, Column#1)] | time:16.671436712s, loops:75, rows:72985                                                                                                    | 8.425058089196682 GB  |
|         ├─IndexMergeJoin_47                      | 2236833.33  | root | inner join, inner:TableReader_45, outer key:Column#18, inner key:Column#9                 | time:16.53063965s, loops:1790, rows:1823514                                                                                                 | 3.796726867556572 GB  |
|         │ ├─HashRightJoin_51                     | 12006815.85 | root | inner join, inner:HashRightJoin_53, equal:[eq(Column#35, Column#20)]                      | time:2.943592967s, loops:11746, rows:12022687                                                                                               | 1.5356521606445312 MB |
|         │ │ ├─HashRightJoin_53                   | 20000.00    | root | inner join, inner:HashRightJoin_64, equal:[eq(Column#42, Column#38)]                      | time:166.297341ms, loops:23, rows:20042                                                                                                     | 2.05859375 KB         |
|         │ │ │ ├─HashRightJoin_64                 | 5.00        | root | inner join, inner:TableReader_69, equal:[eq(Column#46, Column#44)]                        | time:615.791µs, loops:2, rows:5                                                                                                             | 620 Bytes             |
|         │ │ │ │ ├─TableReader_69                 | 1.00        | root | data:Selection_68                                                                         | time:18.693µs, loops:2, rows:1, rpc time:868.274µs                                                                                          | 198 Bytes             |
|         │ │ │ │ │ └─Selection_68                 | 1.00        | cop  | eq(Column#47, "MIDDLE EAST")                                                              | time:0s, loops:1, rows:1                                                                                                                    | N/A                   |
|         │ │ │ │ │   └─TableScan_67               | 5.00        | cop  | table:region, range:[-inf,+inf], keep order:false                                         | time:0s, loops:1, rows:5                                                                                                                    | N/A                   |
|         │ │ │ │ └─TableReader_66                 | 25.00       | root | data:TableScan_65                                                                         | time:579.627µs, loops:2, rows:25, rpc time:1.516286ms                                                                                       | 994 Bytes             |
|         │ │ │ │   └─TableScan_65                 | 25.00       | cop  | table:nation, range:[-inf,+inf], keep order:false                                         | time:1ms, loops:1, rows:25                                                                                                                  | N/A                   |
|         │ │ │ └─TableReader_71                   | 100000.00   | root | data:TableScan_70                                                                         | time:161.002089ms, loops:103, rows:100000, rpc time:161.586458ms                                                                            | 1.52813720703125 MB   |
|         │ │ │   └─TableScan_70                   | 100000.00   | cop  | table:supplier, range:[-inf,+inf], keep order:false                                       | time:149ms, loops:102, rows:100000                                                                                                          | N/A                   |
|         │ │ └─TableReader_73                     | 59986052.00 | root | data:TableScan_72                                                                         | time:1.04352478s, loops:59232, rows:59986052, rpc max:1.656616424s, min:137.145763ms, avg:1.200088653s, p80:1.348882139s, p95:1.487314887s  | 1.3459740281105042 GB |
|         │ │   └─TableScan_72                     | 59986052.00 | cop  | table:lineitem, range:[-inf,+inf], keep order:false                                       | proc max:1.168s, min:96ms, p80:904ms, p95:1.003s, rows:59986052, iters:59231, tasks:153                                                     | N/A                   |
|         │ └─TableReader_45                       | 0.15        | root | data:Selection_44                                                                         | time:3m53.824442702s, loops:11285, rows:1249439, rpc max:887.070028ms, min:1.269268ms, avg:427.577992ms, p80:597.372658ms, p95:709.585346ms | N/A                   |
|         │   └─Selection_44                       | 0.15        | cop  | ge(Column#13, 1994-01-01 00:00:00.000000), lt(Column#13, 1995-01-01)                      | proc max:593ms, min:0s, p80:486ms, p95:561ms, rows:1249439, iters:10818, tasks:591                                                          | N/A                   |
|         │     └─TableScan_43                     | 0.19        | cop  | table:orders, range: decided by [Column#18], keep order:true                              | proc max:585ms, min:0s, p80:479ms, p95:553ms, rows:8236791, iters:10818, tasks:591                                                          | N/A                   |
|         └─TableReader_78                         | 1500000.00  | root | data:TableScan_77                                                                         | time:438.678769ms, loops:1483, rows:1500000, rpc max:607.694099ms, min:409.775717ms, avg:508.329883ms, p80:607.694099ms, p95:607.694099ms   | 11.914450645446777 MB |
|           └─TableScan_77                         | 1500000.00  | cop  | table:customer, range:[-inf,+inf], keep order:false                                       | proc max:535ms, min:370ms, p80:535ms, p95:535ms, rows:1500000, iters:1482, tasks:4                                                          | N/A                   |
+--------------------------------------------------+-------------+------+-------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
23 rows in set (16.68 sec)

The cpu time of Decode disappears.

I will add more performance test and count the memory use.

codecov · 2019-09-06T07:47:12Z

Codecov Report

Merging #12023 into master will decrease coverage by 0.0902%.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             master     #12023        +/-   ##
================================================
- Coverage   79.9846%   79.8944%   -0.0903%     
================================================
  Files           462        462                
  Lines        103171     103240        +69     
================================================
- Hits          82521      82483        -38     
- Misses        14758      14860       +102     
- Partials       5892       5897         +5

wshwsh12 · 2019-09-06T07:48:51Z

/run-all-tests

wshwsh12 · 2019-09-10T02:26:57Z

/run-common-test

wshwsh12 · 2019-09-16T09:13:29Z

/run-all-tests tikv=pr/5461

wshwsh12 · 2019-09-16T14:55:15Z

/run-all-tests tikv=pr/5461 tidb-test=pr/898

distsql/select_result.go

XuHuaiyu · 2019-09-26T02:26:29Z

distsql/distsql.go

+	}
+}
+
+func useChunkIPC(ctx sessionctx.Context) bool {


s/ useChunkIPC/ enableTypeArrow

distsql/select_result.go

XuHuaiyu · 2019-09-26T03:35:51Z

store/mockstore/mocktikv/cop_handler_dag.go

+	return nil
+}
+
+func (h *rpcHandler) constructRespSchema(dagCtx *dagContext) []*types.FieldType {


Is this function also useless, dagCtx.evalCtx.fieldTps is enough?

store/mockstore/mocktikv/cop_handler_dag.go

sessionctx/variable/sysvar.go

XuHuaiyu

LGTM

zz-jason

LGTM

zz-jason · 2019-09-27T06:28:51Z

sessionctx/variable/session.go

@@ -352,6 +352,9 @@ type SessionVars struct {
 	// TODO: remove this after tidb-server configuration "enable-streaming' removed.
 	EnableStreaming bool

+	// EnableArrow indicates whether the coprocessor request can use arrow API.
+	EnableArrow bool


We can make this config item be able to be reloaded without restarting the tidb-server like other hot-reloadable config items. It's not urgent, we can do this in another PR.

sre-bot · 2019-09-27T06:33:33Z

/run-all-tests

The enable-arrow needs to be set to false now.

wshwsh12 · 2019-09-27T06:50:49Z

/bench

sre-bot · 2019-09-27T10:15:25Z

@@                               Benchmark Diff                               @@
================================================================================
--- tidb: 5986c6cbfa9b2007e69cfe2a73bd094d984c5a21
+++ tidb: 388fed5c5652039a05e405cac875de579fb374a8
tikv: a036e28195aa23658e1de9a15e6f802cd1d927ea
pd: e2dbdf6ead101aeb1aa6d1298ecb3f1fc0ee62f1
================================================================================
test-1: < oltp_read_write >
    * QPS : 37286.82 ± 0.2529% (std=58.48) delta: 0.34% (p=0.118)
    * AvgMs : 137.86 ± 0.2481% (std=0.21) delta: -0.34%
    * PercentileMs99 : 267.41 ± 0.0000% (std=0.00) delta: 0.00%
            
test-2: < oltp_point_select >
    * QPS : 81169.01 ± 3.4426% (std=1613.65) delta: -1.09% (p=0.396)
    * AvgMs : 3.15 ± 3.4264% (std=0.06) delta: 1.03%
    * PercentileMs99 : 6.74 ± 2.8478% (std=0.12) delta: 0.36%
            
test-3: < oltp_insert >
    * QPS : 21748.36 ± 0.2988% (std=43.00) delta: -0.48% (p=0.337)
    * AvgMs : 11.77 ± 0.3229% (std=0.02) delta: 0.51%
    * PercentileMs99 : 24.12 ± 1.0696% (std=0.21) delta: 1.45%
            
test-4: < oltp_update_index >
    * QPS : 16999.35 ± 0.3929% (std=48.95) delta: -0.02% (p=0.240)
    * AvgMs : 14.94 ± 0.7230% (std=0.07) delta: 0.17%
    * PercentileMs99 : 30.81 ± 0.0000% (std=0.00) delta: 0.35%
            
test-5: < oltp_update_non_index >
    * QPS : 29083.71 ± 0.0333% (std=7.54) delta: -0.25% (p=0.232)
    * AvgMs : 8.79 ± 0.1422% (std=0.01) delta: 0.19%
    * PercentileMs99 : 19.29 ± 0.0000% (std=0.00) delta: 0.00%

https://perf.pingcap.com

zyxbest · 2019-09-28T11:51:07Z

/run-all-tests

zyxbest · 2019-09-28T11:54:38Z

/rebuild

…ap#12023)

wshwsh12 added status/WIP sig/execution SIG execution component/coprocessor type/enhancement The issue or PR belongs to an enhancement. and removed sig/execution SIG execution labels Sep 4, 2019

wshwsh12 force-pushed the change_chunk branch from ed527ae to 4ed0984 Compare September 6, 2019 07:41

wshwsh12 force-pushed the change_chunk branch from 4ed0984 to 4e5a90f Compare September 9, 2019 02:45

wshwsh12 added 7 commits September 10, 2019 14:35

change_chunk

4470f2c

add performance test code

4f3c090

add benchmark with new chunk

b57c1c7

fix ci

6d34738

remove some ad-hoc code

0fddb75

reuse mem

8547dea

add TODO

818f590

wshwsh12 force-pushed the change_chunk branch from b65cc2f to 818f590 Compare September 10, 2019 06:50

Merge remote-tracking branch 'upstream/master' into change_chunk

d841cc6

wshwsh12 mentioned this pull request Sep 14, 2019

Coprocessor: return Chunk encoded data tikv/tikv#5461

Merged

wshwsh12 added 4 commits September 16, 2019 15:27

check returned data is default/arrow format

17389b3

Merge remote-tracking branch 'upstream/master' into change_chunk

3c0505e

remove useless var

c42e3ba

remove useless var

ae73a0c

wshwsh12 removed the status/WIP label Sep 16, 2019

Merge remote-tracking branch 'upstream/master' into change_chunk

41d2abb

wshwsh12 requested review from XuHuaiyu and qw4990 September 16, 2019 15:00

XuHuaiyu reviewed Sep 26, 2019

View reviewed changes

XuHuaiyu changed the title ~~mocktikv: return Chunk encoded data~~ *: encode the returned data using the Chunk format in mocktikv Sep 26, 2019

wshwsh12 added 2 commits September 26, 2019 14:36

fix comment

07e9804

add comment

a32153d

XuHuaiyu reviewed Sep 26, 2019

View reviewed changes

wshwsh12 added 2 commits September 26, 2019 22:02

address comments

96a7951

Merge remote-tracking branch 'upstream/master' into change_chunk

392ec3c

wshwsh12 requested a review from XuHuaiyu September 26, 2019 14:05

winoros reviewed Sep 26, 2019

View reviewed changes

sessionctx/variable/sysvar.go Outdated Show resolved Hide resolved

XuHuaiyu reviewed Sep 27, 2019

View reviewed changes

XuHuaiyu added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 27, 2019

XuHuaiyu requested a review from zz-jason September 27, 2019 01:29

zz-jason previously approved these changes Sep 27, 2019

View reviewed changes

zz-jason added status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 27, 2019

Merge branch 'master' into change_chunk

388fed5

qw4990 removed their request for review September 27, 2019 07:29

change default value

7f67e84

zz-jason approved these changes Sep 28, 2019

View reviewed changes

Merge branch 'master' into change_chunk

26b704b

zz-jason merged commit 99a5432 into pingcap:master Sep 28, 2019

lzmhhh123 pushed a commit to lzmhhh123/tidb that referenced this pull request Oct 8, 2019

*: encode the returned data using the Chunk format in mocktikv (pingc…

20cda59

…ap#12023)

wshwsh12 deleted the change_chunk branch April 21, 2020 05:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: encode the returned data using the Chunk format in mocktikv #12023

*: encode the returned data using the Chunk format in mocktikv #12023

wshwsh12 commented Sep 4, 2019 •

edited

Loading

codecov bot commented Sep 6, 2019 •

edited

Loading

wshwsh12 commented Sep 6, 2019

wshwsh12 commented Sep 10, 2019

wshwsh12 commented Sep 16, 2019

wshwsh12 commented Sep 16, 2019

XuHuaiyu Sep 26, 2019

XuHuaiyu Sep 26, 2019

XuHuaiyu left a comment

zz-jason left a comment

zz-jason Sep 27, 2019

sre-bot commented Sep 27, 2019

wshwsh12 commented Sep 27, 2019

sre-bot commented Sep 27, 2019

zyxbest commented Sep 28, 2019

zyxbest commented Sep 28, 2019

*: encode the returned data using the Chunk format in mocktikv #12023

*: encode the returned data using the Chunk format in mocktikv #12023

Conversation

wshwsh12 commented Sep 4, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

BenchMark

Default

Arrow

codecov bot commented Sep 6, 2019 • edited Loading

Codecov Report

wshwsh12 commented Sep 6, 2019

wshwsh12 commented Sep 10, 2019

wshwsh12 commented Sep 16, 2019

wshwsh12 commented Sep 16, 2019

XuHuaiyu Sep 26, 2019

Choose a reason for hiding this comment

XuHuaiyu Sep 26, 2019

Choose a reason for hiding this comment

XuHuaiyu left a comment

Choose a reason for hiding this comment

zz-jason left a comment

Choose a reason for hiding this comment

zz-jason Sep 27, 2019

Choose a reason for hiding this comment

sre-bot commented Sep 27, 2019

wshwsh12 commented Sep 27, 2019

sre-bot commented Sep 27, 2019

zyxbest commented Sep 28, 2019

zyxbest commented Sep 28, 2019

wshwsh12 commented Sep 4, 2019 •

edited

Loading

codecov bot commented Sep 6, 2019 •

edited

Loading