Index Knobs Support #1623

17zhangw · 2021-06-23T09:11:15Z

Description

Supports specifying index knobs (inner node split/merge threshold) through SQL statements. The following are the big changes in this PR that enable such support:

Pipeline Metrics now have 2 additional fields (split/merge for B+trees on OUs that touch indexes)
pg_class now has reloptions which contains a string of the key=value options specified
CREATE INDEX ... WITH (key=value) parsing support to pass down the options

Remaining Work

Reconstruct on recovery
Documentation
PR through CI

noisepage-checks · 2021-06-25T20:35:05Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
-1.32%	tpcc	RAM disk	Details master tps=22556.43, commit tps=22259.66, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-0.36%	tpcc	None	Details master tps=29436.04, commit tps=29330.43, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-0.67%	tpcc	HDD	Details master tps=22159.87, commit tps=22011.74, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
7.29%	tatp	RAM disk	Details master tps=6548.83, commit tps=7026.37, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
3.4%	tatp	None	Details master tps=7309.0, commit tps=7557.68, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
7.58%	tatp	HDD	Details master tps=6397.98, commit tps=6882.96, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

codecov · 2021-06-25T20:35:07Z

Codecov Report

Merging #1623 (ff803db) into master (6177239) will decrease coverage by 0.00%.
The diff coverage is 78.36%.

@@            Coverage Diff             @@
##           master    #1623      +/-   ##
==========================================
- Coverage   80.83%   80.82%   -0.01%     
==========================================
  Files         774      774              
  Lines       55033    55342     +309     
==========================================
+ Hits        44484    44729     +245     
- Misses      10549    10613      +64

Impacted Files	Coverage Δ
src/catalog/index_schema.cpp	`0.00% <0.00%> (ø)`
src/include/execution/sql/table_vector_iterator.h	`100.00% <ø> (ø)`
src/include/execution/vm/bytecode_emitter.h	`100.00% <ø> (ø)`
src/include/execution/vm/bytecode_handlers.h	`69.72% <ø> (ø)`
src/include/execution/vm/bytecodes.h	`96.42% <ø> (ø)`
...de/self_driving/modeling/operating_unit_recorder.h	`100.00% <ø> (ø)`
src/include/storage/index/bplustree.h	`90.98% <ø> (-1.07%)`	⬇️
src/include/storage/index/bplustree_index.h	`100.00% <ø> (ø)`
src/include/storage/index/index_builder.h	`100.00% <ø> (ø)`
src/optimizer/physical_operators.cpp	`89.23% <0.00%> (ø)`
... and 41 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6177239...ff803db. Read the comment docs.

src/include/catalog/index_schema.h

linmagit

Looks good! Thanks a lot, William! I only have a minor comment. There are just two additional things:
(1) Could you add a test case for this? I think this could be a SQL test, similar to the analyze_test. We could test that the thresholds for the indexes are set correctly. Ideally, we also want to test that the index is built with the correct number of threads, but it's fine if that's too difficult.
(2) Could you add the OU recording for the parallel create index? I left you some detailed comments on Slack.
Thanks!

noisepage-checks · 2021-07-06T22:11:50Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
-2.4%	tpcc	RAM disk	Details master tps=22401.76, commit tps=21864.81, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
1.54%	tpcc	None	Details master tps=29057.98, commit tps=29504.38, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
1.51%	tpcc	HDD	Details master tps=21769.33, commit tps=22098.17, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
8.73%	tatp	RAM disk	Details master tps=6421.59, commit tps=6982.26, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-0.08%	tatp	None	Details master tps=7430.85, commit tps=7424.59, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
9.64%	tatp	HDD	Details master tps=6267.66, commit tps=6872.01, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

linmagit

LGTM! Thanks a lot, William!

17zhangw added 8 commits June 22, 2021 17:45

IndexBuilder, IndexOptions

672ba38

BPlusTreeIndex Tuning Functions

5975b6d

Optimizer support for IndexOptions

ebdaa7e

Parser for options

bd564f1

DDL Executors

24d2c41

Override TableVectorIterator::ParallelScan num_threads

c34d677

Refactor pass moving IndexOptions -> IndexSchema

0a72765

Major changes to fix the recording

9cce532

17zhangw self-assigned this Jun 23, 2021

17zhangw added the in-progress This PR is being actively worked on and not ready to be reviewed or merged. Mark PRs with this. label Jun 23, 2021

17zhangw added 2 commits June 24, 2021 03:34

Fix recovery

f6c4a33

Docs and refactor

b9ba7ec

17zhangw added the ready-for-ci Indicate that this build should be run through CI. label Jun 24, 2021

17zhangw added 7 commits June 24, 2021 04:51

Fix some clang-tidy

6e74fe4

Known bug: nolintnextline

846c90b

Fix sql_table_vector_iterator_test

b97b418

TVI::ParallelScan

22b6499

TPL fix

a4d5fc4

Fixed recovery_test

9265e7e

ModelServerTest fix?

e85b1d1

17zhangw added ready-for-review This PR passes all checks and is ready to be reviewed. Mark PRs with this. and removed in-progress This PR is being actively worked on and not ready to be reviewed or merged. Mark PRs with this. labels Jun 25, 2021

17zhangw requested a review from linmagit June 25, 2021 21:20

linmagit reviewed Jun 28, 2021

View reviewed changes

src/include/catalog/index_schema.h Show resolved Hide resolved

linmagit reviewed Jun 28, 2021

View reviewed changes

17zhangw added 3 commits July 1, 2021 05:28

SQL test to check B+Tree options

d03f9ed

Refactor and comments

26fa780

ASSERT_EQ

bd65916

17zhangw added 5 commits July 1, 2021 15:54

Test fix

673bec5

IndexOptions

2eaef87

remove unnecessary options

03672e7

Guard check

05cf57f

Execution Runners patch

ff803db

17zhangw requested a review from linmagit July 6, 2021 22:27

linmagit approved these changes Jul 11, 2021

View reviewed changes

linmagit merged commit 7a199e1 into cmu-db:master Jul 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index Knobs Support #1623

Index Knobs Support #1623

17zhangw commented Jun 23, 2021 •

edited

Loading

noisepage-checks bot commented Jun 25, 2021

codecov bot commented Jun 25, 2021 •

edited

Loading

linmagit left a comment

noisepage-checks bot commented Jul 6, 2021

linmagit left a comment

Index Knobs Support #1623

Index Knobs Support #1623

Conversation

17zhangw commented Jun 23, 2021 • edited Loading

Description

Remaining Work

noisepage-checks bot commented Jun 25, 2021

Minor Decrease in Performance

codecov bot commented Jun 25, 2021 • edited Loading

Codecov Report

linmagit left a comment

Choose a reason for hiding this comment

noisepage-checks bot commented Jul 6, 2021

Minor Decrease in Performance

linmagit left a comment

Choose a reason for hiding this comment

17zhangw commented Jun 23, 2021 •

edited

Loading

codecov bot commented Jun 25, 2021 •

edited

Loading