Group SST write in flush, compaction and db open with new stats #11910

hx235 · 2023-09-29T19:41:07Z

Context/Summary

Similar to #11288, #11444, categorizing SST/blob file write according to different io activities allows more insight into the activity.

For that, this PR does the following:

Tag different write IOs by passing down and converting WriteOptions to IOOptions
Add new SST_WRITE_MICROS histogram in WritableFileWriter::Append() and breakdown FILE_WRITE_{FLUSH|COMPACTION|DB_OPEN}_MICROS

Some related code refactory to make implementation cleaner:

Blob stats
- Replace high-level write measurement with low-level WritableFileWriter::Append() measurement for BLOB_DB_BLOB_FILE_WRITE_MICROS. This is to make FILE_WRITE_{FLUSH|COMPACTION|DB_OPEN}_MICROS include blob file. As a consequence, this introduces some behavioral changes on it, see HISTORY and db bench test plan below for more info.
- Fix bugs where BLOB_DB_BLOB_FILE_SYNCED/BLOB_DB_BLOB_FILE_BYTES_WRITTEN include file failed to sync and bytes failed to write.
Refactor WriteOptions constructor for easier construction with io_activity and rate_limiter_priority
Refactor DBImpl::~DBImpl()/BlobDBImpl::Close() to bypass thread op verification
Build table
- TableBuilderOptions now includes Read/WriteOpitons so BuildTable() do not need to take these two variables
- Replace the io_priority passed into BuildTable() with TableBuilderOptions::WriteOpitons::rate_limiter_priority. Similar for BlobFileBuilder.
  This parameter is used for dynamically changing file io priority for flush, see Set Write rate limiter priority dynamically and pass it to FS #9988 for more
- Update ThreadStatus::FLUSH_BYTES_WRITTEN to use io_activity to track flush IO in flush job and db open instead of io_priority

Test

db bench

Flush

./db_bench --statistics=1 --benchmarks=fillseq --num=100000 --write_buffer_size=100

rocksdb.sst.write.micros P50 : 1.830863 P95 : 4.094720 P99 : 6.578947 P100 : 26.000000 COUNT : 7875 SUM : 20377
rocksdb.file.write.flush.micros P50 : 1.830863 P95 : 4.094720 P99 : 6.578947 P100 : 26.000000 COUNT : 7875 SUM : 20377
rocksdb.file.write.compaction.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0
rocksdb.file.write.db.open.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0

compaction, db oopen

Setup: ./db_bench --statistics=1 --benchmarks=fillseq --num=10000 --disable_auto_compactions=1 -write_buffer_size=100 --db=../db_bench

Run:./db_bench --statistics=1 --benchmarks=compact  --db=../db_bench --use_existing_db=1

rocksdb.sst.write.micros P50 : 2.675325 P95 : 9.578788 P99 : 18.780000 P100 : 314.000000 COUNT : 638 SUM : 3279
rocksdb.file.write.flush.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0
rocksdb.file.write.compaction.micros P50 : 2.757353 P95 : 9.610687 P99 : 19.316667 P100 : 314.000000 COUNT : 615 SUM : 3213
rocksdb.file.write.db.open.micros P50 : 2.055556 P95 : 3.925000 P99 : 9.000000 P100 : 9.000000 COUNT : 23 SUM : 66

blob stats - just to make sure they aren't broken by this PR

Integrated Blob DB

Setup: ./db_bench --enable_blob_files=1 --statistics=1 --benchmarks=fillseq --num=10000 --disable_auto_compactions=1 -write_buffer_size=100 --db=../db_bench

Run:./db_bench --enable_blob_files=1 --statistics=1 --benchmarks=compact  --db=../db_bench --use_existing_db=1

pre-PR:
rocksdb.blobdb.blob.file.write.micros P50 : 7.298246 P95 : 9.771930 P99 : 9.991813 P100 : 16.000000 COUNT : 235 SUM : 1600
rocksdb.blobdb.blob.file.synced COUNT : 1
rocksdb.blobdb.blob.file.bytes.written COUNT : 34842

post-PR:
rocksdb.blobdb.blob.file.write.micros P50 : 2.000000 P95 : 2.829360 P99 : 2.993779 P100 : 9.000000 COUNT : 707 SUM : 1614
- COUNT is higher and values are smaller as it includes header and footer write
- COUNT is 3X higher due to each Append() count as one post-PR, while in pre-PR, 3 Append()s counts as one. See https://github.com/facebook/rocksdb/pull/11910/files#diff-32b811c0a1c000768cfb2532052b44dc0b3bf82253f3eab078e15ff201a0dabfL157-L164

rocksdb.blobdb.blob.file.synced COUNT : 1 (stay the same)
rocksdb.blobdb.blob.file.bytes.written COUNT : 34842 (stay the same)

Stacked Blob DB

Run: ./db_bench --use_blob_db=1 --statistics=1 --benchmarks=fillseq --num=10000 --disable_auto_compactions=1 -write_buffer_size=100 --db=../db_bench

pre-PR:
rocksdb.blobdb.blob.file.write.micros P50 : 12.808042 P95 : 19.674497 P99 : 28.539683 P100 : 51.000000 COUNT : 10000 SUM : 140876
rocksdb.blobdb.blob.file.synced COUNT : 8
rocksdb.blobdb.blob.file.bytes.written COUNT : 1043445

post-PR:
rocksdb.blobdb.blob.file.write.micros P50 : 1.657370 P95 : 2.952175 P99 : 3.877519 P100 : 24.000000 COUNT : 30001 SUM : 67924
- COUNT is higher and values are smaller as it includes header and footer write
- COUNT is 3X higher due to each Append() count as one post-PR, while in pre-PR, 3 Append()s counts as one. See https://github.com/facebook/rocksdb/pull/11910/files#diff-32b811c0a1c000768cfb2532052b44dc0b3bf82253f3eab078e15ff201a0dabfL157-L164

rocksdb.blobdb.blob.file.synced COUNT : 8 (stay the same)
rocksdb.blobdb.blob.file.bytes.written COUNT : 1043445 (stay the same)

Rehearsal CI stress test

Trigger 3 full runs of all our CI stress tests

Performance

Flush

TEST_TMPDIR=/dev/shm ./db_basic_bench_pre_pr --benchmark_filter=ManualFlush/key_num:524288/per_key_size:256 --benchmark_repetitions=1000
-- default: 1 thread is used to run benchmark; enable_statistics = true

Pre-pr: avg 507515519.3 ns
497686074,499444327,500862543,501389862,502994471,503744435,504142123,504224056,505724198,506610393,506837742,506955122,507695561,507929036,508307733,508312691,508999120,509963561,510142147,510698091,510743096,510769317,510957074,511053311,511371367,511409911,511432960,511642385,511691964,511730908,

Post-pr: avg 511971266.5 ns, regressed 0.88%
502744835,506502498,507735420,507929724,508313335,509548582,509994942,510107257,510715603,511046955,511352639,511458478,512117521,512317380,512766303,512972652,513059586,513804934,513808980,514059409,514187369,514389494,514447762,514616464,514622882,514641763,514666265,514716377,514990179,515502408,

Compaction

TEST_TMPDIR=/dev/shm ./db_basic_bench_{pre|post}_pr --benchmark_filter=ManualCompaction/comp_style:0/max_data:134217728/per_key_size:256/enable_statistics:1  --benchmark_repetitions=1000
-- default: 1 thread is used to run benchmark

Pre-pr: avg 495346098.30 ns
492118301,493203526,494201411,494336607,495269217,495404950,496402598,497012157,497358370,498153846

Post-pr: avg 504528077.20, regressed 1.85%. "ManualCompaction" include flush so the isolated regression for compaction should be around 1.85-0.88 = 0.97%
502465338,502485945,502541789,502909283,503438601,504143885,506113087,506629423,507160414,507393007

Put with WAL (in case passing WriteOptions slows down this path even without collecting SST write stats)

TEST_TMPDIR=/dev/shm ./db_basic_bench_pre_pr --benchmark_filter=DBPut/comp_style:0/max_data:107374182400/per_key_size:256/enable_statistics:1/wal:1  --benchmark_repetitions=1000
-- default: 1 thread is used to run benchmark

Pre-pr: avg 3848.10 ns
3814,3838,3839,3848,3854,3854,3854,3860,3860,3860

Post-pr: avg 3874.20 ns, regressed 0.68%
3863,3867,3871,3874,3875,3877,3877,3877,3880,3881

include/rocksdb/env.h

hx235 · 2023-09-29T19:50:50Z

db/blob/blob_log_writer.cc

  }
  if (do_flush_ && s.ok()) {
-    s = dest_->Flush();
+    s = dest_->Flush(opts);


Moving BLOB_DB_BLOB_FILE_WRITE_MICROS from above to WritableFileWriter::Append() will not measure this line. This should only affect stacked/legacy BlobDB, as do_flush is only used there. See 431e8af for more.

facebook-github-bot · 2023-09-29T20:38:21Z

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

include/rocksdb/options.h

hx235 · 2023-10-06T04:47:28Z

Requested @ltamasi for review as this PR affects some blob stats - see PR summary for a high-level review. Let me know if there is anything I can do to make the review easier. Thank you!

ajkr

LGTM

include/rocksdb/options.h

ajkr · 2023-12-27T20:24:35Z

include/rocksdb/options.h

+  WriteOptions(Env::IOPriority _rate_limiter_priority,
+               Env::IOActivity _io_activity = Env::IOActivity::kUnknown);


Needs explicit

include/rocksdb/env.h

db/blob/blob_log_writer.cc

facebook-github-bot · 2023-12-28T20:31:32Z

@hx235 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2023-12-28T20:31:57Z

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-12-28T22:23:12Z

@hx235 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2023-12-28T22:23:40Z

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-12-29T19:42:32Z

@hx235 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2023-12-29T19:42:58Z

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-12-29T20:54:16Z

@hx235 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2023-12-29T20:54:38Z

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-12-29T23:34:08Z

@hx235 merged this pull request in 06e5933.

hx235 force-pushed the write_io_activity branch from 346a60e to d9db355 Compare September 29, 2023 19:46

hx235 commented Sep 29, 2023

View reviewed changes

include/rocksdb/env.h Show resolved Hide resolved

hx235 commented Sep 29, 2023

View reviewed changes

hx235 force-pushed the write_io_activity branch from d9db355 to 93d6092 Compare September 29, 2023 20:36

facebook-github-bot added the CLA Signed label Sep 29, 2023

hx235 commented Oct 6, 2023

View reviewed changes

include/rocksdb/options.h Show resolved Hide resolved

hx235 requested review from ajkr and ltamasi October 6, 2023 04:46

hx235 changed the title ~~Group SST write in flush, compaction and db open~~ Group SST write in flush, compaction and db open with new stats Oct 6, 2023

ajkr approved these changes Dec 27, 2023

View reviewed changes

hx235 force-pushed the write_io_activity branch from 93d6092 to 55229c3 Compare December 28, 2023 20:31

hx235 force-pushed the write_io_activity branch from 55229c3 to 40ed346 Compare December 28, 2023 22:23

hx235 force-pushed the write_io_activity branch from 40ed346 to 3b97a64 Compare December 29, 2023 19:42

Tag io activity of sst/blob file write + stats + dbstress

3aa7cb0

hx235 force-pushed the write_io_activity branch from 3b97a64 to 3aa7cb0 Compare December 29, 2023 20:54

facebook-github-bot closed this in 06e5933 Dec 29, 2023

facebook-github-bot added the Merged label Dec 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Group SST write in flush, compaction and db open with new stats #11910

Group SST write in flush, compaction and db open with new stats #11910

hx235 commented Sep 29, 2023 •

edited

Loading

hx235 Sep 29, 2023 •

edited

Loading

facebook-github-bot commented Sep 29, 2023

hx235 commented Oct 6, 2023 •

edited

Loading

ajkr left a comment

ajkr Dec 27, 2023

hx235 Dec 28, 2023

facebook-github-bot commented Dec 28, 2023

facebook-github-bot commented Dec 28, 2023

facebook-github-bot commented Dec 28, 2023

facebook-github-bot commented Dec 28, 2023

facebook-github-bot commented Dec 29, 2023

facebook-github-bot commented Dec 29, 2023

facebook-github-bot commented Dec 29, 2023

facebook-github-bot commented Dec 29, 2023

facebook-github-bot commented Dec 29, 2023

		WriteOptions(Env::IOPriority _rate_limiter_priority,
		Env::IOActivity _io_activity = Env::IOActivity::kUnknown);

Group SST write in flush, compaction and db open with new stats #11910

Group SST write in flush, compaction and db open with new stats #11910

Conversation

hx235 commented Sep 29, 2023 • edited Loading

Context/Summary

Test

db bench

Rehearsal CI stress test

Performance

hx235 Sep 29, 2023 • edited Loading

Choose a reason for hiding this comment

facebook-github-bot commented Sep 29, 2023

hx235 commented Oct 6, 2023 • edited Loading

ajkr left a comment

Choose a reason for hiding this comment

ajkr Dec 27, 2023

Choose a reason for hiding this comment

hx235 Dec 28, 2023

Choose a reason for hiding this comment

facebook-github-bot commented Dec 28, 2023

facebook-github-bot commented Dec 28, 2023

facebook-github-bot commented Dec 28, 2023

facebook-github-bot commented Dec 28, 2023

facebook-github-bot commented Dec 29, 2023

facebook-github-bot commented Dec 29, 2023

facebook-github-bot commented Dec 29, 2023

facebook-github-bot commented Dec 29, 2023

facebook-github-bot commented Dec 29, 2023

hx235 commented Sep 29, 2023 •

edited

Loading

hx235 Sep 29, 2023 •

edited

Loading

hx235 commented Oct 6, 2023 •

edited

Loading