Write Operations Performance improvements #18578
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xinlian12
requested review from
FabianMeiswinkel,
kirankumarkolli,
kushagraThapar,
mbhaskar,
milismsft,
moderakh and
simplynaveen20
as code owners
ghost
added
the
|
@xinlian12 - can you please provide some explanation on the code change from |
moderakh
approved these changes
Jan 12, 2021
There was a problem hiding this comment.
awesome work @xinlian12
Please attach the perf graphs as well.
LGTM.
So before the change, when we release a channel, we are going to add to the tail of this queue, and when we acquire a channel, we still try to get from the tail of the queue, so it could lead to some channels are handling more loads than others (or could cause we are not fully utilize the channels). After the change, it will behave more like round-robin, so hopefully the load can be more balanced. |
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
kushagraThapar
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem:
Currently the write latency and write throughput is not as good as gateway. When we increase concurrency, the throughput does not increase as much as it should be. The issue is more obvious for small collections (with only 1 partition).
Fix:
This change will help to improve the latency and throughput by selecting the next channel in a round-robin fashion which helps load balance the workload when multiplexing.
Tests1: 100,000 RU, 16 CPU core, Linux(ubuntu), West U2, Eventual consistency, WriteLatency
Test2: 100,000 RU, 4 CPU core, Linux(ubuntu), West U2, Eventual consistency, WriteLatency
Tests3: 6000 RU, 16 CPU core, Linux(ubuntu), West U2, Eventual consistency, WriteThroughput (the concurrency is only up to 16 since we start to get throttle on concurrency 4 (direct after fix) and 8 (gateway)
Tests4: 100,000 RU, 16 CPU core, Linux(ubuntu), West U2, Eventual consistency, ReadLatency This test is to make sure there is no regression for read.
Test5: YCSB customized write workload. 4 CPU core, 100000RU, Linux(Ububtu), West US2. 1-2 threads -> total 100000 operation counts, 4-64 threads -> total 1000000 operation counts
