Fix port range allocation with large worker IDs #44213
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Relates to elastic#43983 The IDs gradle uses are incremented for the lifetime of the daemon which can result in port ranges that are outside the valid range. This change implements a modulo based formula to wrap the port ranges when the IDs get too large. Adresses elastic#44134 but elastic#44157 is also required to be able to close it.
alpar-t
added
:Delivery/Build
|
Pinging @elastic/es-core-infra |
|
Pinging @elastic/es-distributed |
| // See also: https://github.com/elastic/elasticsearch/issues/44134 | ||
| final int startAt = Math.round( | ||
| RandomizedTest.systemPropertyAsLong(ESTestCase.TEST_WORKER_VM_ID, 0) | ||
| % 327L |
There was a problem hiding this comment.
This should be 223 to cater for adding 10300 below.
ywelsch
reviewed
Jul 11, 2019
| // This is safe as long as we have fewer than 224 Gradle workers running in parallel | ||
| // See also: https://github.com/elastic/elasticsearch/issues/44134 | ||
| final int startAt = Math.round( | ||
| RandomizedTest.systemPropertyAsLong(ESTestCase.TEST_WORKER_VM_ID, 0) |
There was a problem hiding this comment.
we can use Math.floorMod(long, int) here instead that returns an int. This means that there's no need for Math.round.
Also we should check that the long returned by RandomizedTest.systemPropertyAsLong(ESTestCase.TEST_WORKER_VM_ID, 0) is non-negative.
There was a problem hiding this comment.
This is actually needed because systemPropertyAsLong actually returns a float, seems like a bug in randomized testing, so Math.floorMod won't work.
There was a problem hiding this comment.
you're using the value as system property, not the actual system property. This means that this will always return 0 at the moment. Sounds like we need a test here ;)
I think it would be simpler to write something like the following:
final int startAt = ESTestCase.TEST_WORKER_VM_ID.equals(ESTestCase.DEFAULT_TEST_WORKER_ID) ?
0 :
Math.floorMod(Long.parseLong(ESTestCase.TEST_WORKER_VM_ID), 223);
There was a problem hiding this comment.
🤦♂️
I fixed it and added a test
| return basePort + "-" + (basePort + 99); // upper bound is inclusive | ||
| } | ||
|
|
||
| public static MockNioTransport newMockTransport(Settings settings, Version version, ThreadPool threadPool) { | ||
| settings = Settings.builder().put(TransportSettings.PORT.getKey(), getPortRange()).put(settings).build(); | ||
|
|
|
@elasticmachine run elasticsearch-ci/2 and run elasticsearch-ci/1 |
|
@elasticmachine run elasticsearch-ci/fwc |
|
@elasticmachine run elasticsearch-ci/fwc |
alpar-t
added a commit
that referenced
this pull request
Jul 12, 2019
* Fix port range allocation with large worker IDs Relates to #43983 The IDs gradle uses are incremented for the lifetime of the daemon which can result in port ranges that are outside the valid range. This change implements a modulo based formula to wrap the port ranges when the IDs get too large. Adresses #44134 but #44157 is also required to be able to close it.
alpar-t
added a commit
that referenced
this pull request
Jul 12, 2019
* Fix port range allocation with large worker IDs Relates to #43983 The IDs gradle uses are incremented for the lifetime of the daemon which can result in port ranges that are outside the valid range. This change implements a modulo based formula to wrap the port ranges when the IDs get too large. Adresses #44134 but #44157 is also required to be able to close it.
alpar-t
added a commit
to alpar-t/elasticsearch
that referenced
this pull request
Aug 15, 2019
Moves methods added in elastic#44213 and uses them to configure the port range for `ExternalTestCluster` too. These were still using `9300-9400` ( teh default ) and running into races.
alpar-t
added a commit
that referenced
this pull request
Aug 15, 2019
Moves methods added in #44213 and uses them to configure the port range for `ExternalTestCluster` too. These were still using `9300-9400` ( teh default ) and running into races.
alpar-t
added a commit
that referenced
this pull request
Aug 15, 2019
Moves methods added in #44213 and uses them to configure the port range for `ExternalTestCluster` too. These were still using `9300-9400` ( teh default ) and running into races.
alpar-t
added a commit
that referenced
this pull request
Aug 15, 2019
Moves methods added in #44213 and uses them to configure the port range for `ExternalTestCluster` too. These were still using `9300-9400` ( teh default ) and running into races.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relates to #43983
The IDs gradle uses are incremented for the lifetime of the daemon which
can result in port ranges that are outside the valid range.
This change implements a modulo based formula to wrap the port ranges
when the IDs get too large.
Adresses #44134 but #44157 is also required to be able to close it.