Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebase throttling feature branch #4713

Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
112 commits
Select commit Hold shift + click to select a range
a3fb105
[BUG] Running "opensearch-service.bat start" and "opensearch-service.…
burck1 Aug 24, 2022
0bf6b2f
Removing dead code in RecoveryTarget. (#4278)
mch2 Aug 24, 2022
5dd7947
Update the head ref to changelog verifier (#4296)
kotwanikunal Aug 24, 2022
1bfabed
Add 2.x version to CHANGELOG (#4297)
kotwanikunal Aug 25, 2022
1dbb63a
Do not fail replica shard due to primary closure (#4133)
andrross Aug 26, 2022
c62cecb
Some dependency updates (#4308)
reta Aug 26, 2022
65f966e
Restore using the class ClusterInfoRequest and ClusterInfoRequestBuil…
Aug 26, 2022
7ea6e88
[BUG] Create logs directory before running OpenSearch on Windows (#4305)
burck1 Aug 29, 2022
cd961f3
Use RemoteSegmentStoreDirectory instead of RemoteDirectory (#4240)
sachinpkale Aug 29, 2022
7fe5830
ZIP publication groupId value is configurable (#4156)
lukas-vlcek Aug 29, 2022
f4e041e
[Segment Replication] Add timeout on Mockito.verify to reduce flakyne…
dreamer-89 Aug 29, 2022
beb09af
Adding @dreamer-89 to Opensearch maintainers. (#4342)
kartg Aug 30, 2022
4bccdbe
[CVE] Update snakeyaml dependency (#4341)
adnapibar Aug 30, 2022
82bda89
Fixed commit workflow for dependabot PR helper (#4331)
kotwanikunal Aug 30, 2022
48d6869
Add release notes for patch release 1.3.5 (#4343)
adnapibar Aug 30, 2022
f16ea9c
Add release notes for patch release 2.2.1 (#4344)
adnapibar Aug 30, 2022
4f65ef5
Add label configuration for dependabot PRs (#4348)
kotwanikunal Aug 31, 2022
d72861f
Support for HTTP/2 (server-side) (#3847)
reta Aug 31, 2022
c28221e
Fix token usage for changelog helper (#4351)
kotwanikunal Aug 31, 2022
100120a
Revert "Fix token usage for changelog helper (#4351)" (#4361)
kotwanikunal Aug 31, 2022
19d1a2b
Segment Replication - Implement segment replication event cancellatio…
mch2 Aug 31, 2022
4a6e937
Bug fixes for dependabot changelog verifier (#4364)
kotwanikunal Sep 1, 2022
689a2c4
Add changes for Create PIT and Delete PIT rest layer and rest high le…
bharath-techie Sep 1, 2022
bd11c69
Bump com.diffplug.spotless from 6.9.1 to 6.10.0 (#4319)
dependabot[bot] Sep 1, 2022
236f2f6
Update to Netty 4.1.80.Final (#4359)
reta Sep 1, 2022
5c3cc93
Bump xmlbeans from 5.1.0 to 5.1.1 in /plugins/ingest-attachment (#4354)
dependabot[bot] Sep 1, 2022
715da84
Fix randomized test failure NRTReplicationEngineTests.testUpdateSegme…
adnapibar Sep 1, 2022
70d911c
[AUTO] [main] Added bwc version 2.2.2. (#4383)
opensearch-trigger-bot[bot] Sep 2, 2022
c885686
[Segment Replication] Bump segment infos counter before commit during…
dreamer-89 Sep 2, 2022
b206e98
[Segment Replication] Add check to cancel ongoing replication with ol…
dreamer-89 Sep 2, 2022
0c10674
Adding support for labels on version bump PRs, skip label support for…
kotwanikunal Sep 2, 2022
fab2a12
[Segment Replication] Extend FileChunkWriter to allow cancel on trans…
dreamer-89 Sep 3, 2022
4331d2a
Added RestLayer Changes for PIT stats (#4217)
ajaymovva Sep 4, 2022
4ed0995
Modified cat shards test for pit stats (#4408)
ajaymovva Sep 6, 2022
ff2e4bf
[Remote Store] Add index specific setting for remote repository (#4253)
sachinpkale Sep 6, 2022
b0e1f6a
[Semgnet Replication] Update flaky testOnNewCheckpointFromNewPrimaryC…
dreamer-89 Sep 6, 2022
f97cb4b
Segment Replication - Fix NoSuchFileException errors caused when comp…
mch2 Sep 6, 2022
1889d96
Bump org.gradle.test-retry from 1.4.0 to 1.4.1 (#4411)
dependabot[bot] Sep 6, 2022
fb64a85
Revert to Netty 4.1.79.Final (#4428)
cwperks Sep 6, 2022
bff9d0f
[Segment Replication] Fix timeout issue by calculating time needed to…
Rishikesh1159 Sep 7, 2022
ccf575a
[Bug]: gradle check failing with java heap OutOfMemoryError (#4328)
reta Sep 7, 2022
c1a1f1a
Bump azure-core-http-netty from 1.12.0 to 1.12.4 in /plugins/reposito…
dependabot[bot] Sep 7, 2022
ce6c6a8
Bugfix: Allow opensearch.bat file and opensearch-env.bat files to run…
mwilkinson-imo Sep 7, 2022
3ef0046
[Segment Replication] - Update replicas to commit SegmentInfos instea…
mch2 Sep 7, 2022
dac99d5
Bump reactor-netty-core from 1.0.19 to 1.0.22 in /plugins/repository-…
dependabot[bot] Sep 7, 2022
e305e1e
Add bwcVersion 2.4.0 (#4455)
dreamer-89 Sep 8, 2022
517c19c
add support for s390x architecture (#4001)
pantovad Sep 8, 2022
54364a5
2.3.0 release notes (#4457) (#4464)
dreamer-89 Sep 9, 2022
5dc3d2e
Add getHistoryOperationsFromTranslog method to fetch the history snap…
ankitkala Sep 13, 2022
763a89f
Fixed flaky test: ResourceAwareTasksTests.testTaskIdPersistsInThreadC…
ketanv3 Sep 13, 2022
8366ea3
Getting security exception due to access denied 'java.lang.RuntimePer…
reta Sep 13, 2022
29153fd
[Javadoc] add missing javadocs for :distribution:tools modules (#4483)
dbwiddis Sep 13, 2022
51a529f
Bump reactive-streams from 1.0.3 to 1.0.4 in /plugins/repository-azur…
dependabot[bot] Sep 13, 2022
45c9597
Ignore all malformed objects when ignore_malformed is true (#4494)
hauck-jvsh Sep 15, 2022
72e6801
[AUTO] [main] Added bwc version 2.3.1. (#4513)
opensearch-trigger-bot[bot] Sep 15, 2022
f73a870
[Segment Replication] Add snapshot and restore tests for segment repl…
dreamer-89 Sep 16, 2022
57a4aed
[Remote Store] Change behaviour in replica recovery for remote transl…
ashking94 Sep 17, 2022
4657716
Weighted round-robin scheduling policy for shard coordination traffic…
anshu1106 Sep 19, 2022
fa07cd9
[Javadoc] add missing javadocs for :example-plugin modules (#4540)
dbwiddis Sep 19, 2022
56162a9
Add GeoBounds aggregation on GeoShape field type.(#3980) (#4266)
navneet1v Sep 19, 2022
658f7a6
Updates jackson and snakeyml dependencies (#4556)
owaiskazi19 Sep 19, 2022
f248394
Bump com.diffplug.spotless from 6.10.0 to 6.11.0 (#4547)
dependabot[bot] Sep 20, 2022
4ea3231
Bump reactor-core from 3.4.18 to 3.4.23 in /plugins/repository-azure …
dependabot[bot] Sep 20, 2022
4d264b6
Bump jempbox from 1.8.16 to 1.8.17 in /plugins/ingest-attachment (#4550)
dependabot[bot] Sep 20, 2022
2c27dfd
Bump log4j-core from 2.18.0 to 2.19.0 in /buildSrc/src/testKit/thirdP…
dependabot[bot] Sep 20, 2022
7ebb2af
Ignore lock file when testing cleanupAndPreserveLatestCommitPoint (#4…
dbwiddis Sep 21, 2022
3ca749f
Remove deprecated code to add node name into log pattern of log4j pro…
Sep 22, 2022
08f2232
Unmute RelocationIT.testRelocationWhileIndexingRandom. (#4580)
mch2 Sep 23, 2022
751d069
Add DecommissionService and helper to execute awareness attribute dec…
imRishN Sep 25, 2022
bb47419
Added rest layer changes for List all PITs and PIT segments (#4388)
bharath-techie Sep 25, 2022
9b2202a
Add PUT api to update shard routing weights (#4272)
anshu1106 Sep 27, 2022
ab6849f
Further simplification of the ZIP publication implementation (#4360)
lukas-vlcek Sep 27, 2022
ebbddd3
Add missing Javadoc tag descriptions in unit tests (#4629)
msfroh Sep 29, 2022
cb5e16d
Bump reactor-netty-http from 1.0.18 to 1.0.23 in /plugins/repository-…
dependabot[bot] Sep 29, 2022
7157724
Bump jettison from 1.5.0 to 1.5.1 in /plugins/discovery-azure-classic…
dependabot[bot] Sep 29, 2022
2c0277b
Fix invalid search location of JDK for arm64 (#4613)
heemin32 Sep 29, 2022
2fe6f9a
Remove unused object and import from TransportClusterAllocationExplai…
kotwanikunal Sep 29, 2022
3b49c0e
Relax visibility of the HTTP_CHANNEL_KEY and HTTP_SERVER_CHANNEL_KEY …
reta Sep 30, 2022
551c2d8
Bumping hadoop-hdfs version to 3.3.4 (#4644)
saratvemulapalli Sep 30, 2022
54f8fdd
Bugfix/496 rollover alias with filters (#4499)
stevanbz Sep 30, 2022
ff2d5be
Load the deprecated master role in a dedicated method instead of in s…
Sep 30, 2022
4b08d38
Add APIs (GET/PUT) to decommission awareness attribute (#4261)
imRishN Oct 2, 2022
76c372f
Adding check to make sure checkpoint is not processed when a shard's …
Rishikesh1159 Oct 3, 2022
9e5ea46
Bump azure-storage-common from 12.18.0 to 12.18.1 in /plugins/reposit…
dependabot[bot] Oct 3, 2022
b074c0f
Bump forbiddenapis from 3.3 to 3.4 in /buildSrc/src/testKit/thirdPart…
dependabot[bot] Oct 3, 2022
c20446d
Fixed the SnapshotsInProgress error during index deletion (#4570)
xuezhou25 Oct 3, 2022
69f421f
Update to Apache Lucene 9.4.0 (#4661)
reta Oct 3, 2022
a17692d
Fixed day of year defaulting for round up parser (#4627)
Vishalks Oct 3, 2022
0c80a7a
Fix PIT flaky tests (#4632)
bharath-techie Oct 4, 2022
7b7ca98
Add getter for path field in NestedQueryBuilder (#4636)
psmagin Oct 4, 2022
41f2b9a
Pre-commit checks now check for missing javadoc, and the Gradle check…
yeohbraddy Oct 4, 2022
89aaa9e
Backport Apache Lucene version change for 2.4.0 (#4677)
reta Oct 4, 2022
e403799
Attempt to fix Github workflow for Gradle Check job (#4679)
reta Oct 4, 2022
05d11b2
Adding precommit support for Windows (#4676)
saratvemulapalli Oct 4, 2022
ada2467
Adding precommit support for MacOS (darwin) (#4682)
saratvemulapalli Oct 5, 2022
e189179
Fix flaky DecommissionControllerTests.testTimesOut (#4683)
andrross Oct 5, 2022
3941226
Include Windows OS in Bootstrap initializeNatives() check for definit…
dbwiddis Oct 5, 2022
e354461
Add release notes for 1.3.6 (#4681)
Rishikesh1159 Oct 5, 2022
3186935
Service Layer changes for Recommission API (#4320)
pranikum Oct 6, 2022
563122e
Add GET api to get shard routing weights (#4275)
anshu1106 Oct 6, 2022
ed359f0
Fix new race condition in DecommissionControllerTests (#4688)
andrross Oct 6, 2022
a2f95ce
Fixing SearchStats (de)serialization (#4697)
reta Oct 6, 2022
f5ea1a4
Fixed misunderstanding message 'No OpenSearchException found' when de…
xuezhou25 Oct 7, 2022
b1bc4d9
Update GeoGrid base class access modifier to support extensibility (#…
VijayanB Oct 7, 2022
ef45809
[Remove] LegacyESVersion.V_7_0_* and V_7_1_* constants (#2768)
nknize Oct 7, 2022
588db38
Fixing Gradle warnings associated with publishPluginZipPublicationToX…
reta Oct 7, 2022
ef50f78
Bump gson from 2.9.0 to 2.9.1 in /test/fixtures/hdfs-fixture (#4066)
dependabot[bot] Oct 7, 2022
109319e
Always auto release the flood stage block (#4703)
nknize Oct 7, 2022
2e4b27b
Controlling discovery for decommissioned nodes (#4590)
imRishN Oct 7, 2022
fe3994c
[Remove] LegacyESVersion.V_7_2_* and V_7_3_* constants (#4702)
nknize Oct 7, 2022
77cff55
[Remove] LegacyESVersion.V_7_4_* and V_7_5_* constants (#4704)
nknize Oct 7, 2022
7cdffe2
Merge remote-tracking branch 'upstream/main' into rebase-throttling-0…
dhwanilpatel Oct 8, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Always auto release the flood stage block (#4703)
* Always auto release the flood stage block

Removes support for using a system property to disable the automatic release of
the write block applied when a node exceeds the flood-stage watermark.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* update IAE message

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
  • Loading branch information
nknize authored Oct 7, 2022
commit 109319e0724d5dfa46f5554a7ef819560eaa1fac
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Remove deprecated code to add node name into log pattern of log4j property file ([#4568](https://github.com/opensearch-project/OpenSearch/pull/4568))
- Unused object and import within TransportClusterAllocationExplainAction ([#4639](https://github.com/opensearch-project/OpenSearch/pull/4639))
- Remove LegacyESVersion.V_7_0_* and V_7_1_* Constants ([#2768](https://https://github.com/opensearch-project/OpenSearch/pull/2768))
- Always auto release the flood stage block ([#4703](https://github.com/opensearch-project/OpenSearch/pull/4703))

### Fixed

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.message.ParameterizedMessage;
import org.opensearch.LegacyESVersion;
import org.opensearch.action.ActionListener;
import org.opensearch.action.support.GroupedActionListener;
import org.opensearch.client.Client;
Expand All @@ -54,7 +53,6 @@
import org.opensearch.common.Priority;
import org.opensearch.common.Strings;
import org.opensearch.common.collect.ImmutableOpenMap;
import org.opensearch.common.logging.DeprecationLogger;
import org.opensearch.common.settings.ClusterSettings;
import org.opensearch.common.settings.Settings;
import org.opensearch.common.util.set.Sets;
Expand Down Expand Up @@ -88,7 +86,6 @@ public class DiskThresholdMonitor {
private final RerouteService rerouteService;
private final AtomicLong lastRunTimeMillis = new AtomicLong(Long.MIN_VALUE);
private final AtomicBoolean checkInProgress = new AtomicBoolean();
private final DeprecationLogger deprecationLogger = DeprecationLogger.getLogger(logger.getName());

/**
* The IDs of the nodes that were over the low threshold in the last check (and maybe over another threshold too). Tracked so that we
Expand Down Expand Up @@ -121,14 +118,6 @@ public DiskThresholdMonitor(
this.rerouteService = rerouteService;
this.diskThresholdSettings = new DiskThresholdSettings(settings, clusterSettings);
this.client = client;
if (diskThresholdSettings.isAutoReleaseIndexEnabled() == false) {
deprecationLogger.deprecate(
DiskThresholdSettings.AUTO_RELEASE_INDEX_ENABLED_KEY.replace(".", "_"),
"[{}] will be removed in version {}",
DiskThresholdSettings.AUTO_RELEASE_INDEX_ENABLED_KEY,
LegacyESVersion.V_7_4_0.major + 1
);
}
}

private void checkFinished() {
Expand Down Expand Up @@ -371,23 +360,7 @@ public void onNewInfo(ClusterInfo info) {
.collect(Collectors.toSet());

if (indicesToAutoRelease.isEmpty() == false) {
if (diskThresholdSettings.isAutoReleaseIndexEnabled()) {
logger.info("releasing read-only-allow-delete block on indices: [{}]", indicesToAutoRelease);
updateIndicesReadOnly(indicesToAutoRelease, listener, false);
} else {
deprecationLogger.deprecate(
DiskThresholdSettings.AUTO_RELEASE_INDEX_ENABLED_KEY.replace(".", "_"),
"[{}] will be removed in version {}",
DiskThresholdSettings.AUTO_RELEASE_INDEX_ENABLED_KEY,
LegacyESVersion.V_7_4_0.major + 1
);
logger.debug(
"[{}] disabled, not releasing read-only-allow-delete block on indices: [{}]",
DiskThresholdSettings.AUTO_RELEASE_INDEX_ENABLED_KEY,
indicesToAutoRelease
);
listener.onResponse(null);
}
updateIndicesReadOnly(indicesToAutoRelease, listener, false);
} else {
logger.trace("no auto-release required");
listener.onResponse(null);
Expand Down Expand Up @@ -421,11 +394,9 @@ private void markNodesMissingUsageIneligibleForRelease(
) {
for (RoutingNode routingNode : routingNodes) {
if (usages.containsKey(routingNode.nodeId()) == false) {
if (routingNode != null) {
for (ShardRouting routing : routingNode) {
String indexName = routing.index().getName();
indicesToMarkIneligibleForAutoRelease.add(indexName);
}
for (ShardRouting routing : routingNode) {
String indexName = routing.index().getName();
indicesToMarkIneligibleForAutoRelease.add(indexName);
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
package org.opensearch.cluster.routing.allocation;

import org.opensearch.OpenSearchParseException;
import org.opensearch.Version;
import org.opensearch.common.Strings;
import org.opensearch.common.settings.ClusterSettings;
import org.opensearch.common.settings.Setting;
Expand Down Expand Up @@ -108,18 +109,15 @@ public class DiskThresholdSettings {
private volatile TimeValue rerouteInterval;
private volatile Double freeDiskThresholdFloodStage;
private volatile ByteSizeValue freeBytesThresholdFloodStage;
private static final boolean autoReleaseIndexEnabled;
public static final String AUTO_RELEASE_INDEX_ENABLED_KEY = "opensearch.disk.auto_release_flood_stage_block";

static {
assert Version.CURRENT.major == Version.V_2_0_0.major + 1; // this check is unnecessary in v4
final String AUTO_RELEASE_INDEX_ENABLED_KEY = "opensearch.disk.auto_release_flood_stage_block";

final String property = System.getProperty(AUTO_RELEASE_INDEX_ENABLED_KEY);
if (property == null) {
autoReleaseIndexEnabled = true;
} else if (Boolean.FALSE.toString().equals(property)) {
autoReleaseIndexEnabled = false;
} else {
if (property != null) {
throw new IllegalArgumentException(
AUTO_RELEASE_INDEX_ENABLED_KEY + " may only be unset or set to [false] but was [" + property + "]"
"system property [" + AUTO_RELEASE_INDEX_ENABLED_KEY + "] has been removed in 3.0.0 and is not supported anymore"
);
}
}
Expand Down Expand Up @@ -371,10 +369,6 @@ public ByteSizeValue getFreeBytesThresholdFloodStage() {
return freeBytesThresholdFloodStage;
}

public boolean isAutoReleaseIndexEnabled() {
return autoReleaseIndexEnabled;
}

public boolean includeRelocations() {
return includeRelocations;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ public void testDefaults() {
assertTrue(diskThresholdSettings.includeRelocations());
assertEquals(zeroBytes, diskThresholdSettings.getFreeBytesThresholdFloodStage());
assertEquals(5.0D, diskThresholdSettings.getFreeDiskThresholdFloodStage(), 0.0D);
assertTrue(diskThresholdSettings.isAutoReleaseIndexEnabled());
}

public void testUpdate() {
Expand Down