Merge branch 'main' into issue-13303

Signed-off-by: Jay Deng <jayd0104@gmail.com>

.github/CODEOWNERS

@@ -11,7 +11,7 @@
 #   3. Use the command palette to run the CODEOWNERS: Show owners of current file command, which will display all code owners for the current file.
 
 # Default ownership for all repo files
-* @anasalkouz @andrross @Bukhtawar @CEHENKLE @dblock @dbwiddis @dreamer-89 @gbbafna @kotwanikunal @mch2 @msfroh @nknize @owaiskazi19 @reta @Rishikesh1159 @sachinpkale @saratvemulapalli @shwetathareja @sohami @tlfeng @VachaShah
+* @anasalkouz @andrross @Bukhtawar @CEHENKLE @dblock @dbwiddis @gbbafna @kotwanikunal @mch2 @msfroh @nknize @owaiskazi19 @reta @Rishikesh1159 @sachinpkale @saratvemulapalli @shwetathareja @sohami @VachaShah
 
 /modules/transport-netty4/ @peternied
 
@@ -24,4 +24,4 @@
 
 /.github/ @peternied
 
-/MAINTAINERS.md @anasalkouz @andrross @Bukhtawar @CEHENKLE @dblock @dbwiddis @dreamer-89 @gbbafna @jed326 @kotwanikunal @mch2 @msfroh @nknize @owaiskazi19 @peternied @reta @Rishikesh1159 @sachinpkale @saratvemulapalli @shwetathareja @sohami @tlfeng @VachaShah
+/MAINTAINERS.md @anasalkouz @andrross @Bukhtawar @CEHENKLE @dblock @dbwiddis @gbbafna @jed326 @kotwanikunal @mch2 @msfroh @nknize @owaiskazi19 @peternied @reta @Rishikesh1159 @sachinpkale @saratvemulapalli @shwetathareja @sohami @VachaShah

CHANGELOG.md

@@ -7,6 +7,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ### Added
 - Add fingerprint ingest processor ([#13724](https://github.com/opensearch-project/OpenSearch/pull/13724))
 - [Remote Store] Rate limiter for remote store low priority uploads ([#14374](https://github.com/opensearch-project/OpenSearch/pull/14374/))
+- Apply the date histogram rewrite optimization to range aggregation ([#13865](https://github.com/opensearch-project/OpenSearch/pull/13865))
 
 ### Dependencies
 - Bump `org.gradle.test-retry` from 1.5.8 to 1.5.9 ([#13442](https://github.com/opensearch-project/OpenSearch/pull/13442))
@@ -19,8 +20,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `com.nimbusds:nimbus-jose-jwt` from 9.37.3 to 9.40 ([#14398](https://github.com/opensearch-project/OpenSearch/pull/14398))
 - Bump `org.apache.commons:commons-configuration2` from 2.10.1 to 2.11.0 ([#14399](https://github.com/opensearch-project/OpenSearch/pull/14399))
 - Bump `com.gradle.develocity` from 3.17.4 to 3.17.5 ([#14397](https://github.com/opensearch-project/OpenSearch/pull/14397))
+- Bump `opentelemetry` from 1.36.0 to 1.39.0 ([#14457](https://github.com/opensearch-project/OpenSearch/pull/14457))
 
 ### Changed
+- unsignedLongRangeQuery now returns MatchNoDocsQuery if the lower bounds are greater than the upper bounds ([#14416](https://github.com/opensearch-project/OpenSearch/pull/14416))
 - Updated the `indices.query.bool.max_clause_count` setting from being static to dynamically updateable ([#13568](https://github.com/opensearch-project/OpenSearch/pull/13568))
 - Make the class CommunityIdProcessor final ([#14448](https://github.com/opensearch-project/OpenSearch/pull/14448))
 
@@ -31,6 +34,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ### Fixed
 - Fix handling of Short and Byte data types in ScriptProcessor ingest pipeline ([#14379](https://github.com/opensearch-project/OpenSearch/issues/14379))
 - Switch to iterative version of WKT format parser ([#14086](https://github.com/opensearch-project/OpenSearch/pull/14086))
+- Fix the computed max shards of cluster to avoid int overflow ([#14155](https://github.com/opensearch-project/OpenSearch/pull/14155))
 - Fix aggs result of NestedAggregator with sub NestedAggregator ([#13324](https://github.com/opensearch-project/OpenSearch/pull/13324))
 
 ### Security

MAINTAINERS.md

@@ -26,18 +26,18 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
 | Sarat Vemulapalli        | [saratvemulapalli](https://github.com/saratvemulapalli) | Amazon      |
 | Shweta Thareja           | [shwetathareja](https://github.com/shwetathareja)       | Amazon      |
 | Sorabh Hamirwasia        | [sohami](https://github.com/sohami)                     | Amazon      |
-| Suraj Singh              | [dreamer-89](https://github.com/dreamer-89)             | Amazon      |
-| Tianli Feng              | [tlfeng](https://github.com/tlfeng)                     | Amazon      |
 | Vacha Shah               | [VachaShah](https://github.com/VachaShah)               | Amazon      |
 
 ## Emeritus
 
-| Maintainer            | GitHub ID                                 | Affiliation |
-| --------------------- | ----------------------------------------- | ----------- |
-| Megha Sai Kavikondala | [meghasaik](https://github.com/meghasaik) | Amazon      |
-| Xue Zhou              | [xuezhou25](https://github.com/xuezhou25) | Amazon      |
-| Kartik Ganesh         | [kartg](https://github.com/kartg)         | Amazon      |
-| Abbas Hussain         | [abbashus](https://github.com/abbashus)   | Meta        |
-| Himanshu Setia        | [setiah](https://github.com/setiah)       | Amazon      |
-| Ryan Bogan            | [ryanbogan](https://github.com/ryanbogan) | Amazon      |
-| Rabi Panda            | [adnapibar](https://github.com/adnapibar) | Independent |
+| Maintainer             | GitHub ID                                   | Affiliation |
+| ---------------------- |-------------------------------------------- | ----------- |
+| Megha Sai Kavikondala  | [meghasaik](https://github.com/meghasaik)   | Amazon      |
+| Xue Zhou               | [xuezhou25](https://github.com/xuezhou25)   | Amazon      |
+| Kartik Ganesh          | [kartg](https://github.com/kartg)           | Amazon      |
+| Abbas Hussain          | [abbashus](https://github.com/abbashus)     | Meta        |
+| Himanshu Setia         | [setiah](https://github.com/setiah)         | Amazon      |
+| Ryan Bogan             | [ryanbogan](https://github.com/ryanbogan)   | Amazon      |
+| Rabi Panda             | [adnapibar](https://github.com/adnapibar)   | Independent |
+| Tianli Feng            | [tlfeng](https://github.com/tlfeng)         | Amazon      |
+| Suraj Singh            | [dreamer-89](https://github.com/dreamer-89) | Amazon      |

TESTING.md

@@ -33,6 +33,9 @@ OpenSearch uses [jUnit](https://junit.org/junit5/) for testing, it also uses ran
   - [Bad practices](#bad-practices)
     - [Use randomized-testing for coverage](#use-randomized-testing-for-coverage)
     - [Abuse randomization in multi-threaded tests](#abuse-randomization-in-multi-threaded-tests)
+    - [Use `Thread.sleep`](#use-threadsleep)
+    - [Expect a specific segment topology](#expect-a-specific-segment-topology)
+    - [Leave environment in an unstable state after test](#leave-environment-in-an-unstable-state-after-test)
 - [Test coverage analysis](#test-coverage-analysis)
 - [Building with extra plugins](#building-with-extra-plugins)
 - [Environment misc](#environment-misc)
@@ -455,7 +458,7 @@ Unit tests are the preferred way to test some functionality: most of the time th
 
 The reason why `OpenSearchSingleNodeTestCase` exists is that all our components used to be very hard to set up in isolation, which had led us to having a number of integration tests but close to no unit tests. `OpenSearchSingleNodeTestCase` is a workaround for this issue which provides an easy way to spin up a node and get access to components that are hard to instantiate like `IndicesService`. Whenever practical, you should prefer unit tests.
 
-Finally, if the the functionality under test needs to be run in a cluster, there are two test classes to consider:
+Finally, if the functionality under test needs to be run in a cluster, there are two test classes to consider:
   * `OpenSearchRestTestCase` will connect to an external cluster. This is a good option if the tests cases don't rely on a specific configuration of the test cluster. A test cluster is set up as part of the Gradle task running integration tests, and test cases using this class can connect to it. The configuration of the cluster is provided in the Gradle files.
   * `OpenSearchIntegTestCase` will create a local cluster as part of each test case. The configuration of the cluster is controlled by the test class. This is a good option if different tests cases depend on different cluster configurations, as it would be impractical (and limit parallelization) to keep re-configuring (and re-starting) the external cluster for each test case. A good example of when this class might come in handy is for testing security features, where different cluster configurations are needed to fully test each one.
 
@@ -477,6 +480,27 @@ However, it should not be used for coverage. For instance if you are testing a p
 
 Multi-threaded tests are often not reproducible due to the fact that there is no guarantee on the order in which operations occur across threads. Adding randomization to the mix usually makes things worse and should be done with care.
 
+### Use `Thread.sleep`
+
+`Thread.sleep()` is almost always a bad idea because it is very difficult to know that you've waited long enough. Using primitives like `waitUntil` or `assertBusy`, which use Thread.sleep internally, is okay to wait for a specific condition. However, it is almost always better to instrument your code with concurrency primitives like a `CountDownLatch` that will allow you to deterministically wait for a specific condition, without waiting longer than necessary that will happen with a polling approach used by `assertBusy`.
+
+Example:
+- [PrimaryShardAllocatorIT](https://github.com/opensearch-project/OpenSearch/blob/7ffcd6500e0bd5956cef5c289ee66d9f99d533fc/server/src/internalClusterTest/java/org/opensearch/gateway/ReplicaShardAllocatorIT.java#L208-L235): This test is using two latches: one to wait for a recovery to start and one to block that recovery so that it can deterministically test things that happen during a recovery.
+
+### Expect a specific segment topology
+
+By design, OpenSearch integration tests will vary how the merge policy works because in almost all scenarios you should not depend on a specific segment topology (in the real world your code will see a huge diversity of indexing workloads with OpenSearch merging things in the background all the time!). If you do in fact need to care about the segment topology (e.g. for testing statistics that might vary slightly depending on number of segments), then you must take care to ensure that segment topology is deterministic by doing things like disabling background refreshes, force merging after indexing data, etc.
+
+Example:
+- [SegmentReplicationResizeRequestIT](https://github.com/opensearch-project/OpenSearch/blob/f715ee1a485e550802accc1c2e3d8101208d4f0b/server/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationResizeRequestIT.java#L102-L109): This test disables refreshes to prevent interfering with the segment replication behavior under test.
+
+### Leave environment in an unstable state after test
+
+The default test case will ensure that no open file handles or running threads are left after tear down. You must ensure that all resources are cleaned up at the end of each test case, or else the cleanup may end up racing with the tear down logic in the base test class in a way that is very difficult to reproduce.
+
+Example:
+- [AwarenessAttributeDecommissionIT](https://github.com/opensearch-project/OpenSearch/blob/main/server/src/internalClusterTest/java/org/opensearch/cluster/coordination/AwarenessAttributeDecommissionIT.java#L951): Recommissions any decommissioned nodes at the end of the test to ensure the after-test checks succeed.
+
 # Test coverage analysis
 
 The code coverage report can be generated through Gradle with [JaCoCo plugin](https://docs.gradle.org/current/userguide/jacoco_plugin.html).

buildSrc/version.properties

@@ -74,5 +74,5 @@ jzlib             = 1.1.3
 resteasy          = 6.2.4.Final
 
 # opentelemetry dependencies
-opentelemetry         = 1.36.0
-opentelemetrysemconv  = 1.23.1-alpha
+opentelemetry         = 1.39.0
+opentelemetrysemconv  = 1.25.0-alpha

modules/mapper-extras/src/main/java/org/opensearch/index/mapper/ScaledFloatFieldMapper.java

@@ -35,6 +35,7 @@
 import com.fasterxml.jackson.core.JsonParseException;
 
 import org.apache.lucene.document.Field;
+import org.apache.lucene.document.LongPoint;
 import org.apache.lucene.index.DocValues;
 import org.apache.lucene.index.LeafReaderContext;
 import org.apache.lucene.index.NumericDocValues;
@@ -165,7 +166,7 @@ public ScaledFloatFieldMapper build(BuilderContext context) {
 
     public static final TypeParser PARSER = new TypeParser((n, c) -> new Builder(n, c.getSettings()));
 
-    public static final class ScaledFloatFieldType extends SimpleMappedFieldType {
+    public static final class ScaledFloatFieldType extends SimpleMappedFieldType implements NumericPointEncoder {
 
         private final double scalingFactor;
         private final Double nullValue;
@@ -188,6 +189,21 @@ public ScaledFloatFieldType(String name, double scalingFactor) {
             this(name, true, false, true, Collections.emptyMap(), scalingFactor, null);
         }
 
+        @Override
+        public byte[] encodePoint(Number value) {
+            assert value instanceof Double;
+            double doubleValue = (Double) value;
+            byte[] point = new byte[Long.BYTES];
+            if (doubleValue == Double.POSITIVE_INFINITY) {
+                LongPoint.encodeDimension(Long.MAX_VALUE, point, 0);
+            } else if (doubleValue == Double.NEGATIVE_INFINITY) {
+                LongPoint.encodeDimension(Long.MIN_VALUE, point, 0);
+            } else {
+                LongPoint.encodeDimension(Math.round(scale(value)), point, 0);
+            }
+            return point;
+        }
+
         public double getScalingFactor() {
             return scalingFactor;
         }

plugins/telemetry-otel/build.gradle

@@ -37,7 +37,7 @@ dependencies {
   runtimeOnly "com.squareup.okhttp3:okhttp:4.11.0"
   runtimeOnly "com.squareup.okio:okio-jvm:3.5.0"
   runtimeOnly "io.opentelemetry:opentelemetry-exporter-sender-okhttp:${versions.opentelemetry}"
-  api "io.opentelemetry:opentelemetry-extension-incubator:${versions.opentelemetry}-alpha"
+  api "io.opentelemetry:opentelemetry-api-incubator:${versions.opentelemetry}-alpha"
   testImplementation "io.opentelemetry:opentelemetry-sdk-testing:${versions.opentelemetry}"
 }
 
@@ -48,7 +48,9 @@ thirdPartyAudit {
    'io.opentelemetry.internal.shaded.jctools.queues.MpscArrayQueueProducerIndexField',
    'io.opentelemetry.internal.shaded.jctools.queues.MpscArrayQueueProducerLimitField',
    'io.opentelemetry.internal.shaded.jctools.util.UnsafeAccess',
-   'io.opentelemetry.internal.shaded.jctools.util.UnsafeRefArrayAccess'
+   'io.opentelemetry.internal.shaded.jctools.util.UnsafeRefArrayAccess',
+   'io.opentelemetry.exporter.internal.marshal.UnsafeAccess',
+   'io.opentelemetry.exporter.internal.marshal.UnsafeAccess$UnsafeHolder'
   )
 
   ignoreMissingClasses(
@@ -78,10 +80,6 @@ thirdPartyAudit {
     'org.conscrypt.ConscryptHostnameVerifier',
     'org.openjsse.javax.net.ssl.SSLParameters',
     'org.openjsse.javax.net.ssl.SSLSocket',
-    'io.opentelemetry.api.events.EventBuilder',
-    'io.opentelemetry.api.events.EventEmitter',
-    'io.opentelemetry.api.events.EventEmitterBuilder',
-    'io.opentelemetry.api.events.EventEmitterProvider',
     'io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties',
     'io.opentelemetry.sdk.autoconfigure.spi.logs.ConfigurableLogRecordExporterProvider',
     'io.opentelemetry.sdk.autoconfigure.spi.metrics.ConfigurableMetricExporterProvider',

plugins/telemetry-otel/licenses/opentelemetry-api-1.39.0.jar.sha1

@@ -0,0 +1 @@
+396b89a66526bd5694ad3bef4604b876177e0b44

plugins/telemetry-otel/licenses/opentelemetry-api-incubator-1.39.0-alpha.jar.sha1

@@ -0,0 +1 @@
+1a1fd96155e1b58726300bbf8457630713035e51

plugins/telemetry-otel/licenses/opentelemetry-context-1.39.0.jar.sha1

@@ -0,0 +1 @@
+f0601fb1c06f661afeffbc73a1dbe29797b2f13b

plugins/telemetry-otel/licenses/opentelemetry-exporter-common-1.39.0.jar.sha1

@@ -0,0 +1 @@
+570d71e39e36fe2caad142557bde0c11fcdb3b92

plugins/telemetry-otel/licenses/opentelemetry-exporter-logging-1.39.0.jar.sha1

@@ -0,0 +1 @@
+f5b528f8d6f8531836eabba698979516964b24ed

plugins/telemetry-otel/licenses/opentelemetry-exporter-otlp-1.39.0.jar.sha1

@@ -0,0 +1 @@
+04fc0e4983253ea58430c3d24b6b3c5c95f84dc9

plugins/telemetry-otel/licenses/opentelemetry-exporter-otlp-common-1.39.0.jar.sha1

@@ -0,0 +1 @@
+a2b8571e36b11c3153d31ec87ec69cc168af8036

plugins/telemetry-otel/licenses/opentelemetry-exporter-sender-okhttp-1.39.0.jar.sha1

@@ -0,0 +1 @@
+1a8947a2e28924ad9374e319150a23837926ca4b

plugins/telemetry-otel/licenses/opentelemetry-sdk-1.39.0.jar.sha1

@@ -0,0 +1 @@
+ba9afdf3ef1ea51e42999fd68c959e3ceb219399

plugins/telemetry-otel/licenses/opentelemetry-sdk-common-1.39.0.jar.sha1

@@ -0,0 +1 @@
+fb8168627bf0059445f61081eaa47c4ab787fc2e

plugins/telemetry-otel/licenses/opentelemetry-sdk-logs-1.39.0.jar.sha1

@@ -0,0 +1 @@
+b6b45155399bc9fa563945f3e3a77416d7165948

plugins/telemetry-otel/licenses/opentelemetry-sdk-metrics-1.39.0.jar.sha1

@@ -0,0 +1 @@
+522d46926cc06a4c18829da7e4c4340bdf5673c3

plugins/telemetry-otel/licenses/opentelemetry-sdk-trace-1.39.0.jar.sha1

@@ -0,0 +1 @@
+0b72722a5bbea5f46319bf08b2caed5b8f987a92

plugins/telemetry-otel/licenses/opentelemetry-semconv-1.25.0-alpha.jar.sha1

@@ -0,0 +1 @@
+76b3d4ca0a8f20b27c1590ceece54f0c7fb5857e