Adds method_parameters in neural search query to support ef_search #787

shatejas · 2024-06-11T21:06:22Z

Description

Adds method_parameters in neural search

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

martin-gaievski

changelog entry is missing

src/main/java/org/opensearch/neuralsearch/common/MinClusterVersionUtil.java

src/main/java/org/opensearch/neuralsearch/query/NeuralQueryBuilder.java

shatejas · 2024-06-12T23:51:10Z

changelog entry is missing

Its being merged in feature branch. Was planning to add it when merging in main. does that work?

navneet1v · 2024-06-13T18:07:08Z

src/main/java/org/opensearch/neuralsearch/query/NeuralQueryBuilder.java

@@ -174,6 +178,9 @@ protected void doXContent(XContentBuilder xContentBuilder, Params params) throws
        if (Objects.nonNull(minScore)) {
            xContentBuilder.field(MIN_SCORE_FIELD.getPreferredName(), minScore);
        }
+        if (Objects.nonNull(methodParameters)) {
+            MethodParametersParser.doXContent(xContentBuilder, methodParameters);


is this class MethodParametersParser coming from k-NN?

This is good. Lets ensure that we add some more docs on the MethodParametersParser function that it is now going beyond the boundaries of k-NN plugin. I would hate to see if we change signature of these functions and then neural plugin starts to break.

https://github.com/opensearch-project/k-NN/pull/1758/files

martin-gaievski

We need to think more on how to commonize logic that's shared between neural search and knn plugins. May be a too heavy lift for this change. At the moment this PR is good, and I see we do have an issue to follow up on later: opensearch-project/k-NN#1756

martin-gaievski · 2024-06-14T00:43:38Z

Please check why CI is failing, probably need to rebase on latest main

shatejas · 2024-06-14T17:17:24Z

Please check why CI is failing, probably need to rebase on latest main

The build is failing because the knn changes aren't in main yet. It is expected to fail as it cannot identify MethodParameterParser. Locally I published to local maven repo for neural to build

navneet1v · 2024-06-17T20:21:51Z

@vibrantvarun can you check

qa/restart-upgrade/src/test/java/org/opensearch/neuralsearch/bwc/HybridSearchIT.java

vibrantvarun · 2024-06-17T20:26:35Z

qa/rolling-upgrade/src/test/java/org/opensearch/neuralsearch/bwc/HybridSearchIT.java

@@ -73,6 +73,7 @@ public void testNormalizationProcessor_whenIndexWithMultipleShards_E2EFlow() thr
                    loadModel(modelId);
                    addDocument(getIndexNameForTest(), "2", TEST_FIELD, TEXT_UPGRADED, null, null);
                    validateTestIndexOnUpgrade(totalDocsCountUpgraded, modelId);
+                    validateTestIndexOnUpgrade(totalDocsCountUpgraded, modelId, Map.of("ef_search", 100));


Same comment as above.

Same reply as above

qa/rolling-upgrade/src/test/resources/processor/IndexMappings.json

vibrantvarun · 2024-06-17T20:43:52Z

There are lot of compilation errors which are not related to KNN. @shatejas Can you check why it is happening?

Signed-off-by: Tejas Shah <shatejas@amazon.com>

shatejas · 2024-06-17T20:57:46Z

There are lot of compilation errors which are not related to KNN. @shatejas Can you check why it is happening?

See this comment

…pensearch-project#787) Signed-off-by: Tejas Shah <shatejas@amazon.com>

) (#814) Signed-off-by: Tejas Shah <shatejas@amazon.com>

) (#814) Signed-off-by: Tejas Shah <shatejas@amazon.com> (cherry picked from commit dea1fe3)

) (#814) (#818) Signed-off-by: Tejas Shah <shatejas@amazon.com> (cherry picked from commit dea1fe3) Co-authored-by: Tejas Shah <shatejas@amazon.com>

…pensearch-project#787) (opensearch-project#814) Signed-off-by: Tejas Shah <shatejas@amazon.com>

* Adds method_parameters in neural search query to support ef_search (#787) (#814) Signed-off-by: Tejas Shah <shatejas@amazon.com> * Add BWC for batch ingestion (#769) * Add BWC for batch ingestion Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Update Changelog Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Fix spotlessLicenseCheck Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Fix comments Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Reuse the same code Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rename some functions Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rename a function Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Minor change to trigger rebuild Signed-off-by: Liyun Xiu <xiliyun@amazon.com> --------- Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Neural sparse query two-phase search processor's bwc test (#777) * Poc of pipeline Signed-off-by: conggguan <congguan@amazon.com> * Complete some settings for two phase pipeline. Signed-off-by: conggguan <congguan@amazon.com> * Change the implement of two-phase from QueryBuilderVistor to custom process funciton. Signed-off-by: conggguan <congguan@amazon.com> * Add It and fix some bug on the state of multy same neuralsparsequerybuilder. Signed-off-by: conggguan <congguan@amazon.com> * Simplify some logic, and correct some format. Signed-off-by: conggguan <congguan@amazon.com> * Optimize some format. Signed-off-by: conggguan <congguan@amazon.com> * Add some test case. Signed-off-by: conggguan <congguan@amazon.com> * Optimize some logic for zhichao-aws's comments. Signed-off-by: conggguan <congguan@amazon.com> * Optimize a line without application. Signed-off-by: conggguan <congguan@amazon.com> * Add some comments, remove some redundant lines, fix some format. Signed-off-by: conggguan <congguan@amazon.com> * Remove a redundant null check, fix a if format. Signed-off-by: conggguan <congguan@amazon.com> * Fix a typo for a comment, camelcase format for some variable. Signed-off-by: conggguan <congguan@amazon.com> * Add some comments to illustrate the influence of the modify on 2-phase search pipeline to neural sparse query builder. Signed-off-by: conggguan <congguan@amazon.com> * Add restart and rolling upgrade bwc test for neural sparse two phase processor. Signed-off-by: conggguan <congguan@amazon.com> * Spotless on qa. Signed-off-by: conggguan <congguan@amazon.com> * Update change log for two-phase BWC test. Signed-off-by: conggguan <congguan@amazon.com> * Remove redundant lines of two-phase BWC test. Signed-off-by: conggguan <congguan@amazon.com> * Add changelog. Signed-off-by: conggguan <congguan@amazon.com> * Add the PR link and number for the CHANGELOG.md. Signed-off-by: conggguan <congguan@amazon.com> * [Fix] NeuralSparseTwoPhaseProcessorIT created wrong ingest pipeline, fix it to correct API. Signed-off-by: conggguan <congguan@amazon.com> --------- Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> * Enable '.' for nested field in text embedding processor (#811) * Added nested structure for text embed processor mapping Signed-off-by: Martin Gaievski <gaievski@amazon.com> * Fix linux build CI error due to action runner env upgrade node 20 (#821) * Fix linux build CI error due to action runner env upgrade node 20 Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix linux build on additional integ tests Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Tejas Shah <shatejas@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> Signed-off-by: Martin Gaievski <gaievski@amazon.com> Signed-off-by: Varun Jain <varunudr@amazon.com> Co-authored-by: Tejas Shah <shatejas@amazon.com> Co-authored-by: Liyun Xiu <chishui2@gmail.com> Co-authored-by: conggguan <157357330+conggguan@users.noreply.github.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com>

…827) * Fix jdk version for CI test secure cluster action (#801) (#806) Signed-off-by: Martin Gaievski <gaievski@amazon.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> * [Part 1] Collector for Sorting Results (#797) * [Part 2] Normalization Phase for Sorting (#802) * Normalization Phase for Sorting Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing compile test issue Signed-off-by: Varun Jain <varunudr@amazon.com> * Optimize code Signed-off-by: Varun Jain <varunudr@amazon.com> * Add method description Signed-off-by: Varun Jain <varunudr@amazon.com> * [Part 1] Collector for Sorting Results (#797) * HybridSearchSortUtil class Signed-off-by: Varun Jain <varunudr@amazon.com> * Add Integ Tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Add Sorting Integ tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Add integ test for Sorting Signed-off-by: Varun Jain <varunudr@amazon.com> * Refactoring normalization processor workflow Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix Unit Tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Refactoring Signed-off-by: Varun Jain <varunudr@amazon.com> * Refactoring Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Optimising Normalization Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Vijay comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Vijay Comments Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> * Update bwc workflow to include 2.16.0-SNAPSHOT (#809) (#810) * Increment BWC version * Append 2.16.0-SNAPSHOTn in restart upgrade tests --------- Signed-off-by: Varun Jain <varunudr@amazon.com> * [Part 3] Concurrent segment search bug in Sorting (#808) * Cherry picking Concurrent Segment Search Bug Commit Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix Concurrent Segment Search Bug in Sorting Signed-off-by: Varun Jain <varunudr@amazon.com> * Functional Interface Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Removing comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin commnents Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> * Rebasing with main (#826) * Adds method_parameters in neural search query to support ef_search (#787) (#814) Signed-off-by: Tejas Shah <shatejas@amazon.com> * Add BWC for batch ingestion (#769) * Add BWC for batch ingestion Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Update Changelog Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Fix spotlessLicenseCheck Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Fix comments Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Reuse the same code Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rename some functions Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rename a function Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Minor change to trigger rebuild Signed-off-by: Liyun Xiu <xiliyun@amazon.com> --------- Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Neural sparse query two-phase search processor's bwc test (#777) * Poc of pipeline Signed-off-by: conggguan <congguan@amazon.com> * Complete some settings for two phase pipeline. Signed-off-by: conggguan <congguan@amazon.com> * Change the implement of two-phase from QueryBuilderVistor to custom process funciton. Signed-off-by: conggguan <congguan@amazon.com> * Add It and fix some bug on the state of multy same neuralsparsequerybuilder. Signed-off-by: conggguan <congguan@amazon.com> * Simplify some logic, and correct some format. Signed-off-by: conggguan <congguan@amazon.com> * Optimize some format. Signed-off-by: conggguan <congguan@amazon.com> * Add some test case. Signed-off-by: conggguan <congguan@amazon.com> * Optimize some logic for zhichao-aws's comments. Signed-off-by: conggguan <congguan@amazon.com> * Optimize a line without application. Signed-off-by: conggguan <congguan@amazon.com> * Add some comments, remove some redundant lines, fix some format. Signed-off-by: conggguan <congguan@amazon.com> * Remove a redundant null check, fix a if format. Signed-off-by: conggguan <congguan@amazon.com> * Fix a typo for a comment, camelcase format for some variable. Signed-off-by: conggguan <congguan@amazon.com> * Add some comments to illustrate the influence of the modify on 2-phase search pipeline to neural sparse query builder. Signed-off-by: conggguan <congguan@amazon.com> * Add restart and rolling upgrade bwc test for neural sparse two phase processor. Signed-off-by: conggguan <congguan@amazon.com> * Spotless on qa. Signed-off-by: conggguan <congguan@amazon.com> * Update change log for two-phase BWC test. Signed-off-by: conggguan <congguan@amazon.com> * Remove redundant lines of two-phase BWC test. Signed-off-by: conggguan <congguan@amazon.com> * Add changelog. Signed-off-by: conggguan <congguan@amazon.com> * Add the PR link and number for the CHANGELOG.md. Signed-off-by: conggguan <congguan@amazon.com> * [Fix] NeuralSparseTwoPhaseProcessorIT created wrong ingest pipeline, fix it to correct API. Signed-off-by: conggguan <congguan@amazon.com> --------- Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> * Enable '.' for nested field in text embedding processor (#811) * Added nested structure for text embed processor mapping Signed-off-by: Martin Gaievski <gaievski@amazon.com> * Fix linux build CI error due to action runner env upgrade node 20 (#821) * Fix linux build CI error due to action runner env upgrade node 20 Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix linux build on additional integ tests Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Tejas Shah <shatejas@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> Signed-off-by: Martin Gaievski <gaievski@amazon.com> Signed-off-by: Varun Jain <varunudr@amazon.com> Co-authored-by: Tejas Shah <shatejas@amazon.com> Co-authored-by: Liyun Xiu <chishui2@gmail.com> Co-authored-by: conggguan <157357330+conggguan@users.noreply.github.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> * Add changelog Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Martin Gaievski <gaievski@amazon.com> Signed-off-by: Varun Jain <varunudr@amazon.com> Signed-off-by: Tejas Shah <shatejas@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> Co-authored-by: Tejas Shah <shatejas@amazon.com> Co-authored-by: Liyun Xiu <chishui2@gmail.com> Co-authored-by: conggguan <157357330+conggguan@users.noreply.github.com>

…827) * Fix jdk version for CI test secure cluster action (#801) (#806) Signed-off-by: Martin Gaievski <gaievski@amazon.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> * [Part 1] Collector for Sorting Results (#797) * [Part 2] Normalization Phase for Sorting (#802) * Normalization Phase for Sorting Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing compile test issue Signed-off-by: Varun Jain <varunudr@amazon.com> * Optimize code Signed-off-by: Varun Jain <varunudr@amazon.com> * Add method description Signed-off-by: Varun Jain <varunudr@amazon.com> * [Part 1] Collector for Sorting Results (#797) * HybridSearchSortUtil class Signed-off-by: Varun Jain <varunudr@amazon.com> * Add Integ Tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Add Sorting Integ tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Add integ test for Sorting Signed-off-by: Varun Jain <varunudr@amazon.com> * Refactoring normalization processor workflow Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix Unit Tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Refactoring Signed-off-by: Varun Jain <varunudr@amazon.com> * Refactoring Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Optimising Normalization Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Vijay comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Vijay Comments Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> * Update bwc workflow to include 2.16.0-SNAPSHOT (#809) (#810) * Increment BWC version * Append 2.16.0-SNAPSHOTn in restart upgrade tests --------- Signed-off-by: Varun Jain <varunudr@amazon.com> * [Part 3] Concurrent segment search bug in Sorting (#808) * Cherry picking Concurrent Segment Search Bug Commit Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix Concurrent Segment Search Bug in Sorting Signed-off-by: Varun Jain <varunudr@amazon.com> * Functional Interface Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Removing comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin commnents Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> * Rebasing with main (#826) * Adds method_parameters in neural search query to support ef_search (#787) (#814) Signed-off-by: Tejas Shah <shatejas@amazon.com> * Add BWC for batch ingestion (#769) * Add BWC for batch ingestion Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Update Changelog Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Fix spotlessLicenseCheck Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Fix comments Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Reuse the same code Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rename some functions Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rename a function Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Minor change to trigger rebuild Signed-off-by: Liyun Xiu <xiliyun@amazon.com> --------- Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Neural sparse query two-phase search processor's bwc test (#777) * Poc of pipeline Signed-off-by: conggguan <congguan@amazon.com> * Complete some settings for two phase pipeline. Signed-off-by: conggguan <congguan@amazon.com> * Change the implement of two-phase from QueryBuilderVistor to custom process funciton. Signed-off-by: conggguan <congguan@amazon.com> * Add It and fix some bug on the state of multy same neuralsparsequerybuilder. Signed-off-by: conggguan <congguan@amazon.com> * Simplify some logic, and correct some format. Signed-off-by: conggguan <congguan@amazon.com> * Optimize some format. Signed-off-by: conggguan <congguan@amazon.com> * Add some test case. Signed-off-by: conggguan <congguan@amazon.com> * Optimize some logic for zhichao-aws's comments. Signed-off-by: conggguan <congguan@amazon.com> * Optimize a line without application. Signed-off-by: conggguan <congguan@amazon.com> * Add some comments, remove some redundant lines, fix some format. Signed-off-by: conggguan <congguan@amazon.com> * Remove a redundant null check, fix a if format. Signed-off-by: conggguan <congguan@amazon.com> * Fix a typo for a comment, camelcase format for some variable. Signed-off-by: conggguan <congguan@amazon.com> * Add some comments to illustrate the influence of the modify on 2-phase search pipeline to neural sparse query builder. Signed-off-by: conggguan <congguan@amazon.com> * Add restart and rolling upgrade bwc test for neural sparse two phase processor. Signed-off-by: conggguan <congguan@amazon.com> * Spotless on qa. Signed-off-by: conggguan <congguan@amazon.com> * Update change log for two-phase BWC test. Signed-off-by: conggguan <congguan@amazon.com> * Remove redundant lines of two-phase BWC test. Signed-off-by: conggguan <congguan@amazon.com> * Add changelog. Signed-off-by: conggguan <congguan@amazon.com> * Add the PR link and number for the CHANGELOG.md. Signed-off-by: conggguan <congguan@amazon.com> * [Fix] NeuralSparseTwoPhaseProcessorIT created wrong ingest pipeline, fix it to correct API. Signed-off-by: conggguan <congguan@amazon.com> --------- Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> * Enable '.' for nested field in text embedding processor (#811) * Added nested structure for text embed processor mapping Signed-off-by: Martin Gaievski <gaievski@amazon.com> * Fix linux build CI error due to action runner env upgrade node 20 (#821) * Fix linux build CI error due to action runner env upgrade node 20 Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix linux build on additional integ tests Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Tejas Shah <shatejas@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> Signed-off-by: Martin Gaievski <gaievski@amazon.com> Signed-off-by: Varun Jain <varunudr@amazon.com> Co-authored-by: Tejas Shah <shatejas@amazon.com> Co-authored-by: Liyun Xiu <chishui2@gmail.com> Co-authored-by: conggguan <157357330+conggguan@users.noreply.github.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> * Add changelog Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Martin Gaievski <gaievski@amazon.com> Signed-off-by: Varun Jain <varunudr@amazon.com> Signed-off-by: Tejas Shah <shatejas@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> Co-authored-by: Tejas Shah <shatejas@amazon.com> Co-authored-by: Liyun Xiu <chishui2@gmail.com> Co-authored-by: conggguan <157357330+conggguan@users.noreply.github.com> (cherry picked from commit d22e1b8)

…827) (#829) * Fix jdk version for CI test secure cluster action (#801) (#806) Signed-off-by: Martin Gaievski <gaievski@amazon.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> * [Part 1] Collector for Sorting Results (#797) * [Part 2] Normalization Phase for Sorting (#802) * Normalization Phase for Sorting Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing compile test issue Signed-off-by: Varun Jain <varunudr@amazon.com> * Optimize code Signed-off-by: Varun Jain <varunudr@amazon.com> * Add method description Signed-off-by: Varun Jain <varunudr@amazon.com> * [Part 1] Collector for Sorting Results (#797) * HybridSearchSortUtil class Signed-off-by: Varun Jain <varunudr@amazon.com> * Add Integ Tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Add Sorting Integ tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Add integ test for Sorting Signed-off-by: Varun Jain <varunudr@amazon.com> * Refactoring normalization processor workflow Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix Unit Tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Refactoring Signed-off-by: Varun Jain <varunudr@amazon.com> * Refactoring Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Optimising Normalization Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Vijay comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Vijay Comments Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> * Update bwc workflow to include 2.16.0-SNAPSHOT (#809) (#810) * Increment BWC version * Append 2.16.0-SNAPSHOTn in restart upgrade tests --------- Signed-off-by: Varun Jain <varunudr@amazon.com> * [Part 3] Concurrent segment search bug in Sorting (#808) * Cherry picking Concurrent Segment Search Bug Commit Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix Concurrent Segment Search Bug in Sorting Signed-off-by: Varun Jain <varunudr@amazon.com> * Functional Interface Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Removing comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Martin commnents Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Address Martin Comments Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> * Rebasing with main (#826) * Adds method_parameters in neural search query to support ef_search (#787) (#814) Signed-off-by: Tejas Shah <shatejas@amazon.com> * Add BWC for batch ingestion (#769) * Add BWC for batch ingestion Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Update Changelog Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Fix spotlessLicenseCheck Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Fix comments Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Reuse the same code Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rename some functions Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rename a function Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Minor change to trigger rebuild Signed-off-by: Liyun Xiu <xiliyun@amazon.com> --------- Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Neural sparse query two-phase search processor's bwc test (#777) * Poc of pipeline Signed-off-by: conggguan <congguan@amazon.com> * Complete some settings for two phase pipeline. Signed-off-by: conggguan <congguan@amazon.com> * Change the implement of two-phase from QueryBuilderVistor to custom process funciton. Signed-off-by: conggguan <congguan@amazon.com> * Add It and fix some bug on the state of multy same neuralsparsequerybuilder. Signed-off-by: conggguan <congguan@amazon.com> * Simplify some logic, and correct some format. Signed-off-by: conggguan <congguan@amazon.com> * Optimize some format. Signed-off-by: conggguan <congguan@amazon.com> * Add some test case. Signed-off-by: conggguan <congguan@amazon.com> * Optimize some logic for zhichao-aws's comments. Signed-off-by: conggguan <congguan@amazon.com> * Optimize a line without application. Signed-off-by: conggguan <congguan@amazon.com> * Add some comments, remove some redundant lines, fix some format. Signed-off-by: conggguan <congguan@amazon.com> * Remove a redundant null check, fix a if format. Signed-off-by: conggguan <congguan@amazon.com> * Fix a typo for a comment, camelcase format for some variable. Signed-off-by: conggguan <congguan@amazon.com> * Add some comments to illustrate the influence of the modify on 2-phase search pipeline to neural sparse query builder. Signed-off-by: conggguan <congguan@amazon.com> * Add restart and rolling upgrade bwc test for neural sparse two phase processor. Signed-off-by: conggguan <congguan@amazon.com> * Spotless on qa. Signed-off-by: conggguan <congguan@amazon.com> * Update change log for two-phase BWC test. Signed-off-by: conggguan <congguan@amazon.com> * Remove redundant lines of two-phase BWC test. Signed-off-by: conggguan <congguan@amazon.com> * Add changelog. Signed-off-by: conggguan <congguan@amazon.com> * Add the PR link and number for the CHANGELOG.md. Signed-off-by: conggguan <congguan@amazon.com> * [Fix] NeuralSparseTwoPhaseProcessorIT created wrong ingest pipeline, fix it to correct API. Signed-off-by: conggguan <congguan@amazon.com> --------- Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> * Enable '.' for nested field in text embedding processor (#811) * Added nested structure for text embed processor mapping Signed-off-by: Martin Gaievski <gaievski@amazon.com> * Fix linux build CI error due to action runner env upgrade node 20 (#821) * Fix linux build CI error due to action runner env upgrade node 20 Signed-off-by: Varun Jain <varunudr@amazon.com> * Fix linux build on additional integ tests Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Tejas Shah <shatejas@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> Signed-off-by: Martin Gaievski <gaievski@amazon.com> Signed-off-by: Varun Jain <varunudr@amazon.com> Co-authored-by: Tejas Shah <shatejas@amazon.com> Co-authored-by: Liyun Xiu <chishui2@gmail.com> Co-authored-by: conggguan <157357330+conggguan@users.noreply.github.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> * Add changelog Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Martin Gaievski <gaievski@amazon.com> Signed-off-by: Varun Jain <varunudr@amazon.com> Signed-off-by: Tejas Shah <shatejas@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: conggguan <congguan@amazon.com> Signed-off-by: conggguan <157357330+conggguan@users.noreply.github.com> Co-authored-by: Martin Gaievski <gaievski@amazon.com> Co-authored-by: Tejas Shah <shatejas@amazon.com> Co-authored-by: Liyun Xiu <chishui2@gmail.com> Co-authored-by: conggguan <157357330+conggguan@users.noreply.github.com> (cherry picked from commit d22e1b8) Co-authored-by: Varun Jain <varunudr@amazon.com>

shatejas requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, sean-zheng-amazon, model-collapse, zane-neo, ylwu-amzn, jngz-es, vibrantvarun and zhichao-aws as code owners June 11, 2024 21:06

martin-gaievski reviewed Jun 12, 2024

View reviewed changes

shatejas force-pushed the feature/ef-search branch from cda168e to 0d9a6b2 Compare June 13, 2024 00:09

navneet1v reviewed Jun 13, 2024

View reviewed changes

martin-gaievski approved these changes Jun 14, 2024

View reviewed changes

martin-gaievski added the skip-changelog label Jun 14, 2024

shatejas requested a review from navneet1v June 14, 2024 17:24

vibrantvarun reviewed Jun 17, 2024

View reviewed changes

Adds method_parameters in neural search query to support ef_search

0983ba2

Signed-off-by: Tejas Shah <shatejas@amazon.com>

shatejas force-pushed the feature/ef-search branch from 0d9a6b2 to 0983ba2 Compare June 17, 2024 20:55

shatejas requested a review from vibrantvarun June 17, 2024 20:58

vibrantvarun approved these changes Jun 17, 2024

View reviewed changes

vibrantvarun merged commit 726a482 into opensearch-project:feature/ef-search Jun 17, 2024
9 of 72 checks passed

shatejas added a commit to shatejas/neural-search that referenced this pull request Jul 3, 2024

Adds method_parameters in neural search query to support ef_search (o…

e770cb9

…pensearch-project#787) Signed-off-by: Tejas Shah <shatejas@amazon.com>

shatejas added a commit to shatejas/neural-search that referenced this pull request Jul 3, 2024

Adds method_parameters in neural search query to support ef_search (o…

bcb2566

…pensearch-project#787) Signed-off-by: Tejas Shah <shatejas@amazon.com>

shatejas mentioned this pull request Jul 4, 2024

[Enhancement] Adds method_parameters in neural search query to support ef_search #814

Merged

5 tasks

navneet1v pushed a commit that referenced this pull request Jul 4, 2024

Adds method_parameters in neural search query to support ef_search (#787

dea1fe3

) (#814) Signed-off-by: Tejas Shah <shatejas@amazon.com>

opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 4, 2024

Adds method_parameters in neural search query to support ef_search (#787

6e760ea

) (#814) Signed-off-by: Tejas Shah <shatejas@amazon.com> (cherry picked from commit dea1fe3)

vibrantvarun pushed a commit to vibrantvarun/neural-search that referenced this pull request Jul 9, 2024

Adds method_parameters in neural search query to support ef_search (o…

935829a

…pensearch-project#787) (opensearch-project#814) Signed-off-by: Tejas Shah <shatejas@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds method_parameters in neural search query to support ef_search #787

Adds method_parameters in neural search query to support ef_search #787

shatejas commented Jun 11, 2024

martin-gaievski left a comment

shatejas commented Jun 12, 2024

navneet1v Jun 13, 2024

shatejas Jun 13, 2024

navneet1v Jun 13, 2024

shatejas Jun 14, 2024

martin-gaievski left a comment

martin-gaievski commented Jun 14, 2024

shatejas commented Jun 14, 2024

navneet1v commented Jun 17, 2024

vibrantvarun Jun 17, 2024

shatejas Jun 17, 2024

vibrantvarun commented Jun 17, 2024

shatejas commented Jun 17, 2024

Adds method_parameters in neural search query to support ef_search #787

Adds method_parameters in neural search query to support ef_search #787

Conversation

shatejas commented Jun 11, 2024

Description

Issues Resolved

Check List

martin-gaievski left a comment

Choose a reason for hiding this comment

shatejas commented Jun 12, 2024

navneet1v Jun 13, 2024

Choose a reason for hiding this comment

shatejas Jun 13, 2024

Choose a reason for hiding this comment

navneet1v Jun 13, 2024

Choose a reason for hiding this comment

shatejas Jun 14, 2024

Choose a reason for hiding this comment

martin-gaievski left a comment

Choose a reason for hiding this comment

martin-gaievski commented Jun 14, 2024

shatejas commented Jun 14, 2024

navneet1v commented Jun 17, 2024

vibrantvarun Jun 17, 2024

Choose a reason for hiding this comment

shatejas Jun 17, 2024

Choose a reason for hiding this comment

vibrantvarun commented Jun 17, 2024

shatejas commented Jun 17, 2024