Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] [Enhancement] Add schema validation and placeholders to index mappings #3310

Open
wants to merge 1 commit into
base: 2.x
Choose a base branch
from

Conversation

opensearch-trigger-bot[bot]
Copy link
Contributor

Backport 374cfd5 from #3240

#3240)

* feat(index mappings): fetch mappings and version from json file instead of string constants

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: changing exception being thrown

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* chore: remove unused file

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* chore: fix typo in comment

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* chore: adding new line at the end of files

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* feat: add test cases

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* fix: remove test code

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* fix(test): in main the versions were not updated appropriately

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: move mapping templates under common module

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: ensure that conversationindexconstants reference mlindex enums rather than use their own mappings

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: update comment

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* feat: add enhancements to validate index schema and allow using placeholders

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: modifying comment

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* test: adding testcase for MLIndex to catch failures before runtime

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: rename dir from mappings to index-mappings

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* fix: add null checks

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* fix: modify mappin paths for placeholders

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* fix: adding dependencies for testing

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* fix(test): compare json object rather than strings to avoid eol character issue

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: combine if statements into single check

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactoring: null handling + clean code

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* spotless apply

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* tests: adding more UT

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* fix: dependencies to handle jarhell

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* spotless apply

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: add header and use single instance of mapper

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* fixed: doc syntax

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: renamed files, efficient loading of resources, better exception handling

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

* refactor: cleaner comment

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>

---------

Signed-off-by: Pavan Yekbote <mail2pavanyekbote@gmail.com>
(cherry picked from commit 374cfd5)
@mingshl
Copy link
Collaborator

mingshl commented Dec 31, 2024

rerunning BWC 11

Run plugin_version=`./gradlew properties -q | grep "opensearch_build:" | awk '{print $2}'`

FAILURE: Build failed with an exception.

* Where:
Build file '/home/runner/work/ml-commons/ml-commons/ml-algorithms/build.gradle' line: 78

* What went wrong:
A problem occurred evaluating project ':opensearch-ml-algorithms'.
> Could not get unknown property 'versions' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in [21](https://github.com/opensearch-project/ml-commons/actions/runs/12553722221/job/35001500146?pr=3310#step:4:22)s
plugin_version

Welcome to Gradle 8.11.1!

Here are the highlights of this release:
 - Parallel load and store for Configuration Cache
 - Java compilation errors at the end of the build output
 - Consolidated report for warnings and deprecations

For more details see https://docs.gradle.org/8.11.1/release-notes.html


[Incubating] Problems report is available at: file:///home/runner/work/ml-commons/ml-commons/build/reports/problems/problems-report.html
FAILURE: Build failed with an exception.

* Where:
Build file '/home/runner/work/ml-commons/ml-commons/ml-algorithms/build.gradle' line: 78

* What went wrong:
A problem occurred evaluating project ':opensearch-ml-algorithms'.
> Could not get unknown property 'versions' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org./


Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.11.1/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 1s
Error: Process completed with exit code 1.

@dhrubo-os
Copy link
Collaborator

rerunning BWC 11

Run plugin_version=`./gradlew properties -q | grep "opensearch_build:" | awk '{print $2}'`

FAILURE: Build failed with an exception.

* Where:
Build file '/home/runner/work/ml-commons/ml-commons/ml-algorithms/build.gradle' line: 78

* What went wrong:
A problem occurred evaluating project ':opensearch-ml-algorithms'.
> Could not get unknown property 'versions' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in [21](https://github.com/opensearch-project/ml-commons/actions/runs/12553722221/job/35001500146?pr=3310#step:4:22)s
plugin_version

Welcome to Gradle 8.11.1!

Here are the highlights of this release:
 - Parallel load and store for Configuration Cache
 - Java compilation errors at the end of the build output
 - Consolidated report for warnings and deprecations

For more details see https://docs.gradle.org/8.11.1/release-notes.html


[Incubating] Problems report is available at: file:///home/runner/work/ml-commons/ml-commons/build/reports/problems/problems-report.html
FAILURE: Build failed with an exception.

* Where:
Build file '/home/runner/work/ml-commons/ml-commons/ml-algorithms/build.gradle' line: 78

* What went wrong:
A problem occurred evaluating project ':opensearch-ml-algorithms'.
> Could not get unknown property 'versions' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org./


Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.11.1/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 1s
Error: Process completed with exit code 1.

@pyek-bot you need to add some code from this PR : https://github.com/opensearch-project/ml-commons/pull/2800/files

In 2.x branch we don't have versions variable available.

@pyek-bot
Copy link
Contributor

pyek-bot commented Jan 3, 2025

@dhrubo-os should we add backport label to this pr instead of making changes in mine

Edit: nvm, the pr mentions why it was not backported to 2.x, i will manually introduced versions variable

@pyek-bot
Copy link
Contributor

pyek-bot commented Jan 7, 2025

Raised a manual backport PR with the required changes: #3342

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants