adds storage & query layer to events correlation engine #7350

sbcd90 · 2023-05-01T23:06:49Z

Description

adds hnsw graph based storage & query layer to events correlation engine

Related Issues

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-05-01T23:15:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14710/
CommitID: 7fb1ce2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-01T23:57:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/14711/
CommitID: 1784633

codecov-commenter · 2023-05-02T00:02:04Z

Codecov Report

Merging #7350 (e0f3388) into main (5dce570) will decrease coverage by 0.04%.
The diff coverage is 67.84%.

@@             Coverage Diff              @@
##               main    #7350      +/-   ##
============================================
- Coverage     70.93%   70.89%   -0.04%     
- Complexity    56633    56654      +21     
============================================
  Files          4719     4730      +11     
  Lines        267555   268006     +451     
  Branches      39206    39259      +53     
============================================
+ Hits         189778   189991     +213     
- Misses        61721    61983     +262     
+ Partials      16056    16032      -24

Impacted Files	Coverage Δ
...ch/plugin/correlation/EventsCorrelationPlugin.java	`16.66% <0.00%> (+16.66%)`	⬆️
...tion/core/index/codec/CorrelationCodecService.java	`0.00% <0.00%> (ø)`
...rrelation/core/index/mapper/VectorFieldMapper.java	`44.06% <44.06%> (ø)`
...ex/codec/BasePerFieldCorrelationVectorsFormat.java	`60.00% <60.00%> (ø)`
...tion/core/index/query/CorrelationQueryBuilder.java	`70.99% <70.99%> (ø)`
...tion/core/index/codec/CorrelationCodecVersion.java	`77.77% <77.77%> (ø)`
...rch/plugin/correlation/core/index/VectorField.java	`80.00% <80.00%> (ø)`
...rrelation/core/index/CorrelationParamsContext.java	`87.23% <87.23%> (ø)`
...tion/core/index/query/CorrelationQueryFactory.java	`88.88% <88.88%> (ø)`
...ore/index/mapper/CorrelationVectorFieldMapper.java	`89.28% <89.28%> (ø)`
... and 2 more

... and 466 files with indirect coverage changes

plugins/events-correlation-engine/build.gradle

...ine/src/main/java/org/opensearch/plugin/correlation/core/index/CorrelationParamsContext.java

...ava/org/opensearch/plugin/correlation/core/index/serializer/CorrelationVectorSerializer.java

github-actions · 2023-05-17T00:49:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15534/
CommitID: dcfd057
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-24T18:12:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/16229/
CommitID: 3d56f53
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-24T18:44:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/16230/
CommitID: 8f85470

sbcd90 · 2023-05-24T18:54:39Z

hi @andrross , rebased the pr. all the tests pass now.

getsaurabh02

Thanks @sbcd90 LGTM. Added few comments which seems non-blocking and can be addressed as follow-up.

...java/org/opensearch/plugin/correlation/core/index/codec/correlation950/CorrelationCodec.java

...ine/src/main/java/org/opensearch/plugin/correlation/core/index/mapper/VectorFieldMapper.java

andrross

Mostly looks good to me.

56.53% of diff hit (target 70.63%)

The code coverage target is a bit arbitrary but can we get better coverage here from the unit tests?

...org/opensearch/plugin/correlation/core/index/codec/BasePerFieldCorrelationVectorsFormat.java

andrross · 2023-05-31T21:17:59Z

@sbcd90 Just a friendly ping on my comments from last week.

andrross · 2023-05-31T21:20:35Z

@sbcd90 Can you link the meta issue in the description as well and be sure to keep that issue up to date with your progress?

sbcd90 · 2023-05-31T23:46:44Z

hi @andrross , thanks a lot for the reminders. already looking into your comments.

github-actions · 2023-06-09T00:10:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

URL: https://build.ci.opensearch.org/job/gradle-check/17179/
CommitID: c63caba
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

sbcd90 · 2023-06-09T00:38:31Z

hi @andrross , improved the code coverage for patch to 67.66%. created a separate issue to improve the code coverage. #8120
addressed all review comments as well.

...rrelation-engine/src/main/java/org/opensearch/plugin/correlation/core/index/VectorField.java

plugins/events-correlation-engine/build.gradle

...ava/org/opensearch/plugin/correlation/core/index/serializer/CorrelationVectorSerializer.java

github-actions · 2023-06-20T01:20:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testReplicationPostDeleteAndForceMerge

URL: https://build.ci.opensearch.org/job/gradle-check/17975/
CommitID: cc53e8b
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

sbcd90 · 2023-06-20T01:34:22Z

hi @andrross , thanks a lot for reviewing the pr again. addressed all the comments. please have a look.

...rrelation-engine/src/main/java/org/opensearch/plugin/correlation/core/index/VectorField.java

...ine/src/main/java/org/opensearch/plugin/correlation/core/index/CorrelationParamsContext.java

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

github-actions · 2023-06-20T20:14:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18033/
CommitID: e0f3388
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sbcd90 · 2023-06-20T20:19:30Z

hi @andrross , fixed it. #7350 (comment)
Addressed all the comments. please have a look.

github-actions · 2023-06-22T19:19:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

URL: https://build.ci.opensearch.org/job/gradle-check/18213/
CommitID: e0f3388
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…ne (opensearch-project#7350) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

…ne (opensearch-project#7350) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

…ne (opensearch-project#7350) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

sbcd90 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners May 1, 2023 23:06

sbcd90 force-pushed the test branch from 7fb1ce2 to 1784633 Compare May 1, 2023 23:24

sbcd90 changed the title ~~adds hnsw graph based storage & query layer to events correlation engine~~ adds storage & query layer to events correlation engine May 1, 2023

andrross reviewed May 16, 2023

View reviewed changes

sbcd90 force-pushed the test branch from 1784633 to dcfd057 Compare May 17, 2023 00:27

sbcd90 force-pushed the test branch from dcfd057 to 156d3c1 Compare May 17, 2023 00:49

getsaurabh02 approved these changes May 25, 2023

View reviewed changes

...java/org/opensearch/plugin/correlation/core/index/codec/correlation950/CorrelationCodec.java Show resolved Hide resolved

...ine/src/main/java/org/opensearch/plugin/correlation/core/index/mapper/VectorFieldMapper.java Show resolved Hide resolved

andrross reviewed May 25, 2023

View reviewed changes

...org/opensearch/plugin/correlation/core/index/codec/BasePerFieldCorrelationVectorsFormat.java Show resolved Hide resolved

sbcd90 force-pushed the test branch from 8f85470 to c63caba Compare June 8, 2023 23:38

sbcd90 mentioned this pull request Jun 16, 2023

[Plugins Events Correlation Engine] Enhance code coverage #8120

Open

andrross reviewed Jun 19, 2023

View reviewed changes

sbcd90 force-pushed the test branch from c63caba to cc53e8b Compare June 20, 2023 00:49

sbcd90 requested a review from sachinpkale as a code owner June 20, 2023 00:49

andrross reviewed Jun 20, 2023

View reviewed changes

adds hnsw graph based storage & query layer to events correlation engine

e0f3388

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

sbcd90 force-pushed the test branch from cc53e8b to e0f3388 Compare June 20, 2023 19:41

andrross added the skip-changelog label Jun 22, 2023

andrross approved these changes Jun 22, 2023

View reviewed changes

andrross merged commit 84dffac into opensearch-project:main Jun 22, 2023

jmazanec15 mentioned this pull request Jun 23, 2023

[RFC] OpenSearch Events Correlation Engine #6779

Open

sudarshan-baliga pushed a commit to Gaurav614/OpenSearch that referenced this pull request Jun 29, 2023

Add hnsw graph based storage & query layer to events correlation engi…

12bdebf

…ne (opensearch-project#7350) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adds storage & query layer to events correlation engine #7350

adds storage & query layer to events correlation engine #7350

sbcd90 commented May 1, 2023 •

edited

Loading

github-actions bot commented May 1, 2023

github-actions bot commented May 1, 2023

codecov-commenter commented May 2, 2023 •

edited by codecov bot

Loading

github-actions bot commented May 17, 2023

github-actions bot commented May 24, 2023

github-actions bot commented May 24, 2023

sbcd90 commented May 24, 2023

getsaurabh02 left a comment

andrross left a comment

andrross commented May 31, 2023

andrross commented May 31, 2023

sbcd90 commented May 31, 2023 •

edited

Loading

github-actions bot commented Jun 9, 2023

sbcd90 commented Jun 9, 2023 •

edited

Loading

github-actions bot commented Jun 20, 2023

sbcd90 commented Jun 20, 2023

github-actions bot commented Jun 20, 2023

sbcd90 commented Jun 20, 2023

github-actions bot commented Jun 22, 2023

adds storage & query layer to events correlation engine #7350

adds storage & query layer to events correlation engine #7350

Conversation

sbcd90 commented May 1, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented May 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 1, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented May 2, 2023 • edited by codecov bot Loading

Codecov Report

github-actions bot commented May 17, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 24, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 24, 2023

Gradle Check (Jenkins) Run Completed with:

sbcd90 commented May 24, 2023

getsaurabh02 left a comment

Choose a reason for hiding this comment

andrross left a comment

Choose a reason for hiding this comment

andrross commented May 31, 2023

andrross commented May 31, 2023

sbcd90 commented May 31, 2023 • edited Loading

github-actions bot commented Jun 9, 2023

Gradle Check (Jenkins) Run Completed with:

sbcd90 commented Jun 9, 2023 • edited Loading

github-actions bot commented Jun 20, 2023

Gradle Check (Jenkins) Run Completed with:

sbcd90 commented Jun 20, 2023

github-actions bot commented Jun 20, 2023

Gradle Check (Jenkins) Run Completed with:

sbcd90 commented Jun 20, 2023

github-actions bot commented Jun 22, 2023

Gradle Check (Jenkins) Run Completed with:

sbcd90 commented May 1, 2023 •

edited

Loading

codecov-commenter commented May 2, 2023 •

edited by codecov bot

Loading

sbcd90 commented May 31, 2023 •

edited

Loading

sbcd90 commented Jun 9, 2023 •

edited

Loading