Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Percolate Query implementation in doc-level alerting #399

Merged
merged 1 commit into from
Apr 18, 2022

Conversation

sbcd90
Copy link
Collaborator

@sbcd90 sbcd90 commented Apr 13, 2022

percolate query implementation in doc-level alerting

Issue #, if available:
Percolate Query implementation in Document Level Alerting

Description of changes:
This PR provides Percolate Query based implementation in Document Level Alerting. It implements the following design with its limitations.

image

CheckList:
[X] Commits are signed per the DCO using --signoff
[X] Integration tests work for doc-level alerting
[ ] Performance test results - pending
[ ] Additional tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@sbcd90 sbcd90 requested a review from a team April 13, 2022 00:14
@sbcd90 sbcd90 force-pushed the perc_query_proposal branch 2 times, most recently from bcc41a4 to 4e9a856 Compare April 13, 2022 01:38
@codecov-commenter
Copy link

codecov-commenter commented Apr 13, 2022

Codecov Report

❗ No coverage uploaded for pull request base (doc-level-2.0@5f4d9fd). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             doc-level-2.0     #399   +/-   ##
================================================
  Coverage                 ?   78.68%           
  Complexity               ?      285           
================================================
  Files                    ?      198           
  Lines                    ?     8612           
  Branches                 ?     1148           
================================================
  Hits                     ?     6776           
  Misses                   ?     1248           
  Partials                 ?      588           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f4d9fd...4e9a856. Read the comment docs.

@sbcd90 sbcd90 force-pushed the perc_query_proposal branch 10 times, most recently from fb8dee8 to ba89f5e Compare April 13, 2022 22:23
@sbcd90 sbcd90 force-pushed the perc_query_proposal branch 3 times, most recently from 9a3e538 to 826e827 Compare April 14, 2022 17:08
Copy link
Member

@getsaurabh02 getsaurabh02 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes. I have added few comments/clarification.
I am yet to look into the Transport Action part and will review it later today.

@@ -71,7 +72,7 @@ dependencies {
}

javadoc.enabled = false // turn off javadoc as it barfs on Kotlin code
licenseHeaders.enabled = true
licenseHeaders.enabled = false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we add license headers in the new files being added?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added license to all files missing license.

import static org.opensearch.common.xcontent.ConstructingObjectParser.optionalConstructorArg;
import static org.opensearch.search.SearchService.ALLOW_EXPENSIVE_QUERIES;

public class PercolateQueryBuilderExt extends AbstractQueryBuilder<PercolateQueryBuilderExt> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These Ext classes seems to be the Stop Gap solutions unless we have figured out the right way to extend and use the Percolate client capabilities from the alerting plugin.
Can we add a documentation here and ToDo for cleanup?
Lets also add an issue in core repo to address the issue discovered and tag it here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added comments to the java classes. will create an issue in core.

import static java.util.Collections.singletonList;
import static java.util.Collections.singletonMap;

public class PercolatorPluginExt extends Plugin implements MapperPlugin, SearchPlugin, ExtensiblePlugin {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were we able to create the full distribution and verify the same in single and multinode setup with no issues?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. the integration test build passes for single node & multi node with 3 nodes.


import static org.opensearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder;

public class PercolatorFieldMapperExt extends ParametrizedFieldMapper {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will a change in the PercolatorFieldMapper in core repo create the type cast exception issue with objects if this class is not kept up to date? This could be a huge maintenance overhead if not addressed shortly.
Also wondering if we also evaluated the object deserialization and serialization route for improved handling of such issues?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not PercolatorFieldMapper. But if api signatures from other classes in percolator-client changes, it will need change in the copied classes too.
object deserialization and serialization route needs to handled in core. spi approach to make percolate plugin extensible could also be an idea that could be pursued.

val docsToQueries = mutableMapOf<String, MutableList<String>>()
val docExecutionContext = DocumentExecutionContext(queries, lastRunContext, updatedLastRunContext)
val idQueryMap = mutableMapOf<String, DocLevelQuery>()
queries.forEach { query ->
val matchingDocIds = runForEachQuery(monitorCtx, docExecutionContext, query, index)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets cleanup runForEachQuery impl.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed it.

Comment on lines 414 to 415
logger.info("Failed to run for shard $shard. Error: ${e.message}")
logger.debug("Failed to run for shard $shard", e)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure with the intent of double logging here? To get the exception trace?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed it.

Comment on lines 518 to 483
private fun getAllDocIds(hits: SearchHits): List<String> {
return hits.map { hit -> hit.id }
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replaced by getAllDocs now?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed it.

): SearchHits {
val percolateQueryBuilder = PercolateQueryBuilderExt("query", docs, XContentType.JSON)

val searchRequest = SearchRequest(ScheduledJob.DOC_LEVEL_QUERIES_INDEX)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a safe check if the query index is already created/exists? It could be in Runner itself. Just ensuring the exception is floated up the Runner with the right messaging

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed the safe check.

Copy link
Member

@getsaurabh02 getsaurabh02 Apr 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if DOC_LEVEL_QUERIES_INDEX is not created or is deleted for some reason? I dont think we have solved this use-case yet. Not a blocker, but something for a quick follow up.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. the current logic does not take care of high availability. This can be improved in future.
Currently, if index doesnt exist, there will be no exceptions. only no search results will be returned.

@@ -410,7 +491,47 @@ object DocumentReturningMonitorRunner : MonitorRunner {
return response.hits
}

private fun getMatchedQueries(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this not to get Matching Documents instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this method matches incoming docs with queries. named another method to getMatchingDocs

searchRequest.source(searchSourceBuilder)

return if (monitorCtx.clusterService!!.state().routingTable.hasIndex(ScheduledJob.DOC_LEVEL_QUERIES_INDEX)) {
val response: SearchResponse = monitorCtx.client!!.execute(SearchAction.INSTANCE, searchRequest).actionGet()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical - This will end up running percolate for all the incoming documents against all the queries without the filtering the monitor type against the query index. Should we have the monitor.id field added upfront?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

): SearchHits {
val percolateQueryBuilder = PercolateQueryBuilderExt("query", docs, XContentType.JSON)

val searchRequest = SearchRequest(ScheduledJob.DOC_LEVEL_QUERIES_INDEX)
Copy link
Member

@getsaurabh02 getsaurabh02 Apr 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if DOC_LEVEL_QUERIES_INDEX is not created or is deleted for some reason? I dont think we have solved this use-case yet. Not a blocker, but something for a quick follow up.

@@ -409,6 +424,83 @@ class TransportIndexMonitorAction @Inject constructor(
)
}

@Suppress("UNCHECKED_CAST")
private fun indexDocLevelMonitorQueries(monitor: Monitor, monitorId: String, refreshPolicy: RefreshPolicy) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be code duplication from the monitor execution action for mapping updates. I would rather define this as a util method in one of doc level monitor related class and use at both the places if required.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -74,6 +91,65 @@ class TransportExecuteMonitorAction @Inject constructor(
}
}
}
val indexDocLevelMonitorQueries = fun(monitor: Monitor, monitorId: String) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to update the mapping each time during the monitor run execution? Shouldn't the Index Monitor path take care of any update in the mapping?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are 2 types of execute action. one which takes already created monitor_id as input. the other one directly takes entire monitor payload as input.
update mapping is needed in 2nd case.
https://github.com/opensearch-project/alerting/blob/79a368caa73b64ab08b0cd14c4da0324ded8c1db/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteMonitorAction.kt#L89-#L180

@sbcd90 sbcd90 force-pushed the perc_query_proposal branch 2 times, most recently from 79a368c to a55fe04 Compare April 15, 2022 21:31
@@ -0,0 +1,13 @@
{
"_meta": {
"schema_version": 5
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change this to 1

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
@sbcd90 sbcd90 merged commit 8e991f6 into opensearch-project:doc-level-2.0 Apr 18, 2022
lezzago added a commit that referenced this pull request Apr 18, 2022
* Rebase to push doc level changes on latest main changes (#391)

* Document level alerting dev (#272)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Add last run context to Monitor data model

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* add Update Monitor function

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* fix integ test

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Implemented draft of Finding data model, a new Input type, and some basic unit tests. (#260)

* Implemented draft of Finding data model, and some basic unit tests for it.

Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>

* POC for doc-level-alerting (#277)

Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>

* Add connection to triggers for doc level alerting (#316)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* CRUD APIs integration Tests and validation"conflict resolved" (#362)

Signed-off-by: charliezhangaws <zhanncha@amazon.com>

* Segregate monitor runner logic for separation of concerns (#363)

* Refactor monitor runner logic for separation of concerns and better testability.

Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com>

* Add action and alert flow and findings schema and additional fixes (#381)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Finding Search API (#385)

* Findings search API based on Annie's work

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Fix Search API and add IT tests

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <leeyun@amazon.com>

* Fix integ tests and minor issues from doc level changes

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>

* Add Trigger condition resolver which parses and evaluates the Trigger expression. (#405)

Signed-off-by: Saurabh Singh <sisurab@amazon.com>

* percolate query implementation in doc-level alerting (#399)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Finding Index rollover (#408)

* Finding Index rollover

Signed-off-by: jiahe zhang <zhanncha@amazon.com>

* Apply fixes to make rollover work

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: jiahe zhang <zhanncha@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
AWSHurneyt added a commit to AWSHurneyt/OpenSearch-Alerting that referenced this pull request Apr 21, 2022
* Rebase to push doc level changes on latest main changes (opensearch-project#391)

* Document level alerting dev (opensearch-project#272)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Add last run context to Monitor data model

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* add Update Monitor function

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* fix integ test

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260)

* Implemented draft of Finding data model, and some basic unit tests for it.

Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>

* POC for doc-level-alerting (opensearch-project#277)

Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>

* Add connection to triggers for doc level alerting (opensearch-project#316)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362)

Signed-off-by: charliezhangaws <zhanncha@amazon.com>

* Segregate monitor runner logic for separation of concerns (opensearch-project#363)

* Refactor monitor runner logic for separation of concerns and better testability.

Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com>

* Add action and alert flow and findings schema and additional fixes (opensearch-project#381)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Finding Search API (opensearch-project#385)

* Findings search API based on Annie's work

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Fix Search API and add IT tests

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <leeyun@amazon.com>

* Fix integ tests and minor issues from doc level changes

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>

* Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405)

Signed-off-by: Saurabh Singh <sisurab@amazon.com>

* percolate query implementation in doc-level alerting (opensearch-project#399)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Finding Index rollover (opensearch-project#408)

* Finding Index rollover

Signed-off-by: jiahe zhang <zhanncha@amazon.com>

* Apply fixes to make rollover work

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: jiahe zhang <zhanncha@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>
AWSHurneyt added a commit that referenced this pull request Apr 21, 2022
* fix security test workflow (#407)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Integrate Document Level Alerting changes (#410)

* Rebase to push doc level changes on latest main changes (#391)

* Document level alerting dev (#272)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Add last run context to Monitor data model

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* add Update Monitor function

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* fix integ test

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Implemented draft of Finding data model, a new Input type, and some basic unit tests. (#260)

* Implemented draft of Finding data model, and some basic unit tests for it.

Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>

* POC for doc-level-alerting (#277)

Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>

* Add connection to triggers for doc level alerting (#316)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* CRUD APIs integration Tests and validation"conflict resolved" (#362)

Signed-off-by: charliezhangaws <zhanncha@amazon.com>

* Segregate monitor runner logic for separation of concerns (#363)

* Refactor monitor runner logic for separation of concerns and better testability.

Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com>

* Add action and alert flow and findings schema and additional fixes (#381)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Finding Search API (#385)

* Findings search API based on Annie's work

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Fix Search API and add IT tests

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <leeyun@amazon.com>

* Fix integ tests and minor issues from doc level changes

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>

* Add Trigger condition resolver which parses and evaluates the Trigger expression. (#405)

Signed-off-by: Saurabh Singh <sisurab@amazon.com>

* percolate query implementation in doc-level alerting (#399)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Finding Index rollover (#408)

* Finding Index rollover

Signed-off-by: jiahe zhang <zhanncha@amazon.com>

* Apply fixes to make rollover work

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: jiahe zhang <zhanncha@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Remove write Destination APIs (#412)

* Remove write Destination API REST handlers

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove write Destination transport actions

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove write Destination action, request and response classes

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Create Alerting config index if it doesn't exist before legacy Destination indexing

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove write Destination related security tests

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove unused access roles and imports

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Deprecate the Master nomenclature in 2.0 (#415)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Remove Alerting's notification subproject (#413)

* Remove notification subproject

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove publishing to maven in build script

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Skipping destination migration if alerting index is not initialized (#417)

Signed-off-by: Ravi 6005951+thalurur@users.noreply.github.com
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Alias support for Document Level Monitors (#416)

* Implemented support for defining doc level monitors using aliases.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Fix integ tests and cleaup alias logic

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: AWSHurneyt <hurneyt@amazon.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Fixed a flaky test condition. (#375)

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Replace checked-in ZIP for bwc tests with a dynamic dependency (#411)

* Replace checked-in ZIP with a dynamic dependency

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Replace checked-in ZIP with a dynamic dependency

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Update integTest gradle scripts to run via remote cluster independently (#418)

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Removed tests that were duplicated while resolving merge conflicts.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Co-authored-by: Ashish Agrawal <ashisagr@amazon.com>
Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>
Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com>
Angie-Zhang pushed a commit to Angie-Zhang/alerting that referenced this pull request Jun 29, 2022
* fix security test workflow (opensearch-project#407)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Integrate Document Level Alerting changes (opensearch-project#410)

* Rebase to push doc level changes on latest main changes (opensearch-project#391)

* Document level alerting dev (opensearch-project#272)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Add last run context to Monitor data model

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* add Update Monitor function

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* fix integ test

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260)

* Implemented draft of Finding data model, and some basic unit tests for it.

Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>

* POC for doc-level-alerting (opensearch-project#277)

Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>

* Add connection to triggers for doc level alerting (opensearch-project#316)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362)

Signed-off-by: charliezhangaws <zhanncha@amazon.com>

* Segregate monitor runner logic for separation of concerns (opensearch-project#363)

* Refactor monitor runner logic for separation of concerns and better testability.

Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com>

* Add action and alert flow and findings schema and additional fixes (opensearch-project#381)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Finding Search API (opensearch-project#385)

* Findings search API based on Annie's work

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Fix Search API and add IT tests

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <leeyun@amazon.com>

* Fix integ tests and minor issues from doc level changes

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>

* Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405)

Signed-off-by: Saurabh Singh <sisurab@amazon.com>

* percolate query implementation in doc-level alerting (opensearch-project#399)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Finding Index rollover (opensearch-project#408)

* Finding Index rollover

Signed-off-by: jiahe zhang <zhanncha@amazon.com>

* Apply fixes to make rollover work

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: jiahe zhang <zhanncha@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Remove write Destination APIs (opensearch-project#412)

* Remove write Destination API REST handlers

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove write Destination transport actions

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove write Destination action, request and response classes

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Create Alerting config index if it doesn't exist before legacy Destination indexing

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove write Destination related security tests

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove unused access roles and imports

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Deprecate the Master nomenclature in 2.0 (opensearch-project#415)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Remove Alerting's notification subproject (opensearch-project#413)

* Remove notification subproject

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove publishing to maven in build script

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Skipping destination migration if alerting index is not initialized (opensearch-project#417)

Signed-off-by: Ravi 6005951+thalurur@users.noreply.github.com
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Alias support for Document Level Monitors (opensearch-project#416)

* Implemented support for defining doc level monitors using aliases.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Fix integ tests and cleaup alias logic

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: AWSHurneyt <hurneyt@amazon.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Fixed a flaky test condition. (opensearch-project#375)

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Replace checked-in ZIP for bwc tests with a dynamic dependency (opensearch-project#411)

* Replace checked-in ZIP with a dynamic dependency

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Replace checked-in ZIP with a dynamic dependency

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Update integTest gradle scripts to run via remote cluster independently (opensearch-project#418)

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Removed tests that were duplicated while resolving merge conflicts.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Co-authored-by: Ashish Agrawal <ashisagr@amazon.com>
Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>
Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com>
Signed-off-by: Angie Zhang <langelzh@amazon.com>
Angie-Zhang pushed a commit to Angie-Zhang/alerting that referenced this pull request Jun 29, 2022
* Rebase to push doc level changes on latest main changes (opensearch-project#391)

* Document level alerting dev (opensearch-project#272)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Add last run context to Monitor data model

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* add Update Monitor function

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* fix integ test

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260)

* Implemented draft of Finding data model, and some basic unit tests for it.

Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>

* POC for doc-level-alerting (opensearch-project#277)

Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>

* Add connection to triggers for doc level alerting (opensearch-project#316)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362)

Signed-off-by: charliezhangaws <zhanncha@amazon.com>

* Segregate monitor runner logic for separation of concerns (opensearch-project#363)

* Refactor monitor runner logic for separation of concerns and better testability.

Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com>

* Add action and alert flow and findings schema and additional fixes (opensearch-project#381)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Finding Search API (opensearch-project#385)

* Findings search API based on Annie's work

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Fix Search API and add IT tests

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <leeyun@amazon.com>

* Fix integ tests and minor issues from doc level changes

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>

* Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405)

Signed-off-by: Saurabh Singh <sisurab@amazon.com>

* percolate query implementation in doc-level alerting (opensearch-project#399)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Finding Index rollover (opensearch-project#408)

* Finding Index rollover

Signed-off-by: jiahe zhang <zhanncha@amazon.com>

* Apply fixes to make rollover work

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: jiahe zhang <zhanncha@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: Angie Zhang <langelzh@amazon.com>
Angie-Zhang pushed a commit to Angie-Zhang/alerting that referenced this pull request Jun 29, 2022
* fix security test workflow (opensearch-project#407)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Integrate Document Level Alerting changes (opensearch-project#410)

* Rebase to push doc level changes on latest main changes (opensearch-project#391)

* Document level alerting dev (opensearch-project#272)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Add last run context to Monitor data model

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* add Update Monitor function

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* fix integ test

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260)

* Implemented draft of Finding data model, and some basic unit tests for it.

Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>

* POC for doc-level-alerting (opensearch-project#277)

Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>

* Add connection to triggers for doc level alerting (opensearch-project#316)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362)

Signed-off-by: charliezhangaws <zhanncha@amazon.com>

* Segregate monitor runner logic for separation of concerns (opensearch-project#363)

* Refactor monitor runner logic for separation of concerns and better testability.

Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com>

* Add action and alert flow and findings schema and additional fixes (opensearch-project#381)

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Finding Search API (opensearch-project#385)

* Findings search API based on Annie's work

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Fix Search API and add IT tests

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <leeyun@amazon.com>

* Fix integ tests and minor issues from doc level changes

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>

* Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405)

Signed-off-by: Saurabh Singh <sisurab@amazon.com>

* percolate query implementation in doc-level alerting (opensearch-project#399)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Finding Index rollover (opensearch-project#408)

* Finding Index rollover

Signed-off-by: jiahe zhang <zhanncha@amazon.com>

* Apply fixes to make rollover work

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: jiahe zhang <zhanncha@amazon.com>

Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Remove write Destination APIs (opensearch-project#412)

* Remove write Destination API REST handlers

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove write Destination transport actions

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove write Destination action, request and response classes

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Create Alerting config index if it doesn't exist before legacy Destination indexing

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove write Destination related security tests

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove unused access roles and imports

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Deprecate the Master nomenclature in 2.0 (opensearch-project#415)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Remove Alerting's notification subproject (opensearch-project#413)

* Remove notification subproject

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>

* Remove publishing to maven in build script

Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Skipping destination migration if alerting index is not initialized (opensearch-project#417)

Signed-off-by: Ravi 6005951+thalurur@users.noreply.github.com
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Alias support for Document Level Monitors (opensearch-project#416)

* Implemented support for defining doc level monitors using aliases.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Fix integ tests and cleaup alias logic

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

Co-authored-by: AWSHurneyt <hurneyt@amazon.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Fixed a flaky test condition. (opensearch-project#375)

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Replace checked-in ZIP for bwc tests with a dynamic dependency (opensearch-project#411)

* Replace checked-in ZIP with a dynamic dependency

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Replace checked-in ZIP with a dynamic dependency

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Update integTest gradle scripts to run via remote cluster independently (opensearch-project#418)

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Removed tests that were duplicated while resolving merge conflicts.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Co-authored-by: Ashish Agrawal <ashisagr@amazon.com>
Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com>
Co-authored-by: charliezhangaws <zhanncha@amazon.com>
Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com>
Co-authored-by: Annie Lee <leeyun@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com>
Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com>
Signed-off-by: Angie Zhang <langelzh@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants