[Backport 2.11] Backport #669 to 2.11 #690

* add mapping for indices storing threat intel feed data Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix feed indices mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threat intel feed data dao Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threatIntelEnabled field in detector. Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threat intel feed service and searching feeds Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * ti feed data to doc level query convertor logic added Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * plug threat intel feed into detector creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Preliminary framework for jobscheduler and datasource (opensearch-project#626) Signed-off-by: Joanne Wang <jowg@amazon.com> * create doc level query from threat intel feed data index docs" Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * handle threat intel enabled check during detector updation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add tests for testing threat intel feed integration with detectors Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Threat intel feeds job runner and unit tests (opensearch-project#654) * fix doc level query constructor (opensearch-project#651) Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add mapping for indices storing threat intel feed data * fix feed indices mapping * add threat intel feed data dao Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threatIntelEnabled field in detector. Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threat intel feed service and searching feeds Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * ti feed data to doc level query convertor logic added * plug threat intel feed into detector creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Preliminary framework for jobscheduler and datasource (opensearch-project#626) Signed-off-by: Joanne Wang <jowg@amazon.com> * with listener and processor Signed-off-by: Joanne Wang <jowg@amazon.com> * removed actions Signed-off-by: Joanne Wang <jowg@amazon.com> * clean up Signed-off-by: Joanne Wang <jowg@amazon.com> * added parser Signed-off-by: Joanne Wang <jowg@amazon.com> * add unit tests Signed-off-by: Joanne Wang <jowg@amazon.com> * refactored class names Signed-off-by: Joanne Wang <jowg@amazon.com> * before moving db Signed-off-by: Joanne Wang <jowg@amazon.com> * after moving db Signed-off-by: Joanne Wang <jowg@amazon.com> * added actions to plugin and removed user schedule Signed-off-by: Joanne Wang <jowg@amazon.com> * unit tests Signed-off-by: Joanne Wang <jowg@amazon.com> * fix build error Signed-off-by: Joanne Wang <jowg@amazon.com> * changed transport naming Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> * converge job scheduler code with threat intel feed integration in detectors Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * converge job scheduler and detector threat intel code Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add feed metadata config files in src and test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * adds ioc fields list in log type config files and ioc fields object in LogType POJO Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix compilation issues in tests Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * test udpate detector disabling threat intel Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add tests for detector creation and updation with threat intel Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Threat intel test (opensearch-project#673) * add mapping for indices storing threat intel feed data * fix feed indices mapping * add threat intel feed data dao Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threatIntelEnabled field in detector. Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threat intel feed service and searching feeds Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * ti feed data to doc level query convertor logic added * plug threat intel feed into detector creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Preliminary framework for jobscheduler and datasource (opensearch-project#626) Signed-off-by: Joanne Wang <jowg@amazon.com> * create doc level query from threat intel feed data index docs" Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * handle threat intel enabled check during detector updation * add tests for testing threat intel feed integration with detectors Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Threat intel feeds job runner and unit tests (opensearch-project#654) * fix doc level query constructor (opensearch-project#651) Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add mapping for indices storing threat intel feed data * fix feed indices mapping * add threat intel feed data dao Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threatIntelEnabled field in detector. Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threat intel feed service and searching feeds Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * ti feed data to doc level query convertor logic added * plug threat intel feed into detector creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Preliminary framework for jobscheduler and datasource (opensearch-project#626) Signed-off-by: Joanne Wang <jowg@amazon.com> * with listener and processor Signed-off-by: Joanne Wang <jowg@amazon.com> * removed actions Signed-off-by: Joanne Wang <jowg@amazon.com> * clean up Signed-off-by: Joanne Wang <jowg@amazon.com> * added parser Signed-off-by: Joanne Wang <jowg@amazon.com> * add unit tests Signed-off-by: Joanne Wang <jowg@amazon.com> * refactored class names Signed-off-by: Joanne Wang <jowg@amazon.com> * before moving db Signed-off-by: Joanne Wang <jowg@amazon.com> * after moving db Signed-off-by: Joanne Wang <jowg@amazon.com> * added actions to plugin and removed user schedule Signed-off-by: Joanne Wang <jowg@amazon.com> * unit tests Signed-off-by: Joanne Wang <jowg@amazon.com> * fix build error Signed-off-by: Joanne Wang <jowg@amazon.com> * changed transport naming Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> * converge job scheduler code with threat intel feed integration in detectors Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * refactored out unecessary Signed-off-by: Joanne Wang <jowg@amazon.com> * added headers and cleaned up Signed-off-by: Joanne Wang <jowg@amazon.com> * converge job scheduler and detector threat intel code Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * working on testing Signed-off-by: Joanne Wang <jowg@amazon.com> * fixed the parser and build.gradle Signed-off-by: Joanne Wang <jowg@amazon.com> * add mapping for indices storing threat intel feed data * fix feed indices mapping * add threat intel feed data dao Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threatIntelEnabled field in detector. Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threat intel feed service and searching feeds Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * ti feed data to doc level query convertor logic added * plug threat intel feed into detector creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Preliminary framework for jobscheduler and datasource (opensearch-project#626) Signed-off-by: Joanne Wang <jowg@amazon.com> * create doc level query from threat intel feed data index docs" Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * handle threat intel enabled check during detector updation * add tests for testing threat intel feed integration with detectors Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Threat intel feeds job runner and unit tests (opensearch-project#654) * fix doc level query constructor (opensearch-project#651) Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add mapping for indices storing threat intel feed data * fix feed indices mapping * add threat intel feed data dao Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threatIntelEnabled field in detector. Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threat intel feed service and searching feeds Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * ti feed data to doc level query convertor logic added * plug threat intel feed into detector creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Preliminary framework for jobscheduler and datasource (opensearch-project#626) Signed-off-by: Joanne Wang <jowg@amazon.com> * with listener and processor Signed-off-by: Joanne Wang <jowg@amazon.com> * removed actions Signed-off-by: Joanne Wang <jowg@amazon.com> * clean up Signed-off-by: Joanne Wang <jowg@amazon.com> * added parser Signed-off-by: Joanne Wang <jowg@amazon.com> * add unit tests Signed-off-by: Joanne Wang <jowg@amazon.com> * refactored class names Signed-off-by: Joanne Wang <jowg@amazon.com> * before moving db Signed-off-by: Joanne Wang <jowg@amazon.com> * after moving db Signed-off-by: Joanne Wang <jowg@amazon.com> * added actions to plugin and removed user schedule Signed-off-by: Joanne Wang <jowg@amazon.com> * unit tests Signed-off-by: Joanne Wang <jowg@amazon.com> * fix build error Signed-off-by: Joanne Wang <jowg@amazon.com> * changed transport naming Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> * converge job scheduler code with threat intel feed integration in detectors Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * converge job scheduler and detector threat intel code Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add feed metadata config files in src and test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * clean up some tests Signed-off-by: Joanne Wang <jowg@amazon.com> * fixed merge conflicts Signed-off-by: Joanne Wang <jowg@amazon.com> * adds ioc fields list in log type config files and ioc fields object in LogType POJO * update csv parser and new metadata field Signed-off-by: Joanne Wang <jowg@amazon.com> * fixed job scheduler interval settings Signed-off-by: Joanne Wang <jowg@amazon.com> * add tests for ioc to fields for each log type Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * removed wildcards Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Signed-off-by: Joanne Wang <109310487+jowg-amazon@users.noreply.github.com> Co-authored-by: Joanne Wang <109310487+jowg-amazon@users.noreply.github.com> Co-authored-by: Joanne Wang <jowg@amazon.com> * fix threat intel integ tests and add update detector logic Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * JS for Threat intel feeds - changed extension (opensearch-project#675) * merge conflicts Signed-off-by: Joanne Wang <jowg@amazon.com> * fixed java wildcards and changed update key name Signed-off-by: Joanne Wang <jowg@amazon.com> * integ test failing Signed-off-by: Joanne Wang <jowg@amazon.com> * fix job scheduler params Signed-off-by: Joanne Wang <jowg@amazon.com> * changed extension and has debug messages Signed-off-by: Joanne Wang <jowg@amazon.com> * clean up Signed-off-by: Joanne Wang <jowg@amazon.com> * fixed job scheduler plugin spi jar resolution * cleaned up TODOs and changed job scheduler name Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> * TIF Job Runner Cleanup (opensearch-project#676) * merge conflicts Signed-off-by: Joanne Wang <jowg@amazon.com> * fixed java wildcards and changed update key name Signed-off-by: Joanne Wang <jowg@amazon.com> * integ test failing Signed-off-by: Joanne Wang <jowg@amazon.com> * fix job scheduler params Signed-off-by: Joanne Wang <jowg@amazon.com> * changed extension and has debug messages Signed-off-by: Joanne Wang <jowg@amazon.com> * clean up Signed-off-by: Joanne Wang <jowg@amazon.com> * fixed job scheduler plugin spi jar resolution * cleaned up TODOs and changed job scheduler name Signed-off-by: Joanne Wang <jowg@amazon.com> * removed google commons unused import, updated interval setting, removed rest action Signed-off-by: Joanne Wang <jowg@amazon.com> * removed policy file and updated name for job scheduler Signed-off-by: Joanne Wang <jowg@amazon.com> * responded to comments about parameter validator and TIFMetadata Signed-off-by: Joanne Wang <jowg@amazon.com> * refactored ThreatIntelFeedDataService and changed variables to public static final where possible Signed-off-by: Joanne Wang <jowg@amazon.com> * changed opensearch-sap-threatintel to opensearch-sap-threat-intel Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> Signed-off-by: Joanne Wang <109310487+jowg-amazon@users.noreply.github.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> * fix TIFJobParameter class Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * test detector updation when feed updation job runs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * removed delete job scheduler code and cleaned up (opensearch-project#678) Signed-off-by: Joanne Wang <jowg@amazon.com> * working integ test (opensearch-project#680) Signed-off-by: Joanne Wang <jowg@amazon.com> * fix timeout of tif job creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove unncessary thread forking in put tif job action Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * refactoring code to address review comments Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * detector trigger detection types Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * pull out threat intel rest tests into separate test class Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add detection types testing in detector trigger for rules and threat intel detection scenarios Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add license header Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add threat intel field aliases in mapping view response Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix threat intel feed parser Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix workflow failing test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * spotless check failures fixed Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove dockerfile (opensearch-project#689) Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Signed-off-by: Joanne Wang <109310487+jowg-amazon@users.noreply.github.com> Co-authored-by: Joanne Wang <109310487+jowg-amazon@users.noreply.github.com> Co-authored-by: Joanne Wang <jowg@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 2.11] Backport #669 to 2.11 #690

[Backport 2.11] Backport #669 to 2.11 #690

Commits on Oct 26, 2023