Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Threat Intel Feed Job Scheduler with unit and integ test #664

Merged
merged 46 commits into from
Oct 17, 2023
Merged

Threat Intel Feed Job Scheduler with unit and integ test #664

merged 46 commits into from
Oct 17, 2023

Commits on Oct 7, 2023

  1. add mapping for indices storing threat intel feed data

    @eirsep
    eirsep committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    00da38d View commit details
    Browse the repository at this point in the history

  2. fix feed indices mapping

    @eirsep
    eirsep committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    49a2981 View commit details
    Browse the repository at this point in the history

  3. add threat intel feed data dao 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    9487746 View commit details
    Browse the repository at this point in the history

  4. add threatIntelEnabled field in detector. 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    5378532 View commit details
    Browse the repository at this point in the history

  5. add threat intel feed service and searching feeds 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    805ca58 View commit details
    Browse the repository at this point in the history

  6. ti feed data to doc level query convertor logic added

    @eirsep
    eirsep committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    add8987 View commit details
    Browse the repository at this point in the history

  7. plug threat intel feed into detector creation 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    540837b View commit details
    Browse the repository at this point in the history

  8. Preliminary framework for jobscheduler and datasource (opensearch-pro… 

    …ject#626)


Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon @eirsep
    jowg-amazon authored and eirsep committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    1980f26 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2023

  1. create doc level query from threat intel feed data index docs" 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    8415d9f View commit details
    Browse the repository at this point in the history

  2. handle threat intel enabled check during detector updation

    @eirsep
    eirsep committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    adbfdd5 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. add tests for testing threat intel feed integration with detectors 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    cfd1bf0 View commit details
    Browse the repository at this point in the history

  2. Threat intel feeds job runner and unit tests (opensearch-project#654) 

    * fix doc level query constructor (opensearch-project#651)

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add mapping for indices storing threat intel feed data

* fix feed indices mapping

* add threat intel feed data dao

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add threatIntelEnabled field in detector.

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add threat intel feed service and searching feeds

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* ti feed data to doc level query convertor logic added

* plug threat intel feed into detector creation

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* Preliminary framework for jobscheduler and datasource (opensearch-project#626)


Signed-off-by: Joanne Wang <jowg@amazon.com>

* with listener and processor

Signed-off-by: Joanne Wang <jowg@amazon.com>

* removed actions

Signed-off-by: Joanne Wang <jowg@amazon.com>

* clean up

Signed-off-by: Joanne Wang <jowg@amazon.com>

* added parser

Signed-off-by: Joanne Wang <jowg@amazon.com>

* add unit tests

Signed-off-by: Joanne Wang <jowg@amazon.com>

* refactored class names

Signed-off-by: Joanne Wang <jowg@amazon.com>

* before moving db

Signed-off-by: Joanne Wang <jowg@amazon.com>

* after moving db

Signed-off-by: Joanne Wang <jowg@amazon.com>

* added actions to plugin and removed user schedule

Signed-off-by: Joanne Wang <jowg@amazon.com>

* unit tests

Signed-off-by: Joanne Wang <jowg@amazon.com>

* fix build error

Signed-off-by: Joanne Wang <jowg@amazon.com>

* changed transport naming

Signed-off-by: Joanne Wang <jowg@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Joanne Wang <jowg@amazon.com>
Co-authored-by: Surya Sashank Nistala <snistala@amazon.com>
    @jowg-amazon @eirsep
    jowg-amazon and eirsep authored Oct 10, 2023
    Configuration menu
    Copy the full SHA
    55d332d View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. converge job scheduler code with threat intel feed integration in det… 

    …ectors

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    13b513b View commit details
    Browse the repository at this point in the history

  2. refactored out unecessary 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    61a60b4 View commit details
    Browse the repository at this point in the history

  3. added headers and cleaned up 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    1cd9875 View commit details
    Browse the repository at this point in the history

  4. converge job scheduler and detector threat intel code 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    d0e2511 View commit details
    Browse the repository at this point in the history

  5. working on testing 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    768cc2c View commit details
    Browse the repository at this point in the history

  6. merge from branch 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    e01a70b View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2023

  1. fixed the parser and build.gradle 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    b8aa66d View commit details
    Browse the repository at this point in the history

  2. add mapping for indices storing threat intel feed data

    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    63325ef View commit details
    Browse the repository at this point in the history

  3. fix feed indices mapping

    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    99cb74e View commit details
    Browse the repository at this point in the history

  4. add threat intel feed data dao 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    5aa9720 View commit details
    Browse the repository at this point in the history

  5. add threatIntelEnabled field in detector. 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    41ae481 View commit details
    Browse the repository at this point in the history

  6. add threat intel feed service and searching feeds 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    a5306f4 View commit details
    Browse the repository at this point in the history

  7. ti feed data to doc level query convertor logic added

    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    c7d595b View commit details
    Browse the repository at this point in the history

  8. plug threat intel feed into detector creation 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    d84fa71 View commit details
    Browse the repository at this point in the history

  9. Preliminary framework for jobscheduler and datasource (opensearch-pro… 

    …ject#626)


Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon @eirsep
    jowg-amazon authored and eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    0887d91 View commit details
    Browse the repository at this point in the history

  10. create doc level query from threat intel feed data index docs" 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    6dabe61 View commit details
    Browse the repository at this point in the history

  11. handle threat intel enabled check during detector updation

    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    528b978 View commit details
    Browse the repository at this point in the history

  12. add tests for testing threat intel feed integration with detectors 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    375b231 View commit details
    Browse the repository at this point in the history

  13. Threat intel feeds job runner and unit tests (opensearch-project#654) 

    * fix doc level query constructor (opensearch-project#651)

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add mapping for indices storing threat intel feed data

* fix feed indices mapping

* add threat intel feed data dao

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add threatIntelEnabled field in detector.

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add threat intel feed service and searching feeds

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* ti feed data to doc level query convertor logic added

* plug threat intel feed into detector creation

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* Preliminary framework for jobscheduler and datasource (opensearch-project#626)


Signed-off-by: Joanne Wang <jowg@amazon.com>

* with listener and processor

Signed-off-by: Joanne Wang <jowg@amazon.com>

* removed actions

Signed-off-by: Joanne Wang <jowg@amazon.com>

* clean up

Signed-off-by: Joanne Wang <jowg@amazon.com>

* added parser

Signed-off-by: Joanne Wang <jowg@amazon.com>

* add unit tests

Signed-off-by: Joanne Wang <jowg@amazon.com>

* refactored class names

Signed-off-by: Joanne Wang <jowg@amazon.com>

* before moving db

Signed-off-by: Joanne Wang <jowg@amazon.com>

* after moving db

Signed-off-by: Joanne Wang <jowg@amazon.com>

* added actions to plugin and removed user schedule

Signed-off-by: Joanne Wang <jowg@amazon.com>

* unit tests

Signed-off-by: Joanne Wang <jowg@amazon.com>

* fix build error

Signed-off-by: Joanne Wang <jowg@amazon.com>

* changed transport naming

Signed-off-by: Joanne Wang <jowg@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Joanne Wang <jowg@amazon.com>
Co-authored-by: Surya Sashank Nistala <snistala@amazon.com>
    @jowg-amazon @eirsep
    jowg-amazon and eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    e210ebb View commit details
    Browse the repository at this point in the history

  14. converge job scheduler code with threat intel feed integration in det… 

    …ectors

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    59cd533 View commit details
    Browse the repository at this point in the history

  15. converge job scheduler and detector threat intel code 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    c186b21 View commit details
    Browse the repository at this point in the history

  16. add feed metadata config files in src and test 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    9c73abf View commit details
    Browse the repository at this point in the history

  17. clean up some tests 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    0a3a01c View commit details
    Browse the repository at this point in the history

  18. merge 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    bc86671 View commit details
    Browse the repository at this point in the history

  19. fixed merge conflicts 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    6a687c0 View commit details
    Browse the repository at this point in the history

  20. adds ioc fields list in log type config files and ioc fields object i… 

    …n LogType POJO
    @eirsep
    eirsep committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    6b0dfbf View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. update csv parser and new metadata field 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    3afc5b6 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feature/threat_intel_feeds' of https://github.com/opens… 

    …earch-project/security-analytics into threatIntelTest
    @jowg-amazon
    jowg-amazon committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    baaccfb View commit details
    Browse the repository at this point in the history

  3. fixed job scheduler interval settings 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    0221dc0 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2023

  1. add tests for ioc to fields for each log type 

    Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
    @eirsep
    eirsep committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    590af7e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feature/threat_intel_feeds' into threatIntelTest 

    Signed-off-by: Joanne Wang <109310487+jowg-amazon@users.noreply.github.com>
    @jowg-amazon
    jowg-amazon authored Oct 16, 2023
    Configuration menu
    Copy the full SHA
    e65ac85 View commit details
    Browse the repository at this point in the history

  3. merge conflicts 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    7a2ab27 View commit details
    Browse the repository at this point in the history

  4. more merge confligts

    @jowg-amazon
    jowg-amazon committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    6a44876 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. removed wildcards 

    Signed-off-by: Joanne Wang <jowg@amazon.com>
    @jowg-amazon
    jowg-amazon committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    4ce27b2 View commit details
    Browse the repository at this point in the history