Autoignore mechanism for hijacks of limited impact/visibility

[vkotronis]

Description of PR

What component(s) does this PR affect?

• Back-End (Database, Detection/Configuration/etc. Microservices)
• Front-End (Flask, API, UI, etc)
• Monitor (RIPE RIS, BGPStream RV/RIS/CAIDA, etc.)
• Docs (incl. wiki)
• Build System

database, scheduler, env vars

Does the PR require changes on other components? If yes, please mark the components:

• Back-End (Database, Detection/Configuration/etc. Microservices)
• Front-End (Flask, API, UI, etc)
• Monitor (RIPE RIS, BGPStream RV/RIS/CAIDA, etc.)
• Docs (incl. wiki)
• Build System

Related Issue

Resolves #346

Solution

Type

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Docs update
• None of the above

Checklist:

• I have read the contributing guide and my code conforms to the guidelines.

• This change requires a change in the documentation.

• I have updated the documentation accordingly.

• TODO: Add proper tests (TestAutoIgnore)

• TODO: adjust per configured prefix, consider module separation

[artemis-pr-bot]

🏃 Benchmark Results 🏃

• RMQ Update inserts: 444/s
• RMQ Hijack inserts: 146/s
• PG Update inserts: 190/s
• PG Update updates: 145/s
• PG Hijack inserts: 145/s

[codecov]

Codecov Report

Merging #373 into master will increase coverage by 1.10%.
The diff coverage is 89.52%.

@@            Coverage Diff             @@
##           master     #373      +/-   ##
==========================================
+ Coverage   79.72%   80.82%   +1.10%     
==========================================
  Files           5        6       +1     
  Lines        1859     2044     +185     
==========================================
+ Hits         1482     1652     +170     
- Misses        377      392      +15     

Impacted Files	Coverage Δ
backend/core/detection.py	92.85% <ø> (+1.02%)	⬆️
backend/core/autoignore.py	88.48% <88.48%> (ø)
backend/core/configuration.py	48.09% <95.23%> (+1.48%)	⬆️
backend/core/scheduler.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a11adf...88e0bae. Read the comment docs.

[vkotronis]

TODO:

auto_ignore_rules:
  - auto_ignore_rule_1: 
      threshold_num_peers_seen: <number of peers seen the hijack below which alert should be ignored>
      threshold_num_ases_infected: <number of infected ASes below which alert should be ignored>
      interval: <time passed since last hijack update>
      prefixes: 
      - *prefix_group_1
      - ...
  - auto_ignore_rule_2 
     ...

DONE, will also update the wiki accordingly.

[vkotronis]

@slowr @pgigis This PR is ready for review. Please check the code (I have also added needed tests) as well as this wiki page on how to use the new mechanism: https://github.com/FORTH-ICS-INSPIRE/artemis/wiki/Configuration-file#autoignore

[vkotronis]

@slowr @pgigis any feedback on this PR code review?

[slowr]

LGTM 🎉 What happens if the same prefix is on multiple autoignores?

[vkotronis]

LGTM tada What happens if the same prefix is on multiple autoignores?

If the same prefix is on multiple rules, if at least one of these rules causes a prefix-related alert to be ignored, it will be ignored. This is the correct behavior, right? Or should we do sth else, like issuing a warning to the user? I think that since the user puts it in at least one group that causes autoignore, it should be autoignored (the related alert I mean). See also my comments in the PR. Should I merge the current state or update something in the code?

[slowr]

LGTM tada What happens if the same prefix is on multiple autoignores?

If the same prefix is on multiple rules, if at least one of these rules causes a prefix-related alert to be ignored, it will be ignored. This is the correct behavior, right? Or should we do sth else, like issuing a warning to the user? I think that since the user puts it in at least one group that causes autoignore, it should be autoignored (the related alert I mean). See also my comments in the PR. Should I merge the current state or update something in the code?

Yes, I think this is the correct behaviour for now. In the future we can show an alert that it's on multiple rules for cleanup if needed.