Flood Control: ability to suppress alarms

[slominskir]

It might be useful to handle alarm floods via configured alarm suppression relationships. An alarm flood occurs when an overwhelming number of alarms are triggered, but a smaller number of alarms could convey the same state (alarms sometimes are dependent on or implied by others). Managing alarm floods is important to keep the operator's situational awareness intact.

When an alarm is registered we could add an optional field named "suppressed by" to each registered alarm such that an alarm can optionally be suppressed by another alarm (higher up the logical hierarchy). For example, a component inside a rack could be "suppressed by" the rack itself. This could be recursive to form a tree, if desired. For example, the rack could be suppressed by a room alarm. The suppression info would be informational only (all alarms would still be alarming) - the client (GUI) could simply use the suppression info to collapse suppressed alarms under a parent alarm (or some other visual cue could be used to indicate the alarm is "trumped" by another). This mimics how Exceptions are often handled in programing languages. You see the high level alarm prominently, but can optionally drill down into the details if desired.

[slominskir]

Note: A CALC alarm could be used to handle flood control too. See: #8. A CALC alarm is a more complex and round about way though - you'd need to define a conditional alarm for every alarm prone to join floods and remove it from standard monitoring (epics2kafka) and instead register it as a CALC alarm that only alarms if it's parent alarm is not already alarming.

[slominskir]

Its also worth acknowledging that the source of alarms could handle flood control too - In EPICS for example an IOC could be coded to not alarm if there is already a more prominent alarm active.

[theojlab]

The "suppressed by" has a conceptual simplicity that I like. A nice feature would be that the hierarchy can be viewed in a GUI configuration tool and therefore the relations be made obvious to users. If the logic is embedded in IOC code or even in CALC records, it would be harder or even impossible to give the same high level overview to the alarms manager.

[slominskir]

We have the MaskedOverride and maskedby attribute for this, but we haven't implemented the logic to handle it in the effective-processor yet: JeffersonLab/jaws-effective-processor#2