[Feature] Solve alarm current limiting

[xujiangfeng001]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

In the actual production environment, streampark may have current restriction when using software alarms such as
flash book and nail, which may result in some alarm messages not being sent.
In my opinion, alarms should not be sent because they are an important indicator of task status.I want to solve the alarm problem by adding a blocking queue. When the task fails, the alarm is not sent at the first time, but added to the blocking queue,Then send the alarm through a separate alarm thread. Of course, this change may affect the effectiveness of the alarm. After my test, this change can solve the problem of alarm current limiting.

Usage Scenario

Task alarm

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[MonsterChenzhuo]

+1 good suggestion.
I have some suggestions, the general standard practice in the industry is that there will be some parameters that can be set, such as specifying the maximum number of sends in 30 minutes, as a way to avoid unlimited alarms, I think these flow limiting measures should be optional for the user, rather than the system default

[MonsterChenzhuo]

The picture above, is a very powerful software in the industry to do alarm, I think these can give you some reference.

[MonsterChenzhuo]

I think a core point is that we should provide the user with some rules for alerting, not write dead by default

[xujiangfeng001]

Thank you for your suggestion. I think adding alarm rules is a good proposal. I also heard some requirements for alarm rules, and I will think about it again and improve this idea.

[ziqiang-wang]

I'll add a little feature:
When getting an alarm waiting to be sent in the blocking queue, get the number of elements in the queue. If a certain value is checked, send an alarm related to the alarm backlog first, telling the user that there is an alarm backlog and the number of the backlog.

[xujiangfeng001]

I'll add a little feature: When getting an alarm waiting to be sent in the blocking queue, get the number of elements in the queue. If a certain value is checked, send an alarm related to the alarm backlog first, telling the user that there is an alarm backlog and the number of the backlog.

Thank you for your proposal. I'll take it into consideration.