Plugin broker drops notifications on rapid shutdown (race between Dying channel and pending alerts)

[andreabravetti]

What happened?

The chance the notification is sent is about 50%, literally a roll of the dice.

What did you expect to happen?

Notifications should work.

How can we reproduce it (as minimally and precisely as possible)?

Try to send notification with cscli notifications test [whatever]_default.

Anything else we need to know?

When cscli notifications test or any rapid shutdown triggers the plugin tomb's Kill(), the broker's select loop can pick the Dying() channel before reading PluginChannel, causing the enqueued alert to be silently dropped.

Additionally, after watcher.Kill(), the inner loop races between Dead() and PluginEvents, alerts queued by addProfileAlert via the watcher ticker may be lost if Dead() wins.

Also I have ready a branch with a fix and if there is interest I can make a pull request.

Crowdsec version

Details

$ cscli version
version: 
Codename: 
BuildDate: 
GoVersion: 1.26.1
Platform: linux
libre2: WebAssembly
User-Agent: crowdsec/-linux
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlog, db_mysql, db_postgres, db_sqlite

OS version

Details

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Armbian 26.8.0-trunk.42 trixie"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.5
ID=debian
HOME_URL="https://www.armbian.com"
SUPPORT_URL="https://forum.armbian.com"
BUG_REPORT_URL="https://www.armbian.com/bugs"
ARMBIAN_PRETTY_NAME="Armbian 26.8.0-trunk.42 trixie"
$ uname -a
Linux orion 6.18.32-current-rockchip64 #1 SMP PREEMPT Sun May 17 15:15:37 UTC 2026 aarch64 GNU/Linux

Enabled collections and parsers

Details

$ cscli hub list -o raw
Loaded: 165 parsers, 12 postoverflows, 781 scenarios, 9 contexts, 5 appsec-configs, 213 appsec-rules, 163 collections
Unmanaged items: 3 local, 0 tainted
name,status,version,description,type
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/dovecot-logs,enabled,0.9,Parse dovecot logs,parsers
crowdsecurity/geoip-enrich,enabled,0.5,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/http-logs,enabled,1.3,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers
crowdsecurity/postfix-logs,enabled,0.9,Parse postfix logs,parsers
crowdsecurity/postscreen-logs,enabled,0.3,Parse postscreen logs,parsers
crowdsecurity/public-dns-allowlist,enabled,0.1,Allow events from public DNS servers,parsers
crowdsecurity/sshd-logs,enabled,3.1,Parse openSSH logs,parsers
crowdsecurity/sshd-success-logs,enabled,0.1,Parse successful ssh logins,parsers
crowdsecurity/syslog-logs,enabled,1.0,,parsers
crowdsecurity/traefik-logs,enabled,1.5,Parse Traefik access logs,parsers
crowdsecurity/whitelists,enabled,0.3,Whitelist events from private ipv4 addresses,parsers
lan-whitelist,"enabled,local",,,parsers
stalwart/stalwart-logs,"enabled,local",,,parsers
crowdsecurity/cdn-whitelist,enabled,0.5,Whitelist CDN providers,postoverflows
crowdsecurity/google-special-crawlers-whitelist,enabled,0.1,"Whitelist events from Google special crawlers (e.g. Google-InspectionTool, GoogleOther)",postoverflows
crowdsecurity/rdns,enabled,0.3,Lookup the DNS associated to the source IP only for overflows,postoverflows
crowdsecurity/seo-bots-whitelist,enabled,0.5,Whitelist good search engine crawlers,postoverflows
crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.7,Detect cve-2021-44228 exploitation attempts,scenarios
crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios
crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios
crowdsecurity/CVE-2022-26134,enabled,0.4,Confluence - RCE (CVE-2022-26134),scenarios
crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios
crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios
crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios
crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios
crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios
crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios
crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios
crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios
crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios
crowdsecurity/CVE-2023-22518,enabled,0.3,Detect CVE-2023-22518 exploits,scenarios
crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios
crowdsecurity/CVE-2024-0012,enabled,0.1,Detect CVE-2024-0012 exploitation attempts,scenarios
crowdsecurity/CVE-2024-38475,enabled,0.1,Detect CVE-2024-38475 exploitation attempts,scenarios
crowdsecurity/CVE-2024-9474,enabled,0.1,Detect CVE-2024-9474 exploitation attempts,scenarios
crowdsecurity/dovecot-spam,enabled,0.6,Detect Dovecot bruteforce,scenarios
crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.3,F5 BIG-IP TMUI - RCE (CVE-2020-5902),scenarios
crowdsecurity/fortinet-cve-2018-13379,enabled,0.4,Detect cve-2018-13379 exploitation attempts,scenarios
crowdsecurity/grafana-cve-2021-43798,enabled,0.3,Grafana - Arbitrary File Read (CVE-2021-43798),scenarios
crowdsecurity/http-admin-interface-probing,enabled,0.5,Detect generic HTTP admin interface probing,scenarios
crowdsecurity/http-backdoors-attempts,enabled,0.6,Detect attempt to common backdoors,scenarios
crowdsecurity/http-bad-user-agent,enabled,1.2,Detect usage of bad User Agent,scenarios
crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios
crowdsecurity/http-cve-2021-41773,enabled,0.3,Apache - Path Traversal (CVE-2021-41773),scenarios
crowdsecurity/http-cve-2021-42013,enabled,0.3,Apache - Path Traversal (CVE-2021-42013),scenarios
crowdsecurity/http-cve-probing,enabled,0.6,Detect generic HTTP cve probing,scenarios
crowdsecurity/http-generic-bf,enabled,0.9,Detect generic http brute force,scenarios
crowdsecurity/http-generic-test,enabled,0.2,Crowdsec Generic Test Scenario: basic HTTP trigger,scenarios
crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios
crowdsecurity/http-path-traversal-probing,enabled,0.4,Detect path traversal attempt,scenarios
crowdsecurity/http-probing,enabled,0.4,Detect site scanning/probing from a single ip,scenarios
crowdsecurity/http-sap-interface-probing,enabled,0.1,Detect generic HTTP SAP interface probing,scenarios
crowdsecurity/http-sensitive-files,enabled,0.4,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios
crowdsecurity/http-sqli-probing,enabled,0.4,A scenario that detects SQL injection probing with minimal false positives,scenarios
crowdsecurity/http-technology-probing,enabled,0.1,Detect HTTP technology/vendor probing,scenarios
crowdsecurity/http-wordpress-scan,enabled,0.4,Detect exploitation attempts against common WordPress endpoints,scenarios
crowdsecurity/http-xss-probing,enabled,0.4,A scenario that detects XSS probing with minimal false positives,scenarios
crowdsecurity/jira_cve-2021-26086,enabled,0.4,Detect Atlassian Jira CVE-2021-26086 exploitation attempts,scenarios
crowdsecurity/netgear_rce,enabled,0.4,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios
crowdsecurity/postfix-helo-rejected,enabled,0.1,Detect HELO rejections,scenarios
crowdsecurity/postfix-non-smtp-command,enabled,0.1,Detect scanning of postfix service through non-SMTP commands,scenarios
crowdsecurity/postfix-relay-denied,enabled,0.1,Detect multiple open relay attempts,scenarios
crowdsecurity/postfix-spam,enabled,0.4,Detect spammers,scenarios
crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.4,Detect cve-2019-11510 exploitation attempts,scenarios
crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios
crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-cve-2024-6387,enabled,0.2,Detect exploitation attempt of CVE-2024-6387,scenarios
crowdsecurity/ssh-generic-test,enabled,0.2,Crowdsec Generic Test Scenario: SSH brute force trigger,scenarios
crowdsecurity/ssh-refused-conn,enabled,0.1,Detect sshd refused connections,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios
crowdsecurity/ssh-time-based-bf,enabled,0.2,Detect time-based ssh bruteforce attempts that evade rate limiting (with false positive reduction),scenarios
crowdsecurity/thinkphp-cve-2018-20062,enabled,0.7,Detect ThinkPHP CVE-2018-20062 exploitation attempts,scenarios
crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios
crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.3,Detect VMSA-2021-0027 exploitation attempts,scenarios
ltsich/http-w00tw00t,enabled,0.3,detect w00tw00t,scenarios
stalwart/stalwart-smtp-auth-bf,"enabled,local",,,scenarios
crowdsecurity/bf_base,enabled,0.1,,contexts
crowdsecurity/http_base,enabled,0.3,,contexts
crowdsecurity/base-http-scenarios,enabled,1.4,http common : scanners detection,collections
crowdsecurity/dovecot,enabled,0.2,dovecot support : parser and spammer detection,collections
crowdsecurity/http-cve,enabled,3.0,Detect CVE exploitation in http logs,collections
crowdsecurity/linux,enabled,0.4,core linux support : syslog+geoip+ssh,collections
crowdsecurity/postfix,enabled,0.5,postfix support : parser and spammer detection,collections
crowdsecurity/sshd,enabled,0.9,sshd support : parser and brute-force detection,collections
crowdsecurity/traefik,enabled,0.2,traefik support: parser and generic http scenarios,collections
crowdsecurity/whitelist-good-actors,enabled,0.4,Good actors whitelists,collections

Acquisition config

Details

# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
cat: /etc/crowdsec/acquis.yaml: No such file or directory
source: file
filenames:
  - /srv/mailserver/logs/mail.log
labels:
  type: syslog
#
# Configuration generated by "cscli setup".
# Please check your non-generated configuration files to make sure
# the log sources are not acquired twice. This includes
# the file acquis.yaml created by crowdsec <= 1.7.0.
#
# cscli-checksum: a380e4b3fc5e43e510e8287a3fcbe374

filenames:
  - /var/log/messages
  - /var/log/syslog
  - /var/log/kern.log
labels:
  type: syslog
source: file
#
# Configuration generated by "cscli setup".
# Please check your non-generated configuration files to make sure
# the log sources are not acquired twice. This includes
# the file acquis.yaml created by crowdsec <= 1.7.0.
#
# cscli-checksum: 70986510f568531d62e879aeeea05989

filenames:
  - /var/log/auth.log
  - /var/log/secure
labels:
  type: syslog
source: file
source: file
filenames:
  - /srv/stalwart/logs/stalwart.*
labels:
  type: stalwart
source: file
filenames:
  - /srv/traefik/logs/access.log
labels:
  type: traefik

Config show

Details

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Notification Folder    : /etc/crowdsec/notifications
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              : 
API Client:
  - URL                     : censored
  - Login                   : censored
  - Credentials File        : censored
Local API Server:
  - Listen URL              : censored
  - Listen Socket           : 
  - Profile File            : /etc/crowdsec/profiles.yaml

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 168h0m0s
      - Flush size          : 5000

Prometheus metrics

Details

$ cscli metrics
censored, irrelevant

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

Details

[github-actions]

@andreabravetti: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.