Prometheus metrics with incorrect name

[pierresebastien]

Describe the bug

Prometheus endpoint is exposing metrics with names that does not respect the Prometheus convention (doc) because '.' is not allowed. Because of this issue, Prometheus is generating errors when scraping metrics : 'invalid metric type "sha256_rules_file.falco_rules_incubating_info gauge"'
Problematic metrics:

# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_incubating_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_incubating_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_incubating_info{raw_name="falco.sha256_rules_file.falco_rules_incubating",falco.sha256_rules_file.falco_rules_incubating="6895d60a215297def72ffe9a7e5a4c8034df522c8b79c7ee6bbf29e2838e4985"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_sandbox_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_sandbox_rules_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_sandbox_rules_info{raw_name="falco.sha256_rules_file.falco_sandbox_rules",falco.sha256_rules_file.falco_sandbox_rules="36aaf859414881458a7cb54d36c852746bf1c1b867c25c96749a62da5d06ac1d"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_local_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_local_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_local_info{raw_name="falco.sha256_rules_file.falco_rules_local",falco.sha256_rules_file.falco_rules_local="c4d5914716e52fce3573231e91ff22d5d4e79a7fc559c63c45bf91c57bd6912a"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_sandbox_rules_override_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_sandbox_rules_override_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_sandbox_rules_override_info{raw_name="falco.sha256_rules_file.falco_sandbox_rules_override",falco.sha256_rules_file.falco_sandbox_rules_override="d746226da11b7d5ba4de231ed2cd3d71db0c5e279ba2e9926a89d4b085d49273"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_info{raw_name="falco.sha256_rules_file.falco_rules",falco.sha256_rules_file.falco_rules="b694934ea611fea743f49932dbc64915ac5fedff8fc71ceb4568f78562c68125"} 1
# HELP falcosecurity_falco_falco.sha256_config_file.falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_config_file.falco_info gauge
falcosecurity_falco_falco.sha256_config_file.falco_info{raw_name="falco.sha256_config_file.falco",falco.sha256_config_file.falco="f80032a6ce3cf345beb5ccff7b4918160e0dae337ee2e301a8f28f0c5f8b1989"} 1

How to reproduce it

Deploy falco on a similar system than the one described in the 'Environment' section below and retrieve metrics :
curl localhost:8765/metrics

Expected behaviour

Same naming convention for these metrics than the one described in the Falco documentation

# HELP falcosecurity_falco_falco_sha256_rules_file_falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_file_falco_rules_info gauge
falcosecurity_falco_falco_sha256_rules_file_falco_rules_info{raw_name="falco_sha256_rules_file_falco_rules",falco_sha256_rules_file_falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco_sha256_config_file_falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_config_file_falco_info gauge
falcosecurity_falco_falco_sha256_config_file_falco_info{raw_name="falco_sha256_config_file_falco",falco_sha256_config_file_falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1

Environment

• Falco version:

Fri Jun  7 08:54:06 2024: Falco version: 0.38.0 (x86_64)
Fri Jun  7 08:54:06 2024: Falco initialized with configuration files:
Fri Jun  7 08:54:06 2024:    /etc/falco/falco.yaml
Fri Jun  7 08:54:06 2024: System info: Linux version 6.1.0-18-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01)
{"default_driver_version":"7.2.0+driver","driver_api_version":"8.0.0","driver_schema_version":"2.0.0","engine_version":"40","engine_version_semver":"0.40.0","falco_version":"0.38.0","libs_version":"0.17.1","plugin_api_version":"3.5.0"}

• System info:

{
  "machine": "x86_64",
  "nodename": "dfakto-uat-lin-mon.dfakto.infra",
  "release": "6.1.0-18-amd64",
  "sysname": "Linux",
  "version": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01)"
}

• Cloud provider or hardware configuration: Virtual machine on VMware Vsphere 7 on OVHCloud
• OS:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

• Kernel:

Linux dfakto-uat-lin-mon.dfakto.infra 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

• Installation method: APT package

Additional context

Relevant section in /etc/falco/falco.yaml :

metrics:
  enabled: true
  interval: 1h
  # Typically, in production, you only use `output_rule` or `output_file`, but not both. 
  # However, if you have a very unique use case, you can use both together.
  output_rule: true
  # output_file: /tmp/falco_stats.jsonl
  resource_utilization_enabled: true
  state_counters_enabled: true
  kernel_event_counters_enabled: true
  libbpf_stats_enabled: true
  convert_memory_to_mb: true
  include_empty_values: false

webserver:
  prometheus_metrics_enabled: true
  enabled: true
  # When the `threadiness` value is set to 0, Falco will automatically determine
  # the appropriate number of threads based on the number of online cores in the system.
  threadiness: 0
  listen_port: 8765
  # Can be an IPV4 or IPV6 address, defaults to IPV4
  listen_address: 0.0.0.0
  k8s_healthz_endpoint: /healthz
  ssl_enabled: false
  ssl_certificate: /etc/falco/falco.pem

[incertum]

Thanks for reporting! We already fixed it in #3232 along few other minor fixes. Falco 0.38.1 will be released very soon.

[incertum]

Plus for Falco 0.39.0 we will make it even more robust and expand the metrics framework even more, just FYI. If you have additional feedback re what other metrics could be useful, please share your thoughts 🙃 much appreciated!

[FedeDP]

Falco 0.38.1 is now out and ships with the fix!
/close

[poiana]

@FedeDP: Closing this issue.

In response to this:

Falco 0.38.1 is now out and ships with the fix!
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pierresebastien]

Thanks for the quick answer @incertum, I just tested the release 0.38.1 and it is indeed working like a charm now. I don't have the time to work with these metrics and eventually create a Grafana dashboard for the moment but I won't hesitate to share some thought if I have interesting feedback to provide.

[FedeDP]

Thanks for testing!

[incertum]

@pierresebastien if you have any thoughts on Grafana dashboards, please let us know. We have one open issue here falcosecurity/charts#672 or falcosecurity/cncf-green-review-testing#12. Thank you!