Add setting for metrics endpoint

[link2xt]

Currently metrics are exposed over https under /metrics, but without any authentication. Metrics are generated with a cronjob. Current approach to metric generation is very limited, we have #183 open for exposing disk space and #273 (comment)

Since we want more dynamic metrics and expose data such as disk space that should not be publicly available, we will need to expose them over TCP and only to secure network, e.g. wireguard one.

By default we can expose metrics to localhost:9123 or something like this. Someone who wants to monitor chatmail instance can then create a VPN (e.g. wireguard) interface and point the metrics server there to be collected with prometheus on the other side.

[link2xt]

I think we can remove current /metrics entpoint and replace it with mtail if we make doveauth log all account creation events together with the account name so mtail pattern can distinguish between CI and non-CI accounts. This is way we will have metrics updated in realtime instead of rebuilding them with a cronjob.

Another separate endpoint with metrics that cannot be generated by parsing the logs such as the number of accounts in doveadm who could also be interesting, but currently it does not exist so adding it should not be part of this issue. Maybe doveauth should have its own metrics endpoint with the number of accounts logged in in the last 1 day, 1 hour etc.

[link2xt]

New doveauth-based counter after merging #349 looks like this in mtail:

counter created_accounts
counter created_ci_accounts
counter created_nonci_accounts

/: Created account (?P<addr>.*)$/ {
  created_accounts++

  $addr =~ /ci-/ {
    created_ci_accounts++
  } else {
    created_nonci_accounts++
  }
}

[link2xt]

Current mtail scripts:

counter delivered_mail
/saved mail to INBOX$/ {
  delivered_mail++
}

counter quota_exceeded
/Quota exceeded \(mailbox for user is full\)$/ {
  quota_exceeded++
}

# Essentially the number of outgoing messages.
counter dkim_signed
/DKIM-Signature field added/ {
  dkim_signed++
}

counter created_accounts
counter created_ci_accounts
counter created_nonci_accounts

/: Created address: (?P<addr>.*)$/ {
  created_accounts++

  $addr =~ /ci-/ {
    created_ci_accounts++
  } else {
    created_nonci_accounts++
  }
}

[hpk42]

not sure i fully understand but can you do a PR to produce mtail data?

Does nine have a wireguard setup and if so, where is it documented?

[link2xt]

Does nine have a wireguard setup and if so, where is it documented?

It's all documented in the server readme, both wireguard and mtail setup. But wireguard is not related to chatmail, how metrics are collected is up to the server admin.

[link2xt]

Closed with #388, there is now mtail which only exposes metrics locally or to a configured interface.

We should also do something about /metrics, probably remove it. Will open another issue for this.