Improved logging

[lieut-data]

Summary

Take a sledgehammer to our current logging strategy in order to align around a simple set of ideas:

• Error logs must be human actionable (core misconfiguration, migration failure, panic in job)
• Warning logs are for most "errors": in bulk we may consider them urgent (detected via monitoring)
• Info logs for low-volume transitional events (jobs starting, synthetic user promotion, etc.)
• Debug logs sparingly, and tending towards 0.
• snake_case, consistent parameters (e.g. user_id and teams_user_id and error)
• avoid high-volume logs
• avoid low-quality logs

As part of this, I've gutted all testing of logs: we just accept anything during a test now. I've also fixed the spurious mutex_cron_monitoring_system failures that keep breaking the build.

There are discrete commits if there's interest in reviewing as such. I've also pulled out a separate ticket to investigate high-volume "no results found" type of logs that aren't errors in most cases, but are hard to distinguish from real errors: https://mattermost.atlassian.net/browse/MM-56719.

Ticket Link

Fixes: https://mattermost.atlassian.net/browse/MM-55515
Fixes: https://mattermost.atlassian.net/browse/MM-55516
Fixes: https://mattermost.atlassian.net/browse/MM-55741

[sbishel]

Why the changes to .Maybe()? Seems the mocked functions should still be called.

[lieut-data]

Why the changes to .Maybe()? Seems the mocked functions should still be called.

Some of these are tied to the removal of log changes (i.e. we don't always need to call GetAPI() in order to log something.) But in a practical sense, none of these really "assert" anything except to pass down the mocked implementation at times, so whether they are called or not isn't needful to test, just to "wire up".

This is all going to go away once we abandon (most) mocks in favour of test containers and "real" integration tests.

[jespino]

LGTM. I haven't read too much the specific of each log line, but we can polish that over time anyway.

[jespino]

My only concern is about the proposal of removal the debug logs, but I see a lot of info a warnings replacing the original debug logs, so I guess that is not reduce much our capacity of debugging things. Anyway I still hesitant of that specific point, but to be honest, I think it deserves a try. Let's add and remove debug log as they are needed during the initial phases of development of the future and keep cleaning then up short after. If we don't see ourself struggling with debugging, we can keep that approach.