feat: tedge flows statistics are published over MQTT

[didier-wenzek]

Proposed changes

• Publish basic processing stats over MQTT
• Publish memory stats
• Configure what is published, what is not

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
• I ran just format as mentioned in CODING_GUIDELINES
• I used just check as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[codecov]

Codecov Report

❌ Patch coverage is 76.74419% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/core/tedge_mapper/src/flows/mod.rs	0.00%	9 Missing ⚠️
crates/extensions/tedge_flows/src/stats.rs	81.25%	8 Missing and 1 partial ⚠️
crates/extensions/tedge_flows/src/actor.rs	80.00%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
737	6	3	743	99.19	2h30m10.806697999s

Failed Tests

Name	Message	⏱️ Duration	Suite
Consuming messages from a process stdout	Matching messages on topic 'te/device/main/service/tedge-flows/status/flows' is less than minimum. wanted: 1 got: 0 messages: []	31.792 s	Tedge Flows
Consuming messages from a process stdout, periodically	Matching messages on topic 'te/device/main/service/tedge-flows/status/flows' is less than minimum. wanted: 1 got: 0 messages: []	31.780 s	Tedge Flows
Consuming messages from the tail of file	Matching messages on topic 'te/device/main/service/tedge-flows/status/flows' is less than minimum. wanted: 1 got: 0 messages: []	31.413 s	Tedge Flows
Consuming messages from a file, periodically	Matching messages on topic 'te/device/main/service/tedge-flows/status/flows' is less than minimum. wanted: 1 got: 0 messages: []	31.586 s	Tedge Flows
Appending messages to a file	Matching messages on topic 'te/device/main/service/tedge-flows/status/flows' is less than minimum. wanted: 1 got: 0 messages: []	31.133 s	Tedge Flows
Publishing transformation errors	Matching messages on topic 'te/device/main/service/tedge-flows/status/flows' is less than minimum. wanted: 1 got: 0 messages: []	30.703 s	Tedge Flows

[didier-wenzek]

It's now time to discuss the appropriate topics for flows status and statistics #3846 (comment)

I think that it would make sense to publish the statistics as measurements under {te}/{service_id}/m/statistics. But then these stats will be pushed by default to the cloud.

[reubenmiller]

Yeah publishing under the standard thin-edge.io mqtt topic structure would be more consistent, but we might need to use a different telemetry endpoint to publish this information to avoid sending too much information to the cloud by default.

We could create a convention where some local/internal information which is meant for local/on-device consumption (rather than intended for the cloud):

{te}/{service_id}/local/statistics

Alternatively, we could use flows as the type since the flows statistics maybe published on serveral services in the future, e.g. tedge-mapper-c8y, tedge-mapper-az, tedge-flows etc.

{te}/{service_id}/flows/statistics

[reubenmiller]

A new proposal for the topic would be:

{te}/{service_id}/status/metrics