Add topic tag to producer metrics

[taylanisikdemir]

What changed?
Messaging producer (kafka) metrics were missing the topic tag which makes it impossible to make sense of count/latency metrics emitted by this component. In the past we only had visibility topics per environment so topic tag wasn't needed but with the introduction of Async APIs there can be multiple topics per environment.

Changes:

• Adding optional custom tag support to messaging.NewMetricProducer
• Pass topic tag option in the places calling this constructor
• Add unit tests

Misc change:

• Adding logs for async wf producer creation because producers are lazily initialized by Async API handlers and subject to 5m ttl. These logs will help us debug how often we re-create the producers for the same target topic.

How did you test it?
Unit tests

[codecov]

Codecov Report

Attention: Patch coverage is 85.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 73.00%. Comparing base (fbae51a) to head (1c18eee).
Report is 1 commits behind head on master.

Files	Patch %	Lines
common/messaging/kafka/client_impl.go	0.00%	2 Missing ⚠️
common/messaging/metrics_producer.go	92.85%	1 Missing ⚠️

Additional details and impacted files

Files	Coverage Δ
common/asyncworkflow/queue/kafka/queue.go	82.60% <100.00%> (+1.21%)	⬆️
common/messaging/metrics_producer.go	81.81% <92.85%> (ø)
common/messaging/kafka/client_impl.go	18.29% <0.00%> (-0.23%)	⬇️

... and 7 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fbae51a...1c18eee. Read the comment docs.

[Shaddoll]

why do we export this type?