Add Grafana integration example

[fktkrt]

Which problem is this PR solving?

• The idea came from Add TraceID to Logs in HotRod Example Application #2310 to provide a bundled Grafana integration example

Short description of the changes

• This is a rework of one of the early official Grafana demos and the hotrod compose, having Grafana, Loki and Jaeger packed together to provide easy correlation with metrics, logs and traces in one application.
• If the effort is welcomed, I can add further details to the README regarding the infrastructure.

[codecov]

Codecov Report

Merging #2408 into master will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2408   +/-   ##
=======================================
  Coverage   95.56%   95.57%           
=======================================
  Files         208      208           
  Lines       10676    10682    +6     
=======================================
+ Hits        10203    10209    +6     
+ Misses        401      400    -1     
- Partials       72       73    +1     

Impacted Files	Coverage Δ
cmd/query/app/server.go	91.11% <0.00%> (-2.23%)	⬇️
plugin/storage/es/factory.go	100.00% <0.00%> (ø)
plugin/storage/es/options.go	100.00% <0.00%> (ø)
pkg/multicloser/multicloser.go	100.00% <0.00%> (ø)
...lugin/sampling/strategystore/adaptive/processor.go	100.00% <0.00%> (+0.79%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09fde54...33bc30c. Read the comment docs.

[joe-elliott]

Tested locally and everything worked after noted changes. I'm a little biased, but this is a neat contribution. As soon as a version of HotRod is published with #2384 this example can be improved by providing direct links from logs to traces.

Also, make sure your commit is signed off.

[fktkrt]

Tested locally and everything worked after noted changes. I'm a little biased, but this is a neat contribution. As soon as a version of HotRod is published with #2384 this example can be improved by providing direct links from logs to traces.

Also, make sure your commit is signed off.

Thank you for the review, all should be done.
Yes, the direct link can be easily done with derivedFields and some regexp to match the traceIDs.

[joe-elliott]

Looks good. It's a bit more of a Grafana example than a Jaeger one so it's up to the maintainers to merge, but I think it's ready to go!

Nice work.

[fktkrt]

Thanks again!
Yes, it's a bit Grafana heavy, but I think it has the potential to draw more attention to Jaeger, as now it has native support as a datasource.

[yurishkuro]

It would be useful to explain this in more detail. If someone is new to this whole space, with the main HotROD example we at least have a detailed blog and many videos showing what to do with the example. But here you're assuming substantial familiarity with Grafana and Loki. For example, I think there are some standard Grafana dashboards for Jaeger, could this be included? Or at least a dashboard that shows, ideally, both the HotROD metrics, Loki logs, and traces.

[jpkrohling]

+1 for adding a Grafana dashboard to the main monitoring directory in the repository, similar to what we have for Jaeger itself:

https://github.com/jaegertracing/jaeger/tree/master/monitoring/jaeger-mixin

[fktkrt]

Yes, I agree, that's exactly what I meant in the PR description.
I can think of multiple possible ways to extend/transform this:

• I can remove the Loki integration, this way it's a much more simple solution, basically a clean Grafana demo as the frontend to Jaeger -> the setup is more simple, shorter introduction is needed
• I can give additional details to this README to make it easier to understand the setup AND put together an even more detailed blog post, explaining all aspects thoroughly

As for the dashboards:
As far as I know there isn't any official Jaeger dashboard that could be useful for this scenario (if we would like to focus on the tracing aspect of it), since most of them are for monitoring Jaeger itself currently. These can be added, and should be added, just these serve a bit different purpose in my opinion, so having these under monitoring/ can make sense.

Although, I've just found this one, which is something similar what could be useful for this setup, but some changes would be needed to tailor to our needs (if it's a welcomed effort, I can work on it).
Incorporating a dashboard to filter actual logs (and eventually metrics, since those are not yet implemented in this setup) is a greater task, and it might benefit from a more detailed blog post format, but I am happy to work on those as well.
What do you think?

[jpkrohling]

I'm all for having a dashboard that serves as the official one, as long as we have people committed to maintain it.

[fktkrt]

Alright, I can open a PR to have one for the logs then.

In that case we would need a setup to validate and use it, so based on these, I'd say the Loki setup should be included as well.
Although, another option would be to have a dedicated example for vanilla Grafana (for the sake of simplicity), and one for this more advanced one, where Loki and related dashboards are included.

[yurishkuro]

as long as the overall setup is simple i.e. docker-compose up, I think it makes sense to include Loki integration and have a dashboard that already includes all the relevant panels for HotROD: metrics, logs, and traces.

[fktkrt]

I like the idea.
I will add Prometheus to this setup too, then start working on a full-fledged HotROD overview dashboard.

[fktkrt]

Just pushed a new config w/ Prometheus integration.
I will need to look into it more thoroughly, but after taking a quick look at the metrics including errors and the error labels, it's not quite trivial to build useful metrics panels.

At the moment spotting issues seems easier with Loki, since I can see just filter for {compose_project="grafana-jaeger"} |= "traceID" then I can jump to the traces immediately (and this will be even better after #2384).

Are you aware of any directions/documentation focusing on the HotROD Prometheus metrics to fasten up this part?

[fktkrt]

Added the first version of the HotROD dashboard, reorganized the file structure, extended the README.
Currently it looks like the redis timeouts don't have traceIDs, and the application metrics are not really showing significant error symptoms.
I had a discussion with @joe-elliott, one of us will check the redis logging, and later I might will be able to open another PR to have new metrics.
Anyway, it looks like the current setup can be considered finished from my perspective.
Once we have these metrics, I am planning to write a blogpost introducing this new kind of troubleshooting experience.

[fktkrt]

@yurishkuro All the suggestions are addressed, thank you.
Is there anything else needed?

[yurishkuro]

Thanks!