Istio metrics and dashboards

[gsantoro]

Review the implementation of Istiod_metrics datastream under the Istio Integration according to this comment.

Related PRs

• Feature/istio metrics #4253

Related issues

• [Epic] Release Istio integration to Beta #3552

Make sure that the following Criteria are met before releasing to GA:

• Metricbeat Module / Dataset release checklist

Checklist for GA:

• Supported versions are documented
• Supported operating systems are documented (if applicable)
• Integration tests exist
• System tests exist
• Automated checks that all fields are documented
• Documentation
• Fields follow ECS and naming conventions
• Dashboards exists (if applicable)
• Kibana Home Tutorial (if applicable)
  • Open PR against Kibana repo with tutorial. Examples can be found here. 404 on this link
• Test log files exist for the grok patterns (from Filebeat)
• Generated output for at least 1 log file exists (from Filebeat)
• Example data.json exists and an automated way to generate it exists (go test -data) (from Metricbeat)
• Test environment in Docker exist for integration tests (from Metricbeat)
• Rewrite Dashboards from TSVB to Lens PR

[gsantoro]

Comment has been addressed with release 0.2.2 and PR

[gsantoro]

Notes on checklist for GA:

• integration and system tests
  • pipelines tests for access_logs datastream
  • system tests for istiod_metrics and proxy_metrics datastream with docker-compose
• dashboard exists
  • dashboard are valid and can be loaded. Checks done by elastic-package test asset via jenkins
• Automated checks that all fields are documented
  • all fields in sample_event.json are verified by elastic-package test static. doc
• documentation in the README
• instead of data.json, we have sample_event.json (one per data stream)
• support version is documented in the readme. Istio 1.14.3

The previou checklist should cover all the checks from beats from this doc

[gsantoro]

More checks specific to integrations:

• beats module in GA?
  • istio has a beats module but it doesn't use it for either logs (it uses filestream input) or metrics (it uses prometheus/collector module/metricset)
  • integrations in Beta since Sep 12, 2022
• Have all the data sets been migrated to Integrations as data streams ?
  • Yes. Since with Istio 1.14.3 we don't need as many different datastream. Components like Galley, Citadel, Pilot have been merged into Istiod. The migrated datastream are Proxy and Istiod, which are the only ones necessary. There was no previous module for Istio under filebeat, there is now an access_logs datastream in the integration
• Data stream specific checks
  • All metrics and logs are visible in Kibana UI
  • User can give proper inputs for each stream from the UI (streams.yml.hbs file)
  • Fields.yml, ecs.yml, base-fields.yml presence
  • Data stream dashboards have been imported properly to integrations. Dashboards should not be with snapshot versions of stack
  • pipeline tests (when applicable)
  • System Tests, generation of sample_event.json
  • documentation: fileds, events
• versions of service. The integration should be tested with at least 2 versions of the service (one should be the latest release) ??
  • Upgrade to latest version of Istio. Test it and check for errors or new fields
• kibana version compatiblity (check with all kibana versions that we are supporting)
• Verify Code Owners

[gsantoro]

Some optional todo:

• upgrade the integration to Istio 1.16.1
• add new datastream for Istiod service logs. Istiod expose logs which are different from the access logs gathered from Proxy sidecar containers.

[gizas]

We have an open PR that summarises best Practises for Dashboards/ Visualisations here

Long story short, the best practise is to transform our visualisations to Lens.

There are specific cases that TSVB can still have features that are not supported in Lens, but those should be considered per individual case. Lets always consider what the main outcome of each dashboard should be and what is the goal we want to achieve. (Relevant doc - go to section Use Lens if possible)

For cloudnative team we had to change existing gauges and either replace them with Pie charts or with bar charts. We chose bar charts (with ad-hoc dataviews) because we wanted to depict the latest status of cluster (see relevant discussion here), and Pies could not support or case in 8.5

@gsantoro to answer your question regarding replacement of TSVB gauges, I would either try Bar charts or Pie charts (after 8.6) that support latest value filter. We also have another story in our backlog to investigate Pie charts and if is possible to use them in Kubernetes General Overview Dashboards. So it will be a win-win if you can use them here as well

[gizas]

@gsantoro Reminder to include in new version of dashboards the #4978

[gsantoro]

@gizas are you saying that I should do the dashboards update and #4978 in the same PR?

[gizas]

I would suggest yes. The addition is a small one, why to open 2 prs only for a filter?
But however you feel. It is more like a reminder not to leave it in backlog because we can solve it now