azure storage account: fix capacity and count metrics visualizations in the all dashboards

[zmoog]

Proposed commit message

Fix capacity and count metrics visualizations in the overview, blob, table, and file storage dashboards.

The capacity and count metric visualizations were missing the filter option, so they all displayed "N/A".

Here is the JSON before:

{
  "customLabel": true,
  "dataType": "number",
  "filter": {
    "language": "kuery",
    "query": "" <————————— empty query here 👀 
  },
  "isBucketed": false,
  "label": "File Share Snapshot Size",
  "operationType": "last_value",
  "params": {
    "format": {
      "id": "bytes"
    },
    "sortField": "@timestamp"
  },
  "scale": "ratio",
  "sourceField": "azure.storage_account.file_share_snapshot_size.avg"
}

I just set the filter.query value with the corresponding metric:

{
  "customLabel": true,
  "dataType": "number",
  "filter": {
    "language": "kuery",
    "query": "azure.storage_account.file_share_snapshot_size.avg: *"
  },
  "isBucketed": false,
  "label": "File Share Snapshot Size",
  "operationType": "last_value",
  "params": {
    "format": {
      "id": "bytes"
    },
    "sortField": "@timestamp"
  },
  "scale": "ratio",
  "sourceField": "azure.storage_account.file_share_snapshot_size.avg"
}

With the proper filter.query values are back.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Screenshots

Before

After

Video tour of all the dashboards:

CleanShot.2024-09-13.at.12.05.38.mp4

[elasticmachine]

🚀 Benchmarks report

Package azure_metrics 👍(1) 💚(2) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
compute_vm	166666.67	100000	-66666.67 (-40%)	💔

To see the full report comment with /test benchmark fullreport

[constanca-m]

Looking at the files it all seems good, did you test all dashboards you changed or just the one you put the screenshot in the description?

[zmoog]

Looking at the files it all seems good, did you test all dashboards you changed or just the one you put the screenshot in the description?

I tested all the dashboards. I'm adding a quick video tour in the description after the screenshots.

[zmoog]

Weird.

In the PRs that migrated these visualizations to Lens, I see the data was visible:

• [O11y][Azure Metrics] Update visualizations and remove suffix #9502
• [O11y][Azure Metrics] Migrate Blob Storage Overview dashboard to lens #9503

But now the data is gone on the local environment (tested using 8.15.0). Is the difference now in the stack version or the data?

@harnish-elastic, let me know if you have ideas.

[harnish-elastic]

Weird.

In the PRs that migrated these visualizations to Lens, I see the data was visible:

• [O11y][Azure Metrics] Update visualizations and remove suffix #9502
• [O11y][Azure Metrics] Migrate Blob Storage Overview dashboard to lens #9503

But now the data is gone on the local environment (tested using 8.15.0). Is the difference now in the stack version or the data?

@harnish-elastic, let me know if you have ideas.

In Both PRs, IIRC the data that I used was mock. So that visualizations can atleast populate. The visualizations needs the filter.query that you have set in this PR. For that GTG.

My suggestion is to also include dashboard level filter changes (data_stream.dataset: {{value}}) in this PR. I can see no dashboard level filters present for azure_metrics-1a151f80-32db-11ea-a83e-25b8612d00cc.json. If you can also add those, that would be great.

[gizas]

Seems that is working now! I agree with @harnish-elastic you should add to all visulaisations the data_stream.dataset : azure.storage_account filter (and to the other dashboard metrics the relevant filter as well)

There are eg. Success Server Latency, Success E2E Latency, Ingress Traffic by APIName that use the azure.dimensions.api_name grouping. Those can benefit from a filter azure.dimensions.api_name: * as well

And a minor one, I see in the title the Azure Storage being repeated as a Title under Nvigation titles. Either add them in one or you can remove it. Sth like Azure Storage - Navigation File Storage Overview (same in Overview, Blob, Table and Queue)

[zmoog]

I agree with @harnish-elastic you should add to all visulaisations the data_stream.dataset : azure.storage_account filter (and to the other dashboard metrics the relevant filter as well)

Added a filter to all dashboards to only include documents from data_stream.dataset: azure.storage_account.

CleanShot.2024-09-19.at.12.51.55.mp4

[zmoog]

There are eg. Success Server Latency, Success E2E Latency, Ingress Traffic by APIName that use the azure.dimensions.api_name grouping. Those can benefit from a filter azure.dimensions.api_name: * as well

I'm taking a look.

And a minor one, I see in the title the Azure Storage being repeated as a Title under Nvigation titles. Either add them in one or you can remove it. Sth like Azure Storage - Navigation File Storage Overview (same in Overview, Blob, Table and Queue)

@gizas, I'm not sure I getting this correctly: can you give me a quick example (maybe editing the page in the browser for the final result)?

[zmoog]

/test

[gizas]

It is really minor:

If you want you can edit the settings of visualisation:

And then remove in markdown the title, so as final result:

Maybe is cleaner

[gizas]

FYI the package build reports for missing filters:

elastic-package build
2024/09/19 15:24:16  INFO New version is available - v0.104.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.104.0
Build the package
...
2024/09/19 15:24:16  INFO Skipped errors: found 2 validation errors:
   1. file "/Users/andreasgkizas/elastic/integrations/build/packages/azure_metrics-1.6.6.zip/kibana/dashboard/azure_metrics-9c11ac60-6cf6-11ea-8fe8-71add5fd7c38.json" is invalid: expected filter in dashboard: no filter found and at least one panel does not have a filter (SVR00002)
   2. file "/Users/andreasgkizas/elastic/integrations/build/packages/azure_metrics-1.6.6.zip/kibana/dashboard/azure_metrics-dae20ed0-6d0a-11ea-8fe8-71add5fd7c38.json" is invalid: expected filter in dashboard: no filter found and at least one panel does not have a filter (SVR00002)
Package built: /Users/andreasgkizas/elastic/integrations/build/packages/azure_metrics-1.6.6.zip

[zmoog]

@gizas, here's my first attempt:

CleanShot.2024-09-19.at.16.08.08.mp4

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 51658a1

History

• 💚 Build #16154 succeeded 86d8851
• 💔 Build #16151 failed 65fbcc0
• 💔 Build #16148 failed 65fbcc0
• 💔 Build #16127 failed 9ea8779
• 💚 Build #15883 succeeded cad74bc
• 💚 Build #15877 succeeded 2015d25

cc @zmoog

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
99.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[zmoog]

FYI the package build reports for missing filters:

Oh, you're right.

These are dashboards for the Azure Container Instance and Container Service integration metrics. I'll open new issues to fix them.

[zmoog]

Final recording before merging the PR:

CleanShot.2024-09-20.at.11.46.18.mp4

[elastic-vault-github-plugin-prod]

Package azure_metrics - 1.6.6 containing this change is available at https://epr.elastic.co/search?package=azure_metrics