[azure logs] use one input + routing for all data streams (WIP)

[zmoog]

Proposed commit message

Switch the integration package from the one-input-per-data-stream model to the one-input model.

One input per data stream model:

One input model:

In the one-input model, there is only one azure-eventhub input running and sending events to the events data stream. In the events data stream, the ingest pipeline performs these tasks:

• discover and set the event.dataset field using the category field in the event.
• use the event.dataset field to reroute the event to the target data stream.

The discover process uses the following logic:

• if the event doesn't have a category, it sets event.dataset to azure.eventhub (the generic integration)
• if the event does have a category, it sets event.dataset to azure.platformlogs (it's probably an Azure log)
• if the event category is supported, it sets event.dataset to specific one like azure.activitylogs or azure.signinlogs.

After the discovery step, the routing rules use the event.dataset value to forward the events to the best available target data stream.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes Azure Logs: use one input per agent policy #11251

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package azure 👍(5) 💚(5) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
provisioning	3558.72	2288.33	-1270.39 (-35.7%)	💔
platformlogs	5917.16	4524.89	-1392.27 (-23.53%)	💔

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 533db9d

History

• 💚 Build #17216 succeeded 63ffd49
• 💔 Build #17215 failed 75677ca

cc @zmoog

[elastic-sonarqube]

Quality Gate failed

Failed conditions
26.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[zmoog]

Here's the latest iteration for the integration settings to evidence the recommended single data stream (v2) vs. the legacy multiple data stream (v1) solutions: