[azure] update routing for signinlogs and springcloudlogs

[mmahacek]

Proposed commit message

Fix for routing Azure signinlogs from Azure AD.

The "Windows Azure Active Directory" resource generates logs with the category MicrosoftServicePrincipalSignInLogs. When using the Azure Events v2 stream, these are handled as platform logs instead of as signinLogs. This PR adds routing to flag these events as the correct dataset.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Ingest data with a {"category":"MicrosoftServicePrincipalSignInLogs"} and verify the document ends up in the signinlogs dataset.

Related issues

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[zmoog]

Hey @mmahacek, thanks for contributing this update!

I wonder if we should consider sing-in logs all *SignInLogs log categories, @kaiyan-sheng WDYT?

[mmahacek]

The signinlogs pipeline does drop any non "*signinlogs" before processing, so this does make some sense.

https://github.com/elastic/integrations/blob/main/packages/azure/data_stream/signinlogs/elasticsearch/ingest_pipeline/default.yml#L46-L48

[mmahacek]

While I'm updating routing rules, I notice that the router for springcloudlogs has a few different values:

integrations/packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml

Lines 106 to 110 in 80b9687

	- set:
	field: event.dataset
	value: azure.springcloudlogs
	if: 'ctx.tmp_json?.category == "ApplicationConsole" || ctx.tmp_json?.category == "SystemLogs" || ctx.tmp_json?.category == "IngressLogs" || ctx.tmp_json?.category == "BuildLogs" || ctx.tmp_json?.category == "ContainerEventLogs"'
	description: 'Azure Spring Apps log categories (refs: https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-logs/microsoft-appplatform-spring-logs)'

But the pipeline for that dataset only keeps two of the values:

integrations/packages/azure/data_stream/springcloudlogs/elasticsearch/ingest_pipeline/default.yml

Lines 53 to 55 in 80b9687

	- drop:
	if: "ctx.azure.springcloudlogs.category != 'SystemLogs' && ctx.azure.springcloudlogs.category != 'ApplicationConsole'"
	ignore_failure: true

Should this be updated too?

[kaiyan-sheng]

I wonder if we should consider sing-in logs all *SignInLogs log categories, @kaiyan-sheng WDYT?

@zmoog Yeah I think that's probably safer for whatever new signin logs Azure is going to add!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9983786

History

• 💚 Build #33479 succeeded bbe285e
• 💔 Build #33471 failed 791b638
• 💚 Build #33390 succeeded 65037d9

cc @mmahacek

[efd6]

Remove?