[azure_logs] Add Custom Azure Logs input package

[zmoog]

Proposed commit message

Add a new input package to collect custom logs from Event Hub.

This input package is a thin wrapper on the azure-eventhub input that allows users to receive JSON log events from an event hub and transform them using custom pipelines and mappings.

Users should use this input package instead of generic integration in the Azure Logs integration.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• [ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Add pipeline override
• Replace the logo with the one used in all input packages
• Write the docs (overview, requirements, setup)
• How to deal with routing

Reviewer's Checklist

Please review the following items and, ideally, leave your thoughts in the comments.

• Routing: I could not find a way to add an OOTB pipeline to route logs to other data streams. Do you know a method?
• Logo: all the input packages use the same logo. Should we adopt the same logo or use something Azure-specific?
• Data model: Currently, the input package stores the log events sent by the input without any additional change. The log event contains the JSON document as a string in the message field. This gives users all the options to transform the log event in the custom pipeline. Do you have any suggestions?
• Ownership: I currently set the package ownership to elastic/obs-ds-hosted-services. Should we set the ownership to elastic/obs-infraobs-integrations?

Related issues

• Relates Azure Event Hub Input Package #7821

Screenshots

[zmoog]

cc @lalit-satapathy for the ownership topic in the reviewer's checklist

[zmoog]

Questions,

• How do we make a shift from the Event Hub input integration?
• Are we going to keep both or deprecate the Azure EventHub input integration?

My current proposal is:

1. Ship this input package.
2. Publish a blog post that explains how the input package works and why it's better than the Azure EventHub input integration (the generic integration in Azure Logs).
3. Deprecate the Azure EventHub input integration in Azure Logs, linking to the input package and blog post.

Then, move to Azure Logs and update it to use only one input.

WDYT?

[zmoog]

There is a full round of updates, and the PR is ready for another round of review.

• @muthu-mps, @Kavindu-Dodan, please take another look at the latest changes and see if they address your comments! 🙇
• @alaudazzi, could you take a look at the integration docs in the README file? 🙇

[muthu-mps]

Questions,

• How do we make a shift from the Event Hub input integration?
• Are we going to keep both or deprecate the Azure EventHub input integration?

My current proposal is:

1. Ship this input package.
2. Publish a blog post that explains how the input package works and why it's better than the Azure EventHub input integration (the generic integration in Azure Logs).
3. Deprecate the Azure EventHub input integration in Azure Logs, linking to the input package and blog post.

Then, move to Azure Logs and update it to use only one input.

WDYT?

One more point to add here, The Azure Eventhub input can get deprecated once the input package is moved to GA.

[kaiyan-sheng]

since this is a new package, should we use the latest version of the stack?

[zmoog]

Since we are not offering input v2 support in this version, I thought we could:

• support 8.13+ users during the technical preview/beta phase (0.x)
• ship a GA version (1.0) shortly after
• Then, ship an update (1.1) requiring 8.15+ with input v2 support.

WDYT?

[zmoog]

Questions,

• How do we make a shift from the Event Hub input integration?
• Are we going to keep both or deprecate the Azure EventHub input integration?

My current proposal is:

1. Ship this input package.
2. Publish a blog post that explains how the input package works and why it's better than the Azure EventHub input integration (the generic integration in Azure Logs).
3. Deprecate the Azure EventHub input integration in Azure Logs, linking to the input package and blog post.

Then, move to Azure Logs and update it to use only one input.
WDYT?

One more point to add here, The Azure Eventhub input can get deprecated once the input package is moved to GA.

Yep, I agree.

[muthu-mps]

@zmoog - Do you think we need to use the build.yml file to add ECS reference? Not sure if this is not the case for the newer integrations after @ecsmappings changes.

[alaudazzi]

I left some editing suggestions, otherwise the text LGTM.

[alaudazzi]

Is Custom Logs a name for general integrations?

[zmoog]

No, in this case, I am referring to integration or input packages designed to collect custom logs. With "custom logs", I mean any log format (usually JSON document) that may come from any source.

Maybe I should focus on the Custom Azure Logs integration only and change this phrase into:

The Custom Azure Logs integration gives you all the flexibility you need to collect, parse, and map log events to your needs.

WDYT?

[alaudazzi]

@zmoog
A more general comment about capitalization regarding these terms:

• Consumer group
• Agents
• event hub

Are they capitalized when they refer to something specific and not capitalized when referring to something generic? You might want to double-check when capitalization applies.

[zmoog]

A more general comment about capitalization regarding these terms:

• Consumer group
• Agents
• event hub

Are they capitalized when they refer to something specific and not capitalized when referring to something generic? You might want to double-check when capitalization applies.

I'll review the doc to see if I am not applying the rules consistently, but here are the rules I'm using now (please highlight any wrong usage!)

Consumer group: It's a generic term, so I capitalize "Consumer" only if required by the context (start of a sentence or heading).

Agents: I capitalize it when it means "Elastic Agent" and I leave it lowercase when it's the generic term. Should I use the full name ("Elastic Agent(s)") to avoid ambiguity?

event hub: see the comment #11552 (comment)

[zmoog]

@zmoog - Do you think we need to use the build.yml file to add ECS reference? Not sure if this is not the case for the newer integrations after @ecsmappings changes.

That's a good point. I missed the build.yml file. I'm double-checking, but I should probably include it in this integration to state which version of ECS the integration fields should be tested with.

[elastic-sonarqube]

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 056f44f

History

• 💔 Build #18010 failed 985e4d1
• 💔 Build #18009 failed d7c0c87
• 💚 Build #17963 succeeded 98ddfe0
• 💚 Build #17942 succeeded eca52aa
• 💔 Build #17941 failed 1bce5ab

cc @zmoog

[muthu-mps]

LGTM!

The top level fields like resource_id can be parsed from the message field. This is expected to be available in all the events. This can be taken in next release.

[zmoog]

The top level fields like resource_id can be parsed from the message field. This is expected to be available in all the events. This can be taken in next release.

I didn't want to make any assumptions about the message structure. From public issues, I learned that users use the generic integration to ingest non-Azure log payloads like IoT data or third-party application logs (for example, MuleSoft and ConfluentCloud logs).

But we can, of course, revisit this and add standard Azure field mappings.

[zmoog]

@muthu-mps, do you confirm your team elastic/obs-infraobs-integrations is okay with taking ownership of this input?