map Azure logs into OpenTelemetry fields/attributes

[loomis-relativity]

Description: This change allows users of the Azure Event Hub receiver to choose between two representations of their Azure resource/service logs in OpenTelemetry. The "raw" format (existing behavior) maps the AMQP message properties and data to the OpenTelemetry attributes and body (as bytes), respectively. The new "azure" format parses the AMQP message data and maps the Azure structured log into OpenTelemetry attributes; the body of the LogRecords are empty.

Link to tracking Issue: #16283

Testing: Testing done with Azure Key Vault audit logs and logs from the Azure CDN. The payload appears to be correct compared to the desired mapping.

Documentation: The README has been updated to contain the details of the log formats.

[atoulme]

Something is wrong with the code changes. It looks like most changes are reverted, please take a look.

[loomis-relativity]

Something is wrong with the code changes. It looks like most changes are reverted, please take a look.

@atoulme Can you provide more details here? Looking at the changed files, everything looks as I'd expect. Note that I've not pulled in upstream changes since I forked the repository for the PR. I'll do that before taking the PR out of draft status.

[loomis-relativity]

@atoulme @djaglowski

I've tested the changes by pulling this code into our customized collector and running it locally. The test pulls logs from an event hub with this receiver and formats them with the "data" encoding. These logs look like the following in JSON:

{
   "resourceLogs":[
      {
         "resource":{
            
         },
         "scopeLogs":[
            {
               "scope":{
                  
               },
               "logRecords":[
                  {
                     "timeUnixNano":"1669125795416887400",
                     "body":{
                        
                     },
                     "attributes":[
                        {
                           "key":"azure.resource.id",
                           "value":{
                              "stringValue":"/SUBSCRIPTIONS/..."
                           }
                        },
                        {
                           "key":"azure.operation.version",
                           "value":{
                              "stringValue":"2016-10-01"
                           }
                        },
                        {
                           "key":"azure.result.type",
                           "value":{
                              "stringValue":"Success"
                           }
                        },
                        {
                           "key":"cloud.provider",
                           "value":{
                              "stringValue":"azure"
                           }
                        },
                        {
                           "key":"azure.duration",
                           "value":{
                              "intValue":"19"
                           }
                        },
                        {
                           "key":"azure.identity",
                           "value":{
                              "kvlistValue":{
                                 "values":[
                                    {
                                       "key":"claim",
                                       "value":{
                                          "kvlistValue":{
                                             "values":[
                                                {
                                                   "key":"appid",
                                                   "value":{
                                                      "stringValue":"a4a940d4-..."
                                                   }
                                                },
                                                {
                                                   "key":"appidacr",
                                                   "value":{
                                                      "stringValue":"1"
                                                   }
                                                },
                                                {
                                                   "key":"oid",
                                                   "value":{
                                                      "stringValue":"b39ded2b-..."
                                                   }
                                                }
                                             ]
                                          }
                                       }
                                    }
                                 ]
                              }
                           }
                        },
                        {
                           "key":"azure.properties",
                           "value":{
                              "kvlistValue":{
                                 "values":[
                                    {
                                       "key":"id",
                                       "value":{
                                          "stringValue":"https://kv-global-..."
                                       }
                                    },
                                    {
                                       "key":"clientInfo",
                                       "value":{
                                          "stringValue":"VSTS_..."
                                       }
                                    },
                                    {
                                       "key":"httpStatusCode",
                                       "value":{
                                          "intValue":"200"
                                       }
                                    },
                                    {
                                       "key":"requestUri",
                                       "value":{
                                          "stringValue":"https://kv-global-..."
                                       }
                                    },
                                    {
                                       "key":"isAccessPolicyMatch",
                                       "value":{
                                          "boolValue":true
                                       }
                                    },
                                    {
                                       "key":"tlsVersion",
                                       "value":{
                                          "stringValue":"TLS1_2"
                                       }
                                    }
                                 ]
                              }
                           }
                        },
                        {
                           "key":"azure.operation.name",
                           "value":{
                              "stringValue":"SecretGet"
                           }
                        },
                        {
                           "key":"azure.category",
                           "value":{
                              "stringValue":"AuditEvent"
                           }
                        },
                        {
                           "key":"azure.result.signature",
                           "value":{
                              "stringValue":"OK"
                           }
                        },
                        {
                           "key":"net.sock.peer.addr",
                           "value":{
                              "stringValue":"40.124.26.59"
                           }
                        }
                     ],
                     "traceId":"d27f00faca5042029531b0a6d9a30e47",
                     "spanId":""
                  }
               ]
            }
         ]
      }
   ]
}

In New Relic, these logs have the following form. Note that New Relic does not support nested attributes. so those won't appear here without some flattening within the transport layer.

{
  "azure.category": "AuditEvent",
  "azure.duration": 16,
  "azure.operation.name": "SecretGet",
  "azure.operation.version": "7.3",
  "azure.resource.id": "/SUBSCRIPTIONS/...",
  "azure.result.signature": "OK",
  "azure.result.type": "Success",
  "cloud.provider": "azure",
  "instrumentation.provider": "opentelemetry",
  "net.sock.peer.addr": "52.165.224.153",
  "newrelic.source": "api.logs.otlp",
  "otel.library.name": "",
  "otel.library.version": "",
  "timestamp": 1669125534733,
  "trace.id": "67136fec6fe24230a9af44e61430a6e9"
}

So everything (or at least what's in the original log) looks to be mapped correctly.

This brings up a few questions before trying to finalize the code:

• Should the scope/library name/version be set to something? If so, what values should be used?
• In the end, I did not separate resource/log-level attributes, using only log-level ones. Although my test case has only one service reporting to the event hub, in principle there could be multiple services and hence, multiple resource IDs. The general case would require logic to ensure that only logs with the same resource IDs are combined. Is this worth it?
• Similarly, I did not add any flattening to this receiver. I think that that's better put into one of the processors. Is that OK or should I add the ability to flatten attributes?
• I'm not overly thrilled with the names "encoding", "raw", and "data". Are there better alternatives?
• Just for planning purposes, do you think this is far enough along to target the v0.66.0 release?
• Can someone authorize the PR checks to run? I'd like to verify that they work before moving the PR out of draft status.

[atoulme]

Something is wrong with the code changes. It looks like most changes are reverted, please take a look.

@atoulme Can you provide more details here? Looking at the changed files, everything looks as I'd expect. Note that I've not pulled in upstream changes since I forked the repository for the PR. I'll do that before taking the PR out of draft status.

I guess I don't see problems anymore. I did a review, I hope it helps.

[loomis-relativity]

@atoulme @djaglowski All comments have been addressed. Let me know if more changes are needed.

[atoulme]

I'm good here, you'll need a passing build. I'll ask around for a maintainer to approve a run for this PR.

[loomis-relativity]

@atoulme Looks like the checks need to be approved after every change. 😞

[atoulme]

Yes - make sure to run the following:

make gotidy
make lint
gofmt -w -s .
make impi

in the azureventhubreceiver folder to cover all the linting and common issues we see. That will help some.

[loomis-relativity]

All those checks pass. They were corrected in the last commit. 🤞

[MovieStoreGuy]

Just some comments on how best to get this through and avoid some potential gotchas.

[runforesight]

Foresight Summary

    
Major Impacts

build-and-test duration(43 minutes 29 seconds) has decreased 11 minutes 11 seconds compared to main branch avg(54 minutes 40 seconds).

View More Details

✅  changelog workflow has finished in 1 minute 56 seconds (3 minutes 20 seconds less than main branch avg.) and finished at 1st Dec, 2022.

---

Job	Failed Steps	Tests
changelog	-     🔗	N/A	See Details

---

✅  tracegen workflow has finished in 2 minutes 4 seconds and finished at 1st Dec, 2022.

---

Job	Failed Steps	Tests
publish-latest	-     🔗	N/A	See Details
build-dev	-     🔗	N/A	See Details
publish-stable	-     🔗	N/A	See Details

---

✅  check-links workflow has finished in 2 minutes 9 seconds and finished at 1st Dec, 2022.

---

Job	Failed Steps	Tests
changed files	-     🔗	N/A	See Details
check-links	-     🔗	N/A	See Details

---

✅  build-and-test workflow has finished in 43 minutes 29 seconds (11 minutes 11 seconds less than main branch avg.) and finished at 1st Dec, 2022.

---

Job	Failed Steps	Tests
correctness-metrics	-     🔗	✅ 0  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.19, internal)	-     🔗	✅ 591  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.18, internal)	-     🔗	✅ 591  ❌ 0  ⏭ 0    🔗	See Details
correctness-traces	-     🔗	✅ 0  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.18, extension)	-     🔗	✅ 525  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.19, processor)	-     🔗	✅ 1,462  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.19, extension)	-     🔗	✅ 525  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.18, processor)	-     🔗	✅ 1,462  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.18, receiver-0)	-     🔗	✅ 2,528  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.19, receiver-0)	-     🔗	✅ 2,527  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.18, other)	-     🔗	✅ 4,342  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.19, other)	-     🔗	✅ 4,335  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.18, exporter)	-     🔗	✅ 2,410  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.18, receiver-1)	-     🔗	✅ 1,832  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.19, exporter)	-     🔗	✅ 2,410  ❌ 0  ⏭ 0    🔗	See Details
unittest-matrix (1.19, receiver-1)	-     🔗	✅ 1,832  ❌ 0  ⏭ 0    🔗	See Details
integration-tests	-     🔗	✅ 59  ❌ 0  ⏭ 0    🔗	See Details
setup-environment	-     🔗	N/A	See Details
check-collector-module-version	-     🔗	N/A	See Details
checks	-     🔗	N/A	See Details
check-codeowners	-     🔗	N/A	See Details
lint-matrix (receiver-0)	-     🔗	N/A	See Details
lint-matrix (receiver-1)	-     🔗	N/A	See Details
lint-matrix (processor)	-     🔗	N/A	See Details
lint-matrix (exporter)	-     🔗	N/A	See Details
lint-matrix (extension)	-     🔗	N/A	See Details
lint-matrix (internal)	-     🔗	N/A	See Details
lint-matrix (other)	-     🔗	N/A	See Details
build-examples	-     🔗	N/A	See Details
lint	-     🔗	N/A	See Details
unittest (1.19)	-     🔗	N/A	See Details
unittest (1.18)	-     🔗	N/A	See Details
cross-compile (darwin, amd64)	-     🔗	N/A	See Details
cross-compile (darwin, arm64)	-     🔗	N/A	See Details
cross-compile (linux, 386)	-     🔗	N/A	See Details
cross-compile (linux, amd64)	-     🔗	N/A	See Details
cross-compile (linux, arm)	-     🔗	N/A	See Details
cross-compile (linux, arm64)	-     🔗	N/A	See Details
cross-compile (linux, ppc64le)	-     🔗	N/A	See Details
cross-compile (windows, 386)	-     🔗	N/A	See Details
cross-compile (windows, amd64)	-     🔗	N/A	See Details
build-package (deb)	-     🔗	N/A	See Details
build-package (rpm)	-     🔗	N/A	See Details
windows-msi	-     🔗	N/A	See Details
publish-check	-     🔗	N/A	See Details
publish-stable	-     🔗	N/A	See Details
publish-dev	-     🔗	N/A	See Details

---

✅  load-tests workflow has finished in 12 minutes 41 seconds and finished at 1st Dec, 2022.

---

Job	Failed Steps	Tests
loadtest (TestIdleMode)	-     🔗	✅ 1  ❌ 0  ⏭ 0    🔗	See Details
loadtest (TestTraceAttributesProcessor)	-     🔗	✅ 3  ❌ 0  ⏭ 0    🔗	See Details
loadtest (TestMetric10kDPS	TestMetricsFromFile)	-     🔗	✅ 6  ❌ 0  ⏭ 0    🔗
loadtest (TestMetricResourceProcessor	TestTrace10kSPS)	-     🔗	✅ 12  ❌ 0  ⏭ 0    🔗
loadtest (TestTraceNoBackend10kSPS	TestTrace1kSPSWithAttrs)	-     🔗	✅ 8  ❌ 0  ⏭ 0    🔗
loadtest (TestTraceBallast1kSPSWithAttrs	TestTraceBallast1kSPSAddAttrs)	-     🔗	✅ 10  ❌ 0  ⏭ 0    🔗
loadtest (TestBallastMemory	TestLog10kDPS)	-     🔗	✅ 19  ❌ 0  ⏭ 0    🔗
setup-environment	-     🔗	N/A	See Details

---

✅  prometheus-compliance-tests workflow has finished in 13 minutes 3 seconds (⚠️ 4 minutes 55 seconds more than main branch avg.) and finished at 1st Dec, 2022.

---

Job	Failed Steps	Tests
prometheus-compliance-tests	-     🔗	✅ 21  ❌ 0  ⏭ 0    🔗	See Details

---

🔎 See details on Foresight

^{*You can configure Foresight comments in your organization settings page.}