Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/elasticsearch] Support preserving attributes when mapping to ECS #33670

Merged
merged 5 commits into from
Jun 21, 2024

Conversation

ChrsMark
Copy link
Member

Description:

This PR adds support for preserving resource attributes that are valid ECS fields in addition of mapping them. At the moment the host.name is mapped to the host.hostname. Both are valid ECS fields but can differ in some cases. Since there is no such distinction in SemConv right now, it does makes sense to preserve both.

refs:

Link to tracking Issue:

Testing:

Using the testing notes from #33622.

Stored document:

 {
    "app": {
      "label": {
        "component": "migration-logger"
      }
    },
    "kubernetes": {
      "node": {
        "name": "kind-control-plane"
      },
      "pod": {
        "uid": "0eda57cd-a4ae-4e89-88fa-c771d3bf0c77",
        "name": "daemonset-logs-4sqjq"
      },
      "namespace": "default"
    },
    "agent": {
      "name": "otlp"
    },
    "@timestamp": "2024-06-20T07:27:42.589678923Z",
    "log": {
      "iostream": "stdout",
      "file": {
        "path": "/var/log/pods/default_daemonset-logs-4sqjq_0eda57cd-a4ae-4e89-88fa-c771d3bf0c77/busybox/0.log"
      }
    },
    "service": {
      "name": "migration-logger"
    },
    "k8s": {
      "container": {
        "restart_count": "0",
        "name": "busybox"
      },
      "pod": {
        "start_time": "2024-06-20T07:27:21Z"
      },
      "daemonset": {
        "name": "daemonset-logs"
      }
    },
    "host": {
      "hostname": "daemonset-opentelemetry-collector-agent-l6pzp",
      "os": {
        "type": "linux",
        "platform": "linux"
      },
      "name": "daemonset-opentelemetry-collector-agent-l6pzp"
    },
    "time": "2024-06-20T07:27:42.589678923Z",
    "message": "otel logs at 07:27:42",
    "logtag": "F"
  }

Documentation: ~

/cc @lahsivjar @andrzej-stencel @carsonip

…o ECS

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
Copy link
Member

@gregkalapos gregkalapos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More context on the history of these two fields can be seen in elastic/ecs#144 (comment).

Since there is no such distinction in SemConv right now, it does makes sense to preserve both.

➕ on this.
SemConv says: "...On Unix ... contain what hostname ... returns, or the fully qualified hostname, or another name specified by the user." -> In ECS user specified would be host.name, otherwise either host.name or host.hostname. But we don't know from OTel data if it's user specified or not, so I agree, best is to populate both.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
Copy link
Member

@lahsivjar lahsivjar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (one comment on the changelog).

.chloggen/es_preserve_host_name.yaml Outdated Show resolved Hide resolved
Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
@andrzej-stencel andrzej-stencel merged commit 86278d9 into open-telemetry:main Jun 21, 2024
154 checks passed
@github-actions github-actions bot added this to the next release milestone Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants