Skip to content

Latest commit

 

History

History
312 lines (248 loc) · 10.1 KB

README.md

File metadata and controls

312 lines (248 loc) · 10.1 KB
Raw

Sumo Logic Schema Processor

Stability level: Beta

The Sumo Logic Schema processor (config name: sumologic_schema) modifies the metadata on logs, metrics and traces sent to Sumo Logic so that the Sumo Logic apps can make full use of the ingested data.

Supported pipeline types: logs, metrics, traces.

Configuration

processors:
  sumologic_schema:
    # Defines whether the `cloud.namespace` resource attribute should be added.
    # default = true
    add_cloud_namespace: {true,false}

    # Defines whether attributes should be translated
    # from OpenTelemetry to Sumo Logic conventions;
    # see "Attribute translation" documentation chapter from this document.
    # default = true
    translate_attributes: {true,false}

    # Specifies whether telegraf metric names should be translated to match
    # Sumo Logic conventions expected in Sumo Logic host related apps (for example
    # `procstat_num_threads` => `Proc_Threads` or `cpu_usage_irq` => `CPU_Irq`).
    # See `translate_telegraf_metrics_processor.go` for full list of translations.
    # default = true
    translate_telegraf_attributes: {true, false}

    # Specifies whether docker stats metric names should be translated to match
    # Sumo Logic conventions expected in Sumo Logic host related apps (for example
    # `container.cpu.usage.kernelmode` => `cpu_usage.usage_in_kernelmode` or `container.memory.usage.max` => `max_usage`).
    # It also translates some resource attribute names (for example `container.id` => `container.FullID`).
    # See `translate_docker_metrics_processor.go` for full list of translations.
    # default = false
    translate_docker_metrics: {true, false}

    # Specifies if attributes should be nested, basing on their keys.
    # See "Nesting attributes" documentation chapter from this document.
    nest_attributes:
      # Defines whether attributes should be nested.
      # default = false
      enabled: {true, false}

      # Defines the string used to separate key names in attributes that are to be nested.
      # default = "."
      separator: <separator>

      # Defines a list of allowed prefixes to be nested.
      # For example, if "kubernetes." is in this list, then all "kubernetes.*" attributes will be nested.
      # default = []
      include: [<prefix>]

      # Defines a list of prefixes not allowed to be nested.
      # For example, if "k8s." is in this list, then all "k8s.*" attributes will not be nested.
      # default = []
      exclude: [<prefix>]

      # If enabled, then maps that would have only one value will be squashed.
      # For example,{"k8s": {"pods": {"a": "A", "b": "B"}}}
      # will be squashed to {"k8s.pods": {"a": "A", "b": "B"}}
      # default = false
      squash_single_values: {true, false}

    # Specifies if attributes matching given pattern should be mapped to a common key.
    # See "Aggregating attributes" documentation chapter from this document.
    # default = []
    aggregate_attributes:
      - attribute: <attribute>
        prefixes: [<prefix>]
      - ...

    # Defines whether attribute should be added to record attributes.
    # Currently available attributes are `severity_number`, `severity_text`,
    # `trace_id`, `span_id`
    # all are disabled by default, enabled: false
    field_attributes:
      severity_number:
        enabled: true
        name: "loglevel"

Features

Adding cloud.namespace resource attribute

Some of the apps in Sumo Logic require the cloud.namespace attribute to be set to better understand the data coming from AWS EC2, AWS ECS and AWS Elactic Beanstalk. This attribute is similar to the standard OpenTelemetry attribute cloud.provider. In the future, the Sumo Logic apps might switch to the standard cloud.provider attribute. Before this happens, the following mapping defines the relationship between cloud.provider and cloud.namespace values:

cloud.platform cloud.namespace
aws_ec2 aws/ec2
aws_ecs ecs
aws_elastic_beanstalk ElasticBeanstalk

When this processor's add_cloud_namespace setting is set to true, the processor looks for the above mentioned cloud.platform resource attribute values and if found, adds the corresponding cloud.namespace resource attribute.

If the cloud.platform resource attribute is not found or has a value that is not in the table, nothing is added.

Attribute translation

Attribute translation changes some of the attribute keys from OpenTelemetry convention to Sumo Logic convention. For example, OpenTelemetry convention for the attribute containing Kubernetes pod name is k8s.pod.name, but Sumo Logic expects it to be in attribute named pod.

If attribute with target name eg. pod already exists, translation is not being done for corresponding attribute (k8s.pod.name in this example).

This feature is turned on by default. To turn it off, set the translate_attributes configuration option to false. Note that this may cause some of Sumo Logic apps, built-in dashboards to not work correctly.

Note: the attributes are not translated for traces.

Below is a list of all attribute keys that are being translated.

OTC key name Sumo Logic key name
cloud.account.id AccountId
cloud.availability_zone AvailabilityZone
cloud.platform aws_service
cloud.region Region
host.id InstanceId
host.name host
host.type InstanceType
k8s.cluster.name Cluster
k8s.container.name container
k8s.daemonset.name daemonset
k8s.deployment.name deployment
k8s.namespace.name namespace
k8s.node.name node
k8s.service.name service
k8s.pod.hostname host
k8s.pod.name pod
k8s.pod.uid pod_id
k8s.replicaset.name replicaset
k8s.statefulset.name statefulset
service.name service
log.file.path_resolved _sourceName

Nesting attributes

Nesting attributes allows to change the structure of attributes (both resource level and record level attributes) basing on their keys by nesting attributes with common paths into maps. Common path is defined as a common prefix that consists of strings separated by a given separator string (by default a dot - .) and end with that separator, for example: if the separator is ., then xyz is a common prefix path for xyz.abc.qwe and xyz.foo, but sumo is not a prefix path for neither sumologic.foo.baz nor sumologic.bar.baz.

Example

The following set of attributes:

{
  "kubernetes.container_name": "xyz",
  "kubernetes.host.name": "the host",
  "kubernetes.host.address": "127.0.0.1",
  "kubernetes.namespace_name": "sumologic",
  "another_attr": "42"
}

Should be translated into such set:

{
  "kubernetes": {
    "container_name": "xyz",
    "namespace_name": "sumologic",
    "host": {
      "name": "the host",
      "address": "127.0.0.1"
    }
  },
 "another_attr": "42"
}

Allowlist and denylist

Properties include and exclude in the config allow to define list of prefixes that are allowed or not allowed to be nested.

For example, with the following config:

nest_attributes:
  enabled: true
  include: ["kubernetes.host."]
  exclude: ["kubernetes.host.naming"]

and following input:

{
"kubernetes.container_name": "xyz",
"kubernetes.host.name": "the host",
"kubernetes.host.naming_convention": "random",
"kubernetes.host.address": "127.0.0.1",
"kubernetes.namespace_name": "sumologic",
}

The output is:

{
"kubernetes.container_name": "xyz",
"kubernetes": {
  "host": {
    "name": "the host",
    "address": "127.0.0.1"
    }
  },
"kubernetes.host.naming_convention": "random",
"kubernetes.namespace_name": "sumologic",
}

Known issues

This feature has undefined behavior when in input there are various keys that will be mapped to the same nested structure. For example, given the following attributes:

{
  "a.b.c.": "d",
  "a": {
    "b": {
      "c": "e"
    }
  }
}

It is not possible to predict, what will be the result. It will have the following structure:

{
  "a": {
    "b": {
      "c": ...
    }
  }
}

However, it is not possible to know a priori if the value under key c will be equal to d or e.

Aggregating attributes

Aggregating attributes allows to map attributes with keys with given prefixes to a common key. For example, it is possible to map all attributes with keys with prefixes pod_ to a key pods. The names in resulting maps are the original key names with trimmed prefix.

For given input, mapping keys with prefixes pod_ to a key pods:

{
  "pod_a": "x",
  "pod_b": "y",
  "pod_c": "z"
}

The result is:

{
  "pods": {
    "a": "x",
    "b": "y",
    "c": "z"
  }
}

Severity Attribute

Some fields that log entries consist of are not displayed as fields in Sumo Logic out of the box. It's possible to convert specific attributes to fields. List of currently supported attributes is below:

severity_number severity_text span_id trace_id

In order to report one of them as field, following configuration is needed (below example for severity_number):

  field_attributes:
      severity_number:
        enabled: true
        name: "loglevel"

In this case severity_number from log record will be visible in sumo backend as a field named "loglevel".