Logs Transform Processor - unknown type: "logstransform"

[Jessimon]

Component(s)

processor/logstransform

What happened?

Hello Everyone,

I receive logs where the body of the logs are in json format.
I would like to see them as json.

I added the following in my config.
processors: logstransform: operators: - type: json_parser parse_from: body

But when starting i get the following error:
otel-collector_1 | * error decoding 'processors': unknown type: "logstransform" for id: "logstransform" (valid values: [servicegraph span tail_sampling k8sattributes experimental_metricsgeneration metricstransform probabilistic_sampler transform cumulativetodelta filter remoteobserver resourcedetection resource batch memory_limiter attributes datadog routing spanmetrics deltatorate groupbyattrs groupbytrace redaction])

I used the following documentation:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/logstransformprocessor/README.md
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/README.md
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/json_parser.md

Is the config wrong?

Thanks for the help.

Collector version

0.88

Environment information

Environment

OTLP is running in a docker on a Debian 11 server.

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4319

  awscloudwatch:
    region: *************
    logs:
      poll_interval: 1m

exporters:
  loki:
    endpoint: http://loki-test:3100/loki/api/v1/push
    tls:
      insecure: true

processors:
  logstransform:
    operators:
      - type: json_parser
        parse_from: body

service:
  pipelines:
    logs:
      receivers: [awscloudwatch]
      processors: [logstransform]
      exporters: [loki]

Log output

otel-collector_1  | 2023/11/13 11:18:27 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
otel-collector_1  |
otel-collector_1  | * error decoding 'processors': unknown type: "logstransform" for id: "logstransform" (valid values: [servicegraph span tail_sampling k8sattributes experimental_metricsgeneration metricstransform probabilistic_sampler transform cumulativetodelta filter remoteobserver resourcedetection resource batch memory_limiter attributes datadog routing spanmetrics deltatorate groupbyattrs groupbytrace redaction])

Additional context

No response

[github-actions]

Pinging code owners:

• processor/logstransform: @djaglowski @dehaansa

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dehaansa]

It looks like you're using a build that doesn't have the processor configured for use. Where are you getting your collector or building your collector?

[Jessimon]

Heya dehaanse,

It comes from:
otel/opentelemetry-collector-contrib:0.88.0
https://hub.docker.com/r/otel/opentelemetry-collector-contrib

Kind regards

[crobert-1]

Hi @Jessimon, from the status table in the README it looks like the processor isn't included in the contrib distribution.

@dehaansa and @djaglowski: Do you think it's ready to include in contrib releases, or should it still purposefully be left out?

[dehaansa]

I'm not aware of any reasons to not include the processor in the contrib deliverable at this time.

[crobert-1]

Thanks @dehaansa! In that case, anyone's welcome to submit PRs to get the logstransform processor in a release.

The required steps are found in the adding a new component and releasing new components sections of the contributing documentation. Who ever is able to help complete this, please make sure all steps are complete for the component to be available in a future release.

[djaglowski]

This processor was developed as a stopgap solution until transform processor could reimplement the same functionality. It will be deprecated in the future and should not be included in the contrib distro.

[Jessimon]

What is the recommanded way to convert a log body to a json fromat?
The json structure is not fixed.

[djaglowski]

The transform processor should be able to handle this with the ParseJSON function.

[Jessimon]

Thanks @djaglowski
Would this be used in the processors:?
Do you have an example?
I looked at the documentation and from what i understand ParseJSON can not be used in a transform process.

[RoryCrispin]

@djaglowski - is there an issue somewhere tracking which functionality is known to be only possible in the logstransform processor?
For instance, it seems like the transform processor does not yet support stanza/recombine - but perhaps I am mistaken.
Thanks

[malhomaid]

It doesn't seem like transform processor supports regex parsing