[opentelemetry input plugin] grpc: received message larger than max

[rainbowwiki]

Use Case

client send opentelemetry to telegraf

Expected behavior

telegraf can receive the opentelemetry metrics

Actual behavior

Thu, 25 Apr 2024 12:48:54 GMT - severe: [telemetry.OpenTelemetry_Exporter.f5telemetry_default::My_Consumer] error: Service request
Thu, 25 Apr 2024 12:48:54 GMT - severe: [telemetry.OpenTelemetry_Exporter.f5telemetry_default::My_Consumer] Error on attempt to send metrics:
Traceback:
Error: 8 RESOURCE_EXHAUSTED: grpc: received message larger than max (7560819 vs. 4194304)

Additional info

How can I adjust the max size for opentelemetry plugin?

[powersj]

Hi,

Those logs do not look like Telegraf logs, but instead the logs from your exporter service?

Can you please share your telegraf logs and config please.

[rainbowwiki]

the telegraf side no log for this issue,
I go through the opentelemetry input plugin, no config for increase the max size

[powersj]

Please share your logs and config, you can omit any IP addresses or secrets.

[rainbowwiki]

[global_tags]


[agent]
  interval = "10s"
  round_interval = true

  metric_batch_size = 20480

  metric_buffer_limit = 2000000

  collection_jitter = "0s"

  flush_interval = "20s"
  flush_jitter = "0s"

  precision = ""

  debug = true
  quiet = false

  logtarget = "file"

  logfile = "/var/log/telegraf/telegraf.log"

  logfile_rotation_interval = "0d"

  logfile_rotation_max_size = "10MB"

  logfile_rotation_max_archives = 5

  hostname = ""
  omit_hostname = false


[[outputs.file]]
   files = ["stdout", "/tmp/metrics.out"]
   rotation_interval = "10m"
   rotation_max_archives = 10
   rotation_max_size = "100MB"
   data_format = "json"



[[outputs.health]]


[[outputs.kafka]]
  brokers = ["xxxxx"]
  topic = "zeus-source-f5"
  data_format = "json"
  json_timestamp_units = "1ms"
  
  
# Receive OpenTelemetry traces, metrics, and logs over gRPC
[[inputs.opentelemetry]]
  ## Override the default (0.0.0.0:4317) destination OpenTelemetry gRPC service
  ## address:port
  service_address = "0.0.0.0:4317"

  ## Override the default (5s) new connection timeout
  timeout = "30s"

  ## Override the default (prometheus-v1) metrics schema.
  ## Supports: "prometheus-v1", "prometheus-v2"
  ## For more information about the alternatives, read the Prometheus input
  ## plugin notes.
  metrics_schema = "prometheus-v1"

  ## Optional TLS Config.
  ## For advanced options: https://github.com/influxdata/telegraf/blob/v1.18.3/docs/TLS.md
  ##
  ## Set one or more allowed client CA certificate file names to
  ## enable mutually authenticated TLS connections.
  # tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]
  ## Add service certificate and key.
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"

[rainbowwiki]

this is my config, telegraf side no log

[powersj]

Thanks for the config

telegraf side no log

Your config has logs sent here:

logtarget = "file"
logfile = "/var/log/telegraf/telegraf.log"

I have also put up #15231 which introduces a new option called max_msg_size. In 30mins after this message, artifacts should be attached to that PR that you can download and try. Please let me know if setting that value to something like "8MB" or higher helps out.

[rainbowwiki]

thanks for your support! will test and give you feedback~

[powersj]

@rainbowwiki have you had a chance to give the PR a try? Thanks!

[rainbowwiki]

@powersj Hi man, sorry for the message delay, we tryed and work now, thanks!

[powersj]

Thanks for confirming!