Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka 2.11 consumer panic: runtime error #5082

Closed
akamac opened this issue Dec 4, 2018 · 3 comments
Closed

Kafka 2.11 consumer panic: runtime error #5082

akamac opened this issue Dec 4, 2018 · 3 comments
Labels
area/kafka bug unexpected problem or unintended behavior
Milestone
1.9.1

Comments

@akamac
Copy link

akamac commented Dec 4, 2018

Relevant telegraf.conf:

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  debug = true
  quiet = false
  logfile = ""

[[inputs.kafka_consumer]]
  brokers = ["kafka-vx.infra.local:9092"]
  topics = ['linux', 'hyperv']
  # client_id = "Telegraf"
  version = "2.0.0"
  consumer_group = "influxdb-vx-2"
  offset = "oldest"
  # max_undelivered_messages = 1000
  data_format = "influx"

[[aggregators.basicstats]]
  period = "30s"
  drop_original = false
  stats = ["mean"]
  [aggregators.basicstats.tags]
    aggr = "basicstats"

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "linux"
  skip_database_creation = false
  retention_policy = "1month"
  http_proxy = ""
  content_encoding = "gzip"
  influx_uint_support = false
  [outputs.influxdb.tagpass]
    kafka_topic = ["linux"]
  [outputs.influxdb.tagdrop]
    aggr = ["basicstats"]

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "linux"
  skip_database_creation = false
  retention_policy = "1year"
  http_proxy = ""
  content_encoding = "gzip"
  influx_uint_support = false
  [outputs.influxdb.tagpass]
    kafka_topic = ["linux"]
    aggr = ["basicstats"]

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "hyperv"
  skip_database_creation = false
  retention_policy = "1month"
  http_proxy = ""
  content_encoding = "gzip"
  influx_uint_support = false
  [outputs.influxdb.tagpass]
    kafka_topic = ["hyperv"]
  [outputs.influxdb.tagdrop]
    aggr = ["basicstats"]

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "hyperv"
  skip_database_creation = false
  retention_policy = "1year"
  http_proxy = ""
  content_encoding = "gzip"
  influx_uint_support = false
  [outputs.influxdb.tagpass]
    kafka_topic = ["hyperv"]
    aggr = ["basicstats"]

System info:

Ubuntu 18.04.1
Telegraf 1.9.0 (git: HEAD 2bf21c0)
confluent-kafka-2.11

Steps to reproduce:

  1. Fill Kafka with metrics via Telegraf Kafka output plugin.
  2. Try to consume metrics from Kafka cluster using Telegraf to store them in InfluxDB

Logs:

Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z I! Starting Telegraf 1.9.0
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z I! Loaded inputs: inputs.kafka_consumer
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z I! Loaded aggregators: aggregators.basicstats
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z I! Loaded processors:
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z I! Loaded outputs: influxdb influxdb influxdb influxdb
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z I! Tags enabled: host=influxdb-vx-2
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"influxdb-vx-2", Flush Interval:10s
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Connecting outputs
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Attempting connection to output: influxdb
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Successfully connected to output: influxdb
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Attempting connection to output: influxdb
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Successfully connected to output: influxdb
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Attempting connection to output: influxdb
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Successfully connected to output: influxdb
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Attempting connection to output: influxdb
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Successfully connected to output: influxdb
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z D! [agent] Starting service inputs
Dec 03 23:54:36 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:36Z I! Started the kafka consumer service, brokers: [kafka-vx.infra.local:9092], topics: [hyperv linux]
Dec 03 23:54:50 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:54:50Z D! [outputs.influxdb] buffer fullness: 0 / 10000 metrics.
Dec 03 23:55:00 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:55:00Z D! [outputs.influxdb] buffer fullness: 0 / 10000 metrics.
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:55:05Z D! [outputs.influxdb] wrote batch of 1000 metrics in 95.75191ms
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: 2018-12-04T07:55:05Z E! [inputs.kafka_consumer] Could not mark message delivered: 1
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0xe8e85f]
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: goroutine 45 [running]:
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: github.com/influxdata/telegraf/vendor/github.com/bsm/sarama-cluster.(*Consumer).MarkOffset(0xc000180690, 0x0, 0x0, 0x0)
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]:         /go/src/github.com/influxdata/telegraf/vendor/github.com/bsm/sarama-cluster/consumer.go:143 +0x2f
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: github.com/influxdata/telegraf/plugins/inputs/kafka_consumer.(*Kafka).markOffset(0xc000112000, 0x0)
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]:         /go/src/github.com/influxdata/telegraf/plugins/inputs/kafka_consumer/kafka_consumer.go:244 +0x64
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: github.com/influxdata/telegraf/plugins/inputs/kafka_consumer.(*Kafka).onDelivery(0xc000112000, 0x24f8160, 0xc002a24d40)
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]:         /go/src/github.com/influxdata/telegraf/plugins/inputs/kafka_consumer/kafka_consumer.go:273 +0xfa
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: github.com/influxdata/telegraf/plugins/inputs/kafka_consumer.(*Kafka).receiver(0xc000112000, 0x2510180, 0xc0000b2540, 0x251de20, 0xc00034ab20)
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]:         /go/src/github.com/influxdata/telegraf/plugins/inputs/kafka_consumer/kafka_consumer.go:216 +0x613
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: github.com/influxdata/telegraf/plugins/inputs/kafka_consumer.(*Kafka).Start.func1(0xc000112000, 0x2510180, 0xc0000b2540, 0x251de20, 0xc00034ab20)
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]:         /go/src/github.com/influxdata/telegraf/plugins/inputs/kafka_consumer/kafka_consumer.go:194 +0x84
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]: created by github.com/influxdata/telegraf/plugins/inputs/kafka_consumer.(*Kafka).Start
Dec 03 23:55:05 influxdb-vx-2 telegraf[70895]:         /go/src/github.com/influxdata/telegraf/plugins/inputs/kafka_consumer/kafka_consumer.go:192 +0x227
@akamac
Copy link
Author

akamac commented Dec 4, 2018 •
edited
Loading

@sparrc Could you please take a look? Thanks!

@sparrc
Copy link
Contributor

sparrc commented Dec 4, 2018

@akamac it's not very polite practice to @ maintainers on your issue, they already know it's been opened. Also I'm not a maintainer, it's been 2 years since I worked at InfluxData.

@akamac
Copy link
Author

akamac commented Dec 4, 2018 •
edited
Loading

@akamac it's not very polite practice to @ maintainers on your issue, they already know it's been opened. Also I'm not a maintainer, it's been 2 years since I worked at InfluxData.

@sparrc Sorry for bothering you. I thought, just in case you have time, you could take a look, since I saw you fixed the similar issues before while being a maintainer. Sorry again.

@glinton glinton added bug unexpected problem or unintended behavior area/kafka labels Dec 6, 2018
@glinton glinton added this to the 1.9.1 milestone Dec 6, 2018
@glinton glinton mentioned this issue Dec 6, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/kafka bug unexpected problem or unintended behavior
Projects
None yet
Milestone
1.9.1
Development

No branches or pull requests
3 participants
@akamac @glinton @sparrc