[telegraf] Error parsing response: parsing time "null"

[zinuzoid]

Metric in Grafana seems doesn't work, so I check telegraf and there're some error logs from pods.

I'm running cluster in GKE k8s version 1.4.6
Images: quay.io/deis/telegraf:v2.7.0

2016-12-13T09:01:07.656026655Z Node Name set (gke-deis-default-pool-6ba9e8cc-v3kd)
2016-12-13T09:01:07.824924771Z Node IP set (10.240.0.19)
2016-12-13T09:01:07.824989798Z Creating topic with URL: http://10.103.253.156:4151/topic/create?topic=metrics
2016-12-13T09:01:07.836194361Z Setting KUBERNETES_URL: http://10.240.0.19:10255
2016-12-13T09:01:07.836252337Z Building config.toml!
2016-12-13T09:01:07.849382935Z Finished building toml...
2016-12-13T09:01:07.849422582Z ###########################################
2016-12-13T09:01:07.849431084Z ###########################################
2016-12-13T09:01:07.850343415Z # Set Tag Configuration
2016-12-13T09:01:07.850365550Z [tags]
2016-12-13T09:01:07.850373997Z # Set Agent Configuration
2016-12-13T09:01:07.850381146Z [agent]
2016-12-13T09:01:07.850387919Z   interval = "10s"
2016-12-13T09:01:07.850394825Z   round_interval = true
2016-12-13T09:01:07.850400548Z   metric_buffer_limit = 100000
2016-12-13T09:01:07.850406425Z   collection_jitter = "1s"
2016-12-13T09:01:07.850412938Z   flush_interval = "1s"
2016-12-13T09:01:07.850417002Z   flush_jitter = "0s"
2016-12-13T09:01:07.850420935Z   debug = true
2016-12-13T09:01:07.850424675Z   quiet = false
2016-12-13T09:01:07.850428540Z   flush_buffer_when_full = true
2016-12-13T09:01:07.850433117Z   hostname = "gke-deis-default-pool-6ba9e8cc-v3kd" 
2016-12-13T09:01:07.850437102Z # Set output configuration
2016-12-13T09:01:07.850441009Z [[outputs.influxdb]]
2016-12-13T09:01:07.850444630Z   urls = ["http://10.103.253.43:80"]
2016-12-13T09:01:07.850448397Z   database = "kubernetes"
2016-12-13T09:01:07.850455322Z   precision = "ns"
2016-12-13T09:01:07.850459065Z   timeout = "5s"
2016-12-13T09:01:07.850462690Z   
2016-12-13T09:01:07.850466214Z   
2016-12-13T09:01:07.850469755Z   
2016-12-13T09:01:07.850473390Z   
2016-12-13T09:01:07.850476925Z # Set Input Configuration
2016-12-13T09:01:07.850480562Z [[inputs.netstat]]
2016-12-13T09:01:07.850484788Z [[inputs.swap]]
2016-12-13T09:01:07.850490907Z [[inputs.system]]
2016-12-13T09:01:07.850496625Z [[inputs.mem]]
2016-12-13T09:01:07.850502727Z [[inputs.cpu]]
2016-12-13T09:01:07.850508447Z   percpu = true
2016-12-13T09:01:07.850513963Z   totalcpu = true
2016-12-13T09:01:07.850519322Z [[inputs.disk]]
2016-12-13T09:01:07.850524783Z   
2016-12-13T09:01:07.850530315Z [[inputs.diskio]]
2016-12-13T09:01:07.850536095Z   
2016-12-13T09:01:07.850553141Z   
2016-12-13T09:01:07.850559799Z [[inputs.net]]
2016-12-13T09:01:07.850566049Z   
2016-12-13T09:01:07.850571555Z [[inputs.influxdb]]
2016-12-13T09:01:07.850576800Z   urls = ["http://10.103.253.43:80/debug/vars"]
2016-12-13T09:01:07.850582587Z [[inputs.kubernetes]]
2016-12-13T09:01:07.850588418Z   url = "http://10.240.0.19:10255"
2016-12-13T09:01:07.850594063Z   bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
2016-12-13T09:01:07.850600021Z   
2016-12-13T09:01:07.850605122Z   
2016-12-13T09:01:07.850610547Z   
2016-12-13T09:01:07.850615908Z   insecure_skip_verify = true
2016-12-13T09:01:07.850621341Z [[inputs.nsq]]
2016-12-13T09:01:07.850627237Z   endpoints = ["http://10.103.253.156:4151"]
2016-12-13T09:01:07.850632803Z   interval = "1s"
2016-12-13T09:01:07.850638463Z [[inputs.redis]]
2016-12-13T09:01:07.850643955Z   servers = ["tcp://:foo@10.103.243.126:6379"]
2016-12-13T09:01:07.850649618Z # Set Service Input Configuration
2016-12-13T09:01:07.850655265Z [[inputs.nsq_consumer]]
2016-12-13T09:01:07.850660967Z   server = "10.103.253.156:4150"
2016-12-13T09:01:07.850666620Z   topic = "metrics"
2016-12-13T09:01:07.850672023Z   channel = "consumer"
2016-12-13T09:01:07.850678743Z   max_in_flight = 100
2016-12-13T09:01:07.850684593Z   data_format = "influx"
2016-12-13T09:01:07.850692843Z ###########################################
2016-12-13T09:01:07.850700329Z ###########################################
2016-12-13T09:01:07.910949043Z 2016/12/13 09:01:07 INF    1 [metrics/consumer] (10.103.253.156:4150) connecting to nsqd
2016-12-13T09:01:10.587872549Z 2016/12/13 09:01:10 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:20.813784155Z 2016/12/13 09:01:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:30.488890095Z 2016/12/13 09:01:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:41.219745287Z 2016/12/13 09:01:41 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:50.722231932Z 2016/12/13 09:01:50 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:01.229486206Z 2016/12/13 09:02:01 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:11.007993912Z 2016/12/13 09:02:11 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:20.600314059Z 2016/12/13 09:02:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:30.652158165Z 2016/12/13 09:02:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:40.198493854Z 2016/12/13 09:02:40 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:50.223258238Z 2016/12/13 09:02:50 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:00.858215308Z 2016/12/13 09:03:00 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:10.705881319Z 2016/12/13 09:03:10 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:20.771301571Z 2016/12/13 09:03:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:30.437874621Z 2016/12/13 09:03:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:40.603927629Z 2016/12/13 09:03:40 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:51.060918864Z 2016/12/13 09:03:51 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:00.266918214Z 2016/12/13 09:04:00 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:10.435277890Z 2016/12/13 09:04:10 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:20.816313126Z 2016/12/13 09:04:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:30.839003676Z 2016/12/13 09:04:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]

Is there a way to dig it deeper. I did try to add AGENT_DEBUG:true, AGENT_QUITE:false but doesn't help much from log.

[vdice]

Hmmm, I'm not able to reproduce this issue w/ same telegraf image (also GKE, k8s server 1.4.7)

Is there a way to dig it deeper. I did try to add AGENT_DEBUG:true, AGENT_QUITE:false but doesn't help much from log.

@jchauncey any suggestions on how to debug?

[jchauncey]

Nope. I've seen this before but I'm not sure what the cause was. Is it still happening?

…

On Jan 11, 2017 4:40 PM, "Vaughn Dice" ***@***.***> wrote: Hmmm, I'm not able to reproduce this issue w/ same telegraf image (also GKE, k8s server 1.4.7) Is there a way to dig it deeper. I did try to add AGENT_DEBUG:true, AGENT_QUITE:false but doesn't help much from log. @jchauncey <https://github.com/jchauncey> any suggestions on how to debug? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAaRGG44WRs6bIBsrBRaYhCJH2f-7BPGks5rRUxLgaJpZM4LLgQt> .

[zinuzoid]

@jchauncey yes, it's still happen.

2017-01-12T11:00:40.151027731Z 2017/01/12 11:00:40 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2017-01-12T11:00:50.990826710Z 2017/01/12 11:00:50 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]

[stuszynski]

@zinuzoid I encounter a similar issue. Telegraf is using url = "http://10.240.0.19:10255" for connecting to kubelet, but your kubelet is most likely listen on HTTPS only.

[stuszynski]

Sorry, my bad. The http://<node-ip>:10255 refer to the unsecured read-only port available on each kubelet.

     --read-only-port int32                                    The read-only port for the Kubelet to serve on with no authentication/authorization, and for localhost healthz endpoint (set to 0 to disable) (default 10255)

I check it manually and I'm sure it provide data on this URL from a telegraf container. We still, however, experience above issue.

[Cryptophobia]

This issue was moved to teamhephy/monitor#9