Closed
Description
openedon May 14, 2020
After enabling the Okta module in Filebeat 7.7.0 we have been observing frequent crashes with the error message panic: non-positive interval for NewTicker
Debug logs:
2020-05-14T15:52:45.184Z DEBUG [httpjson] httpjson/input.go:242 Rate Limit: Wait until 2020-05-14 15:52:47 +0000 UTC for the rate limit to reset. {"url": "https://instance.okta.com/api/v1/logs"}
2020-05-14T15:52:45.600Z DEBUG [elasticsearch] elasticsearch/client.go:217 PublishEvents: 50 events have been published to elasticsearch in 66.640319ms.
2020-05-14T15:52:45.600Z DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [236: 0, 50]
2020-05-14T15:52:45.601Z DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=50, start-seq=11701, end-seq=11750
2020-05-14T15:52:45.601Z DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2020-05-14T15:52:45.601Z DEBUG [acker] beater/acker.go:69 stateless ack {"count": 50}
2020-05-14T15:52:45.601Z DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2020-05-14T15:52:45.697Z DEBUG [elasticsearch] elasticsearch/client.go:217 PublishEvents: 50 events have been published to elasticsearch in 96.090322ms.
2020-05-14T15:52:45.699Z DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [237: 0, 50]
2020-05-14T15:52:45.699Z DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=50, start-seq=11751, end-seq=11800
2020-05-14T15:52:45.699Z DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2020-05-14T15:52:45.699Z DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2020-05-14T15:52:45.699Z DEBUG [acker] beater/acker.go:69 stateless ack {"count": 50}
2020-05-14T15:52:45.799Z DEBUG [elasticsearch] elasticsearch/client.go:217 PublishEvents: 50 events have been published to elasticsearch in 101.995502ms.
2020-05-14T15:52:45.799Z DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [238: 0, 50]
2020-05-14T15:52:45.799Z DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=50, start-seq=11801, end-seq=11850
2020-05-14T15:52:45.799Z DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2020-05-14T15:52:45.800Z DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2020-05-14T15:52:45.800Z DEBUG [acker] beater/acker.go:69 stateless ack {"count": 50}
2020-05-14T15:52:45.913Z DEBUG [elasticsearch] elasticsearch/client.go:217 PublishEvents: 50 events have been published to elasticsearch in 113.897908ms.
2020-05-14T15:52:45.913Z DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [239: 0, 50]
2020-05-14T15:52:45.913Z DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=50, start-seq=11851, end-seq=11900
2020-05-14T15:52:45.913Z DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2020-05-14T15:52:45.913Z DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2020-05-14T15:52:45.914Z DEBUG [acker] beater/acker.go:69 stateless ack {"count": 50}
2020-05-14T15:52:47.000Z DEBUG [httpjson] httpjson/input.go:250 Rate Limit: time is up. {"url": "https://instance.okta.com/api/v1/logs"}
2020-05-14T15:52:47.000Z INFO [httpjson] httpjson/input.go:366 Continuing with pagination to URL: https://instance.okta.com/api/v1/logs?after=1588875667507_1 {"url": "https://instance.okta.com/api/v1/logs"}
2020-05-14T15:52:47.194Z DEBUG [httpjson] httpjson/input.go:302 HTTP request failed {"url": "https://instance.okta.com/api/v1/logs", "http.response.status_code": 429, "http.response.body": "{\"errorCode\":\"E0000047\",\"errorSummary\":\"API call exceeded rate limit due to too many requests.\",\"errorLink\":\"E0000047\",\"errorId\":\"oae26ps49lmR7m6OmE6oSGLyA\",\"errorCauses\":[]}"}
2020-05-14T15:52:47.194Z DEBUG [httpjson] httpjson/input.go:242 Rate Limit: Wait until 2020-05-14 15:52:47 +0000 UTC for the rate limit to reset. {"url": "https://instance.okta.com/api/v1/logs"}
2020-05-14T15:52:47.194Z INFO [httpjson] runtime/panic.go:679 httpjson input worker has stopped. {"url": "https://instance.okta.com/api/v1/logs"}
panic: non-positive interval for NewTicker
goroutine 272 [running]:
time.NewTicker(0xfffffffff469c21f, 0xed64f604f)
/usr/local/go/src/time/tick.go:23 +0x147
github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*HttpjsonInput).applyRateLimit(0xc0006460f0, 0x3e13a20, 0xc000654140, 0xc001940060, 0xc0004c5560, 0x0, 0x0)
/go/src/github.com/elastic/beats/x-pack/filebeat/input/httpjson/input.go:243 +0x1a2
github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*HttpjsonInput).processHTTPRequest(0xc0006460f0, 0x3e13a20, 0xc000654140, 0xc0006425a0, 0xc0005b9e08, 0xdf8475800, 0x3da8ae0)
/go/src/github.com/elastic/beats/x-pack/filebeat/input/httpjson/input.go:304 +0x337
github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*HttpjsonInput).run(0xc0006460f0, 0x0, 0x0)
/go/src/github.com/elastic/beats/x-pack/filebeat/input/httpjson/input.go:425 +0x38f
github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*HttpjsonInput).Run.func1.1(0xc0006460f0)
/go/src/github.com/elastic/beats/x-pack/filebeat/input/httpjson/input.go:122 +0x1d6
created by github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*HttpjsonInput).Run.func1
/go/src/github.com/elastic/beats/x-pack/filebeat/input/httpjson/input.go:117 +0x63
Config:
filebeat.yml: |-
filebeat.modules:
- module: okta
system:
enabled: true
# API key to access Okta
var.api_key: "${OKTA_API_KEY}"
# URL of the Okta REST API
var.url: ${OKTA_API_URL}
var.authentication_scheme: "SSWS"
setup.template.enabled: false
output.elasticsearch:
hosts: 'https://${ELASTICSEARCH_ADDRESS}'
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl:
enabled: true
supported_protocols: TLSv1.2
logging.level: debug
For confirmed bugs, please report:
- Version: 7.7.0
- Operating System: GKE CoreOS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment