servertech_sentry3 scraping broke between 0.20.0 and 0.21.0 #1080

robbat2 · 2023-12-26T22:52:08Z

Between 0.20.0 and 0.21.0, the servertech_sentry3 module stopped working, as is still broken as of v0.25.0.

Host operating system: output of `uname -a`

Linux (redacted) 6.1.38-gentoo-dist #1 SMP PREEMPT_DYNAMIC Thu Jul 6 11:45:23 -00 2023 x86_64 QEMU Virtual CPU version 2.5+ GenuineIntel GNU/Linux

snmp_exporter version: output of `snmp_exporter -version`

broken: 0.25.0
broken: 0.24.1
broken: 0.21.0
works: 0.20.0

What device/snmpwalk OID are you using?

servertech_sentry3

What did you do that produced an error?

curl 'localhost:9116/snmp?target=10.0.1.18&auth=(redacted)&module=servertech_sentry3'

What did you expect to see?

Metrics

What did you see instead?

# curl -sq 'localhost:9116/snmp?target=10.0.1.18&auth=(redacted)&module=servertech_sentry3' 
An error has occurred while serving metrics:

error collecting metric Desc{fqName: "snmp_error", help: "Error scraping target", constLabels: {module="servertech_sentry3"}, variableLabels: {}}: error walking target 10.0.1.18: error in unmarshalResponse: error decoding value: bytes: 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 0e 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 0f 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 10 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 01 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 02 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 03 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 04 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 05 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 06 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 07 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 08 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 09 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0a 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0b 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0c 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0d 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0e 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0f 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 10 02 00 30 0e 06 0a 2b 06 01 06 03 01 01 06 01 00 02 00 30 0e 06 0a 2b 06 01 06 03 0b 02 01 01 00 41 00 30 0e 06 0a 2b 06 01 06 03 0b 02 01 02 00 41 00 30 0e 06 0a 2b 06 01 06 03 0b 02 01 03 00 41 00 30 0e 06 0a 2b 06 01 06 03 0b 02 01 03 00 82 00 err: zero length integer

logging output with errors, as seen on 0.21.0, 0.24.1:

# /usr/bin/snmp_exporter --config.file=/etc/snmp_exporter/snmp.yml.0.21.0 
ts=2023-12-26T21:16:51.898Z caller=main.go:148 level=info msg="Starting snmp_exporter" version="(version=0.21.0, branch=non-git, revision=0d8c3527)"
ts=2023-12-26T21:16:51.898Z caller=main.go:149 level=info build_context="(go=go1.21.5, user=portage@localhost, date=20231226-21:09:51)"
ts=2023-12-26T21:16:52.214Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9116
ts=2023-12-26T21:16:52.214Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9116
ts=2023-12-26T21:16:53.904Z caller=collector.go:282 level=info module=servertech_sentry3 target=10.0.1.18 msg="Error scraping target" err="error walking target 10.0.1.18: error in unmarshalResponse: error decoding value: bytes: 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 0e 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 0f 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 10 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 01 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 02 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 03 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 04 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 05 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 06 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 07 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 08 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 09 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0a 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0b 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0c 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0d 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0e 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0f 02 00 30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 10 02 00 30 0e 06 0a 2b 06 01 06 03 01 01 06 01 00 02 00 30 0e 06 0a 2b 06 01 06 03 0b 02 01 01 00 41 00 30 0e 06 0a 2b 06 01 06 03 0b 02 01 02 00 41 00 30 0e 06 0a 2b 06 01 06 03 0b 02 01 03 00 41 00 30 0e 06 0a 2b 06 01 06 03 0b 02 01 03 00 82 00 err: zero length integer"

logging output as seen on 0.20.0:

# /usr/bin/snmp_exporter-0.20.0 --config.file=/etc/snmp_exporter/snmp.yml.old  --log.level=debug --log.format=json 
{"caller":"main.go:152","level":"info","msg":"Starting snmp_exporter","ts":"2023-12-26T22:41:58.051Z","version":"(version=0.20.0, branch=non-git, revision=c33572b6)"}
{"build_context":"(go=go1.21.5, user=portage@localhost, date=20231226-21:27:49)","caller":"main.go:153","level":"info","ts":"2023-12-26T22:41:58.052Z"}
{"address":":9116","caller":"main.go:246","level":"info","msg":"Listening on address","ts":"2023-12-26T22:41:58.322Z"}
{"caller":"tls_config.go:191","http2":false,"level":"info","msg":"TLS is disabled.","ts":"2023-12-26T22:41:58.323Z"}
{"caller":"main.go:102","level":"debug","module":"servertech_sentry3","msg":"Starting scrape","target":"10.0.1.18","ts":"2023-12-26T22:41:59.519Z"}
{"caller":"collector.go:131","level":"debug","module":"servertech_sentry3","msg":"Getting OIDs","oids":1,"target":"10.0.1.18","ts":"2023-12-26T22:41:59.520Z"}
{"caller":"collector.go:140","duration_seconds":"6.733727ms","level":"debug","module":"servertech_sentry3","msg":"Get of OIDs completed","oids":1,"target":"10.0.1.18","ts":"2023-12-26T22:41:59.527Z"}
{"caller":"collector.go:164","level":"debug","module":"servertech_sentry3","msg":"Walking subtree","oid":"1.3.6.1.4.1.1718.3.2.2","target":"10.0.1.18","ts":"2023-12-26T22:41:59.527Z"}
{"caller":"collector.go:177","duration_seconds":"31.290953ms","level":"debug","module":"servertech_sentry3","msg":"Walk of subtree completed","oid":"1.3.6.1.4.1.1718.3.2.2","target":"10.0.1.18","ts":"2023-12-26T22:41:59.558Z"}
{"caller":"collector.go:164","level":"debug","module":"servertech_sentry3","msg":"Walking subtree","oid":"1.3.6.1.4.1.1718.3.2.3","target":"10.0.1.18","ts":"2023-12-26T22:41:59.558Z"}
{"caller":"collector.go:177","duration_seconds":"292.805443ms","level":"debug","module":"servertech_sentry3","msg":"Walk of subtree completed","oid":"1.3.6.1.4.1.1718.3.2.3","target":"10.0.1.18","ts":"2023-12-26T22:41:59.851Z"}
{"caller":"main.go:113","duration_seconds":0.347416139,"level":"debug","module":"servertech_sentry3","msg":"Finished scrape","target":"10.0.1.18","ts":"2023-12-26T22:41:59.867Z"}

curl output as on 0.20.0:

# curl -sq 'localhost:9116/snmp?target=10.0.1.18&auth=(redacted)&module=servertech_sentry3' |head
# HELP infeedCapabilities The capabilities of the input feed. - 1.3.6.1.4.1.1718.3.2.2.1.4 (Bits)
# TYPE infeedCapabilities gauge
infeedCapabilities{infeedCapabilities="branchLoadSense",infeedIndex="1",towerIndex="1"} 0
infeedCapabilities{infeedCapabilities="branchLoadSense",infeedIndex="1",towerIndex="2"} 0
infeedCapabilities{infeedCapabilities="branchOnSense",infeedIndex="1",towerIndex="1"} 0
infeedCapabilities{infeedCapabilities="branchOnSense",infeedIndex="1",towerIndex="2"} 0
...
snmp_scrape_duration_seconds 0.357499634
# HELP snmp_scrape_pdus_returned PDUs returned from walk.
# TYPE snmp_scrape_pdus_returned gauge
snmp_scrape_pdus_returned 337
# HELP snmp_scrape_walk_duration_seconds Time SNMP walk/bulkwalk took.
# TYPE snmp_scrape_walk_duration_seconds gauge
snmp_scrape_walk_duration_seconds 0.348002039
# HELP sysUpTime The time (in hundredths of a second) since the network management portion of the system was last re-initialized. - 1.3.6.1.2.1.1.3
# TYPE sysUpTime gauge
sysUpTime 1.24420047e+09

The text was updated successfully, but these errors were encountered:

robbat2 · 2023-12-26T23:02:23Z

Git bisect shows the error was introduced with 4449004.
That commit included an update to github.com/gosnmp/gosnmp: v1.32.0 -> v1.34.0

git bisect start
# status: waiting for both good and bad commits
# good: [c33572b6c8c8e43a479fde0f9fa8ac86e15598bc] Merge pull request #612 from prometheus/richih/0.20.0
git bisect good c33572b6c8c8e43a479fde0f9fa8ac86e15598bc
# status: waiting for bad commit, 1 good commit known
# bad: [0d8c3527cac0c26f1d6005b84b74413d14264c37] Release 0.21.0 (#819)
git bisect bad 0d8c3527cac0c26f1d6005b84b74413d14264c37
# good: [15dd541b71cbf768135fb5f2a3c3f33bad61c886] Merge pull request #670 from Crabbey/main
git bisect good 15dd541b71cbf768135fb5f2a3c3f33bad61c886
# bad: [f0de4cf14f87760e911106a8733ad198eb85915b] Bump github.com/gosnmp/gosnmp from 1.34.0 to 1.35.0 (#765)
git bisect bad f0de4cf14f87760e911106a8733ad198eb85915b
# bad: [4449004cf8f07be51bec99070e9d96c43abe7f65] Update build (#697)
git bisect bad 4449004cf8f07be51bec99070e9d96c43abe7f65
# good: [3726d4612f55ae5b9cb497cfd7bda03011b582aa] Merge pull request #677 from candlerb/candlerb/vrrpstats
git bisect good 3726d4612f55ae5b9cb497cfd7bda03011b582aa
# good: [8a56b28af255377f9d52d823df906d7e246742f9] Hacky workaround for net-snmp version detection (#704)
git bisect good 8a56b28af255377f9d52d823df906d7e246742f9
# good: [83399c23888fc08b7c943a471ad39331e0ba3d96] Fix broken link (#705)
git bisect good 83399c23888fc08b7c943a471ad39331e0ba3d96
# first bad commit: [4449004cf8f07be51bec99070e9d96c43abe7f65] Update build (#697)

robbat2 · 2023-12-26T23:17:21Z

Revert to gosnmp@v1.32.0, with commenting out the LocalAddr feature, does make the excporter work again.

SuperQ · 2023-12-27T07:29:05Z

It seems like this error message was introduced in gosnmp/gosnmp#373 to fix invalid SNMP packet data.

SuperQ · 2023-12-27T07:37:55Z

Interestingly, I also have a servertech sentry3 device. I tested with v0.25.0 and don't run into this. I wonder if your device firmware has a bug. What version of the firmware are you running?

robbat2 · 2023-12-27T23:08:35Z

Sentry3-MIB::systemVersion.0 = STRING: Sentry Version 5.3q

SuperQ · 2023-12-28T08:04:10Z

That version is from 2008. There have been major updates to the Sentry firmware since then. I'm guessing this is a firmware bug in the old SNMP library.

https://cdn10.servertech.com/assets/documents/documents/909/original/swcdu-history.txt

robbat2 · 2023-12-29T06:26:08Z

snmpbulkwalk does handle this case correctly, so I'm wondering if it's worthwhile to provide parity. I'll chase the people with admin to the device to update it anyway.

SuperQ · 2023-12-29T09:08:02Z

Adding a workaround for this would probably require upstream changes in gosnmp. Opening an issue there, with some tcpdump packet captures of net-snmp snmpbulkwalk would be needed in order to add reproduction cases.

Especially helpful would be to reduce the walk to the specific OIDs that are returning the weird data.

robbat2 · 2024-01-13T18:25:48Z

I patched go-snmp to print the OID when it fails to decode.
1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13 is the OID when it hits the bug, but it's WEIRD.

This bug is only happening when snmp_exporter is walking the OID tree.
It does NOT happen with other gosnmp examples.

Here's the byte dump from before, split up:

02 00 # ASN.1 int, length 0; trips the error the first time
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 0e 02 00 # 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.14, ASN.1 int, asnlength 0
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 0f 02 00  # 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.15, ASN.1 int, asnlength 0
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 10 02 00 # 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.16, ASN.1 int, asnlength 0
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 01 02 00 # 1.3.6.1.4.1.1718.3.2.3.1.11.1.2.1, ASN.1 int, asnlength 0 
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 02 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 03 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 04 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 05 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 06 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 07 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 08 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 09 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0a 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0b 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0c 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0d 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0e 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0f 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 10 02 00 # 1.3.6.1.4.1.1718.3.2.3.1.11.1.2.16, ASN.1 int, asnlength 0

In the debug output from snmp_exporter, we see an ASN.1 encoded integer supposedly as "02 00"
02 = integer
00 = asnlength

But the net-snmp output for it shows boring and correct ASN.1 (02 01 00)

net-snmp source should also error on a integer w/ asnlength==0
https://github.com/net-snmp/net-snmp/blob/master/snmplib/asn1.c#L604-L607

trace: snmp_pdu_parse(): snmp_api.c, 4724:
dumph_recv:     VarBindList
trace: snmp_pdu_parse(): snmp_api.c, 4740:
dumph_recv:       VarBind
trace: snmp_parse_var_op(): snmp.c, 165:
dumph_recv:         Name
dumpx_recv:          06 0F 2B 06 01 04 01 8D 36 03 02 03 01 0B 01 01 0D 
dumpv_recv:            ObjID: SNMPv2-SMI::enterprises.1718.3.2.3.1.11.1.1.13
trace: snmp_pdu_parse(): snmp_api.c, 4749:
dumph_recv:         Value
dumpx_recv:          02 01 00 
dumpv_recv:            Integer:	0 (0x00)
trace: _sess_process_packet_parse_pdu(): snmp_api.c, 5677:

gosnmp example2 (minor patches to pass OID & community) also WORKS:

gosnmp/examples/example2 $ GOSNMP_TARGET=10.0.1.18 GOSNMP_COMMUNITY=(redacted)  go run . 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13
2024/01/13 17:58:14 OIDs: [1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13]
SEND INIT
SENDING PACKET: Version:2c, MsgFlags:NoAuthNoPriv, SecurityModel:SnmpV3SecurityModel(0), SecurityParameters:, ContextEngineID:, ContextName:, Community:(redacted), PDUType:GetRequest, MsgID:0, RequestID:2068017646, MsgMaxSize:0, Error:NoError, ErrorIndex:0, NonRepeaters:0, MaxRepetitions:0, Variables:[{<nil> 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13 Null}]
WAITING RESPONSE...
2024/01/13 17:58:14 Query latency in seconds: 0.014885664
GET RESPONSE OK: [48 53 2 1 1 4 10 79 83 76 95 112 117 98 108 105 99 162 36 2 4 123 67 113 238 2 1 0 2 1 0 48 22 48 20 6 15 43 6 1 4 1 141 54 3 2 3 1 11 1 1 13 2 1 0]
Packet sanity verified, we got all the bytes (55)
parseRawField: version
Parsed version 1
parseRawField: community
Parsed community (redacted)
UnmarshalPayload Meet PDUType 0x476574526573706f6e7365. Offset 17
getResponseLength: 38
parseRawField: request id
requestID: 2068017646
parseRawField: error-status
errorStatus: 0
parseRawField: error index
error-index: 0
vblLength: 24
parseRawField: OID
OID: .1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13
decodeValue: type is Integer
decodeValue: value is 0
0: oid: .1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13 number: 0

next up, trying to a tcpdump of snmp_exporter and comparing to the tcpdump of gosnmp examples.

robbat2 · 2024-01-14T23:35:58Z

Did the tcpdumps, found a reproduction case.

snmp_exporter-0.20.0 just silently returned a value of 0 for these fields instead of throwing an error.

Frame 33: 102 bytes on wire (816 bits), 102 bytes captured (816 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: (redacted:host-ip), Dst: (redacted:device-ip)
User Datagram Protocol, Src Port: (redacted:host-port), Dst Port: 161
Simple Network Management Protocol
    version: v2c (1)
    community: (redacted:community)
    data: getBulkRequest (5)
        getBulkRequest
            request-id: 1366421655
            non-repeaters: 0
            max-repetitions: 25
            variable-bindings: 1 item
                1.3.6.1.4.1.1718.3.2.3.1.11.1.1.12: Value (Null)
                    Object Name: 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.12 (iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.12)
                    Value (Null)

Frame 34: 587 bytes on wire (4696 bits), 587 bytes captured (4696 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: (redacted:device-ip), Dst: (redacted:host-ip)
User Datagram Protocol, Src Port: 161, Dst Port: (redacted:host-port)
Simple Network Management Protocol
    version: v2c (1)
    community: (redacted:community)
    data: get-response (2)
        get-response
            request-id: 1366421655
            error-status: noError (0)
            error-index: 24
            variable-bindings: 25 items
                1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13:
                    Object Name: 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13 (iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.13)
                    Integral value is zero-length
                        [Expert Info (Note/Undecoded): Integral value is zero-length]
                            [Integral value is zero-length]
                            [Severity level: Note]
                            [Group: Undecoded]
                1.3.6.1.4.1.1718.3.2.3.1.11.1.1.14:
                    Object Name: 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.14 (iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.14)
                    Integral value is zero-length
                        [Expert Info (Note/Undecoded): Integral value is zero-length]
                            [Integral value is zero-length]
                            [Severity level: Note]
                            [Group: Undecoded]

...

robbat2 · 2024-01-14T23:55:14Z

Deeper debugging shows there's a subtle firmware bug in requesting the last OID values available, depending on the max repetitions.

PR forthcoming to change max repetitions for this device, via Generator

The device I have access to, exposes 349 OID values, when I do snmpbulkwalk.

With max-repetitions from range 1-4 inclusive, no OIDs fail for BULKGET.
With max-repetitions=5, the very last OID, 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.16 fails.
With max-repetitions=6, 2nd to last OID, 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.15 fails.
...
With max-repetitions=350, the 355th to last OID (so the 4th OID), 1.3.6.1.4.1.1718.3.1.4.0
With max-repetitions=353, the 1st OID fails! 1.3.6.1.4.1.1718.3.1.1.0
With max-repetitions=354-359, no OIDs fail again!
With max-repetitions=360+ the pattern starts again

robbat2 · 2024-01-14T23:58:10Z

I'm going to use max-repetitions=4 in the fix, because that's the highest value for which no OID probed will ever trigger the bug.

servertech_sentry3 devices can return bad ASN.1 data, an Integer with asnlength=0 under certain cases. - Certain firmware versions only - SNMP BULKGET - Specific OIDs deep into the tree, at different repetition values Work around the problem by setting max_repetitions=4, which doesn't trigger the device bug. Closes: prometheus#1080 Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

SuperQ · 2024-01-15T08:27:16Z

Thanks for all the investigation work.

servertech_sentry3 devices can return bad ASN.1 data, an Integer with asnlength=0 under certain cases. - Certain firmware versions only - SNMP BULKGET - Specific OIDs deep into the tree, at different repetition values Work around the problem by setting max_repetitions=4, which doesn't trigger the device bug. Closes: #1080 Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

) servertech_sentry3 devices can return bad ASN.1 data, an Integer with asnlength=0 under certain cases. - Certain firmware versions only - SNMP BULKGET - Specific OIDs deep into the tree, at different repetition values Work around the problem by setting max_repetitions=4, which doesn't trigger the device bug. Closes: prometheus#1080 Signed-off-by: Robin H. Johnson <robbat2@gentoo.org> Signed-off-by: Harshavardhan Musanalli <Harshavardhan.Musanalli@amadeus.com>

SuperQ added the device-bug label Dec 28, 2023

robbat2 mentioned this issue Jan 15, 2024

fix: workaround servertech_sentry3 zero-length integers #1090

Merged

SuperQ closed this as completed in #1090 Jan 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

servertech_sentry3 scraping broke between 0.20.0 and 0.21.0 #1080

servertech_sentry3 scraping broke between 0.20.0 and 0.21.0 #1080

robbat2 commented Dec 26, 2023

robbat2 commented Dec 26, 2023

robbat2 commented Dec 26, 2023

SuperQ commented Dec 27, 2023 •

edited

Loading

SuperQ commented Dec 27, 2023

robbat2 commented Dec 27, 2023

SuperQ commented Dec 28, 2023

robbat2 commented Dec 29, 2023

SuperQ commented Dec 29, 2023

robbat2 commented Jan 13, 2024

robbat2 commented Jan 14, 2024

robbat2 commented Jan 14, 2024

robbat2 commented Jan 14, 2024

SuperQ commented Jan 15, 2024

servertech_sentry3 scraping broke between 0.20.0 and 0.21.0 #1080

servertech_sentry3 scraping broke between 0.20.0 and 0.21.0 #1080

Comments

robbat2 commented Dec 26, 2023

Host operating system: output of uname -a

snmp_exporter version: output of snmp_exporter -version

What device/snmpwalk OID are you using?

What did you do that produced an error?

What did you expect to see?

What did you see instead?

robbat2 commented Dec 26, 2023

robbat2 commented Dec 26, 2023

SuperQ commented Dec 27, 2023 • edited Loading

SuperQ commented Dec 27, 2023

robbat2 commented Dec 27, 2023

SuperQ commented Dec 28, 2023

robbat2 commented Dec 29, 2023

SuperQ commented Dec 29, 2023

robbat2 commented Jan 13, 2024

robbat2 commented Jan 14, 2024

robbat2 commented Jan 14, 2024

robbat2 commented Jan 14, 2024

SuperQ commented Jan 15, 2024

Host operating system: output of `uname -a`

snmp_exporter version: output of `snmp_exporter -version`

SuperQ commented Dec 27, 2023 •

edited

Loading