Reload does not seem to be working

[keithf4]

Describe the bug
Was curious if there was support for reloading and saw that a feature was added quite a while ago for at least reloading the contents of the collector files (#15). However I cannot seem to get this to work, assuming I'm doing it right. Some additional documentation on this would be helpful.

To Reproduce
Steps to reproduce the behavior:
Running sql_exporter with reload feature enabled

ps aux | grep sql_exporter
ccp_mon+  136764  0.0  3.1 748216 40916 ?        Ssl  16:19   0:00 /usr/bin/sql_exporter -config.file=/etc/sql_exporter/sql_exporter.yml -web.listen-address=0.0.0.0:9399 -log.level=info -web.enable-reload

I changed the content of one of my collector files to remove a metric. I then hit the reload endpoint

# curl localhost:9399/reload
curl: (52) Empty reply from server

No change in the output of the metrics endpoint. Restarting sql_exporter confirms the change in collector file is valid and the metric is removed/readded on change

Expected behavior
Metrics are added/removed when reload endpoint is hit

[burningalchemist]

Hey @keithf4, is there any log on the sql_exporter side you could share? That'd help. But I'll have a look, it worked before. 👍

I briefly ran it on some empty instance, here's my setup:

~/Projects/sql_exporter (master ✔) ./sql_exporter -web.enable-reload --log.level=debug
ts=2023-08-25T20:45:59.874Z caller=klog.go:108 level=warn func=Warningf msg="Starting SQL exporter (version=0.12.1, branch=master, revision=1cfbd6095ac7303621a73aef74841721dc2e22c2) (go=go1.21.0, platform=darwin/arm64, date=20230822-15:07:23, tags=netgo)"
ts=2023-08-25T20:45:59.874Z caller=klog.go:84 level=debug func=Infof msg="Loading configuration from sql_exporter.yml"
ts=2023-08-25T20:45:59.875Z caller=klog.go:96 level=warn func=Warning msg="Listening on :9399"
ts=2023-08-25T20:45:59.875Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9399
ts=2023-08-25T20:45:59.875Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9399
ts=2023-08-25T20:46:02.640Z caller=klog.go:84 level=debug func=Infof msg="Reloading the collectors..."
ts=2023-08-25T20:46:02.640Z caller=klog.go:84 level=debug func=Infof msg="Query collectors have been successfully reloaded"

And then the call:

~/Projects/sql_exporter (master ✔) curl localhost:9399/reload -vvv
*   Trying 127.0.0.1:9399...
* Connected to localhost (127.0.0.1) port 9399 (#0)
> GET /reload HTTP/1.1
> Host: localhost:9399
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 204 No Content
< Date: Fri, 25 Aug 2023 20:46:02 GMT
<
* Connection #0 to host localhost left intact

It's interesting that it returns empty response to you. I'd expect the errors on the sql_exporter side. It could be we need to use debug log to see them. But this can be fixed too. Let me know. :)

[keithf4]

Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: 2023/08/25 17:48:03 http: panic serving [::1]:49170: runtime error: invalid memory address or nil pointer dereference
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: goroutine 228 [running]:
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: net/http.(*conn).serve.func1()
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: #011/usr/local/go/src/net/http/server.go:1854 +0xbf
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: panic({0x17c6d60, 0x28a3760})
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: #011/usr/local/go/src/runtime/panic.go:890 +0x263
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: main.reloadCollectors.func1({0x1c66aa0, 0xc0007648c0}, 0x0?)
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: #011/app/cmd/sql_exporter/main.go:123 +0x119
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: net/http.HandlerFunc.ServeHTTP(0xc000148780?, {0x1c66aa0?, 0xc0007648c0?}, 0x40dd0a?)
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: #011/usr/local/go/src/net/http/server.go:2122 +0x2f
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: net/http.(*ServeMux).ServeHTTP(0x0?, {0x1c66aa0, 0xc0007648c0}, 0xc000128a00)
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: #011/usr/local/go/src/net/http/server.go:2500 +0x149
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: net/http.serverHandler.ServeHTTP({0xc00019bf80?}, {0x1c66aa0, 0xc0007648c0}, 0xc000128a00)
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: #011/usr/local/go/src/net/http/server.go:2936 +0x316
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: net/http.(*conn).serve(0xc000125560, {0x1c69208, 0xc000838ea0})
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: #011/usr/local/go/src/net/http/server.go:1995 +0x612
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: created by net/http.(*Server).Serve
Aug 25 17:48:03 cd-dc1-pg1 sql_exporter[33034]: #011/usr/local/go/src/net/http/server.go:3089 +0x5ed

[keithf4]

[root@cd-dc1-pg1 sql_exporter]# curl localhost:9399/reload -vvv
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9399 (#0)
> GET /reload HTTP/1.1
> Host: localhost:9399
> User-Agent: curl/7.61.1
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

[burningalchemist]

@keithf4 yeah, the server crashes when trying to serve this request.

It could be that the configuration is wrong, so it cannot be parsed before being reloaded to the collector structure.

I need more context to fix this. Could you share the configuration? You can omit the sql query itself.

[burningalchemist]

I think I know where the error might be.

I bet you're using jobs, right? Reload currently repopulates a single target struct. And since you don't have it configured in the config file, the app is trying to repopulate nonexistent object, hence wrong memory access.

I'll check it. Multiple connections feature isn't that popular, so it apparently needs more attention.

[keithf4]

I am using jobs yes. Basically the same config I gave in the previous example for the docs.

[sebastianhoenig]

I second this request, just ran into the same thing using configmaps in openshift (and multiple jobs).

Looking forward to have this implemented 👍

[burningalchemist]

@sebastianhoenig @keithf4 the draft PR is already here, I'm testing it. Hope to merge it this week. Stay tuned 🎶

[burningalchemist]

Hey folks, apologies - it took one week longer with testing, fixes, and daily job.
I plan to cut a release next week, but you can already test it with your own build. 👍