backupDSLoop exception: mark end of backup operation: driver: bad connection; driver: bad connection

[kalavt]

Describe the bug
mark end of backup operation: driver: bad connection; driver: bad connection

Expected behavior
version: 1.22.1

Screenshots
`
2024-04-02T13:32:11.775Z ERROR utils.misc misc/misc.go:926 Panic detected. Application will crash. {"stack": "goroutine 430 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()\n\t/rudder-server/utils/misc/misc.go:927 +0x37e\nsync.(*Once).doSlow(0x60f1280?, 0xc002b31e18?)\n\t/usr/local/go/src/sync/once.go:74 +0xbf\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()\n\t/rudder-server/utils/misc/misc.go:916 +0xbb\npanic({0x32a8f60?, 0xc08b668dc0?})\n\t/usr/local/go/src/runtime/panic.go:914 +0x21f\ngithub.com/rudderlabs/rudder-server/jobsdb.(*Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})\n\t/rudder-server/jobsdb/backup.go:105 +0x345\ngithub.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.func1()\n\t/rudder-server/jobsdb/jobsdb.go:1016 +0x1f\ngithub.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.WithBugsnag.func2()\n\t/rudder-server/utils/misc/misc.go:945 +0x5c\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573\n\t/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96\n", "panic": "mark end of backup operation: driver: bad connection; driver: bad connection", "team": "Core", "goRoutines": 1169, "version": "1.22.1", "releaseStage": "development"}
2024-04-02T13:32:11.776Z ERROR utils.misc misc/misc.go:926 goroutine 430 [running]:
github.com/rudderlabs/rudder-go-kit/logger.(*logger).Fatalw(0xc001490a50, {0x38fbe93?, 0x4bf?}, {0xc089ae1500?, 0x1?, 0x4ffbcc?})
/go/pkg/mod/github.com/rudderlabs/rudder-go-kit@v0.23.2/logger/logger.go:332 +0xa5
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()
/rudder-server/utils/misc/misc.go:926 +0x5f6
sync.(*Once).doSlow(0x60f1280?, 0xc002b31e18?)
/usr/local/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:65
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()
/rudder-server/utils/misc/misc.go:916 +0xbb
panic({0x32a8f60?, 0xc08b668dc0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})
/rudder-server/jobsdb/backup.go:105 +0x345
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.func1()
/rudder-server/jobsdb/jobsdb.go:1016 +0x1f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.WithBugsnag.func2()
/rudder-server/utils/misc/misc.go:945 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573
/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96

2024/04/02 13:32:11 notifying bugsnag: mark end of backup operation: driver: bad connection; driver: bad connection
2024/04/02 13:32:11 bugsnag.Notify: bugsnag/payload.deliver: invalid api key: ''
2024/04/02 13:32:11 bugsnag/sessions/publisher.publish invalid API key: ''
panic: mark end of backup operation: driver: bad connection; driver: bad connection [recovered]
panic: mark end of backup operation: driver: bad connection; driver: bad connection [recovered]
panic: mark end of backup operation: driver: bad connection; driver: bad connection

goroutine 430 [running]:
github.com/bugsnag/bugsnag-go/v2.AutoNotify({0xc06e6099f0, 0x3, 0x7f09e72068b8?})
/go/pkg/mod/github.com/bugsnag/bugsnag-go/v2@v2.3.0/bugsnag.go:114 +0x3e8
panic({0x32a8f60?, 0xc08b668dc0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()
/rudder-server/utils/misc/misc.go:935 +0x614
sync.(*Once).doSlow(0x60f1280?, 0xc002b31e18?)
/usr/local/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:65
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()
/rudder-server/utils/misc/misc.go:916 +0xbb
panic({0x32a8f60?, 0xc08b668dc0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})
/rudder-server/jobsdb/backup.go:105 +0x345
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.func1()
/rudder-server/jobsdb/jobsdb.go:1016 +0x1f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.WithBugsnag.func2()
/rudder-server/utils/misc/misc.go:945 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573
/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96
`

[gitcommitshow]

@kalavt can you please share more details? How to reproduce this issue?

[Digital365Staking]

The error message you're seeing indicates that there's a problem with the database connection during the backup operation in the rudder-server application. The error message "driver: bad connection" typically means that the application is unable to establish a connection to the database⁴.

Here are a few steps you can take to troubleshoot this issue:

1. Check the Database Connection: Ensure that the database is running and that the application has the correct connection details. If the database is running on a different server, check that the network connection between the two servers is working correctly.

2. Check the Database Driver: The error message suggests there might be an issue with the database driver. Make sure you're using the correct driver for your database and that the driver is up-to-date.

3. Inspect the Application Logs: The logs can provide more details about what the application was doing when the error occurred. Look for any error messages or warnings that occurred around the same time as the "driver: bad connection" error.

4. Update Rudder Server: The error logs indicate that you're using Rudder Server version 1.22.1. If there's a newer version available, consider updating to see if the issue has been resolved in a later release.

Regarding the "invalid api key" error for Bugsnag, it seems like the API key provided to Bugsnag is not valid. You might want to check if the API key is correctly set in your configuration¹. If you're using different environments (like development and production), make sure to use the correct API key for each environment¹.

If none of these steps resolve the issue, you might want to consider reaching out to the Rudder Server community or support for more specific guidance. They might be able to provide more insight based on the specifics of your setup and the exact error messages you're seeing²⁵.

(1) Troubleshooting :: Rudder Documentation. https://docs.rudder.io/reference/8.1/administration/troubleshooting.html.
(2) javascript - Bugsnag: Invalid API key 'undefined' - Stack Overflow. https://stackoverflow.com/questions/42693851/bugsnag-invalid-api-key-undefined.
(3) Docker setup is broken · Issue #448 · rudderlabs/rudder-server. #448.
(4) backupDSLoop exception: mark end of backup operation: driver: bad .... #4535.
(5) Bugsnag | RudderStack Docs. https://www.rudderstack.com/docs/destinations/streaming-destinations/bugsnag/.
(6) undefined. https://docs.rudderstack.com/get-started/installing-and-setting-up-rudderstack/docker.
(7) undefined. segment-boneyard/analytics.js-integrations#61.
(8) undefined. https://d[2](https://stackoverflow.com/questions/42693851/bugsnag-invalid-api-key-undefined)wy8f7a9ursnm.cloudfront.net/bugsnag-3.min.js.