Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic when starting influxd after upgrade #19958

Closed
russorat opened this issue Nov 9, 2020 · 1 comment · Fixed by #19959
Closed

Panic when starting influxd after upgrade #19958

russorat opened this issue Nov 9, 2020 · 1 comment · Fixed by #19959
Assignees
@danxmoran

Comments

@russorat
Copy link
Contributor

russorat commented Nov 9, 2020 

➜  darwin git:(master) ✗ ./influxd upgrade --bolt-path /Users/rsavage/workspace/influxdb_testing/.influxdbv2/influxd.bolt -b default --config-file /Users/rsavage/workspace/influxdb_testing/default.conf --engine-path /Users/rsavage/workspace/influxdb_testing/.influxdbv2/engine --force --influx-configs-path /Users/rsavage/workspace/influxdb_testing/.influxdbv2/configs --org influxdata --password something --retention 24h --username russ -v
{"level":"info","ts":1604964431.031484,"caller":"upgrade/upgrade.go:331","msg":"Starting InfluxDB 1.x upgrade"}
{"level":"info","ts":1604964431.0316002,"caller":"upgrade/upgrade.go:334","msg":"Upgrading config file","file":"/Users/rsavage/workspace/influxdb_testing/default.conf"}
{"level":"info","ts":1604964431.0330062,"caller":"upgrade/config.go:110","msg":"Config file upgraded.","1.x config":"/Users/rsavage/workspace/influxdb_testing/default.conf","2.x config":"/Users/rsavage/workspace/influxdb_testing/config.toml"}
{"level":"info","ts":1604964431.03307,"caller":"upgrade/upgrade.go:359","msg":"Upgrade source paths","meta":"/Users/rsavage/workspace/influxdb_testing/.influxdb/meta","data":"/Users/rsavage/workspace/influxdb_testing/.influxdb/data"}
{"level":"info","ts":1604964431.033092,"caller":"upgrade/upgrade.go:360","msg":"Upgrade target paths","bolt":"/Users/rsavage/workspace/influxdb_testing/.influxdbv2/influxd.bolt","engine":"/Users/rsavage/workspace/influxdb_testing/.influxdbv2/engine"}
{"level":"info","ts":1604964431.062824,"caller":"bolt/bbolt.go:67","msg":"Resources opened","service":"bolt","path":"/Users/rsavage/workspace/influxdb_testing/.influxdbv2/influxd.bolt"}
{"level":"info","ts":1604964431.076789,"caller":"migration/migration.go:241","msg":"Migration \"initial migration\" started (up)","service":"migrations"}
{"level":"info","ts":1604964431.671076,"caller":"migration/migration.go:241","msg":"Migration \"initial migration\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964431.671315,"caller":"migration/migration.go:241","msg":"Migration \"add index \\\"userresourcemappingsbyuserindexv1\\\"\" started (up)","service":"migrations"}
{"level":"info","ts":1604964431.7134452,"caller":"migration/migration.go:241","msg":"Migration \"add index \\\"userresourcemappingsbyuserindexv1\\\"\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964431.7135239,"caller":"migration/migration.go:241","msg":"Migration \"migrate task owner id\" started (up)","service":"migrations"}
{"level":"info","ts":1604964431.741972,"caller":"migration/migration.go:241","msg":"Migration \"migrate task owner id\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964431.7421231,"caller":"migration/migration.go:241","msg":"Migration \"create DBRP buckets\" started (up)","service":"migrations"}
{"level":"info","ts":1604964431.8123991,"caller":"migration/migration.go:241","msg":"Migration \"create DBRP buckets\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964431.812513,"caller":"migration/migration.go:241","msg":"Migration \"create pkger stacks buckets\" started (up)","service":"migrations"}
{"level":"info","ts":1604964431.8686872,"caller":"migration/migration.go:241","msg":"Migration \"create pkger stacks buckets\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964431.868775,"caller":"migration/migration.go:241","msg":"Migration \"delete sessionsv1 bucket\" started (up)","service":"migrations"}
{"level":"info","ts":1604964431.91352,"caller":"migration/migration.go:241","msg":"Migration \"delete sessionsv1 bucket\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964431.913623,"caller":"migration/migration.go:241","msg":"Migration \"Create TSM metadata buckets\" started (up)","service":"migrations"}
{"level":"info","ts":1604964431.9554691,"caller":"migration/migration.go:241","msg":"Migration \"Create TSM metadata buckets\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964431.9555352,"caller":"migration/migration.go:241","msg":"Migration \"Create Legacy authorization buckets\" started (up)","service":"migrations"}
{"level":"info","ts":1604964432.0115788,"caller":"migration/migration.go:241","msg":"Migration \"Create Legacy authorization buckets\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964432.011751,"caller":"migration/migration.go:241","msg":"Migration \"Create legacy auth password bucket\" started (up)","service":"migrations"}
{"level":"info","ts":1604964432.053447,"caller":"migration/migration.go:241","msg":"Migration \"Create legacy auth password bucket\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964432.053536,"caller":"migration/migration.go:241","msg":"Migration \"add index \\\"telegrafbyorgindexv1\\\"\" started (up)","service":"migrations"}
{"level":"info","ts":1604964432.096574,"caller":"migration/migration.go:241","msg":"Migration \"add index \\\"telegrafbyorgindexv1\\\"\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964432.096725,"caller":"migration/migration.go:241","msg":"Migration \"populate dashboards owner id\" started (up)","service":"migrations"}
{"level":"info","ts":1604964432.124474,"caller":"migration/migration.go:241","msg":"Migration \"populate dashboards owner id\" completed (up)","service":"migrations"}
{"level":"info","ts":1604964432.38989,"caller":"upgrade/setup.go:153","msg":"CLI config has been stored.","path":"/Users/rsavage/workspace/influxdb_testing/.influxdbv2/configs"}
{"level":"info","ts":1604964432.389997,"caller":"upgrade/database.go:35","msg":"Checking space"}
{"level":"info","ts":1604964432.400511,"caller":"upgrade/database.go:51","msg":"Disk space info","Free space":"67 GB","Requested space":"70 MB"}
{"level":"info","ts":1604964432.400586,"caller":"upgrade/database.go:58","msg":"Upgrading databases"}
{"level":"info","ts":1604964432.400606,"caller":"upgrade/database.go:65","msg":"Skipping _internal "}
{"level":"info","ts":1604964432.4006221,"caller":"upgrade/database.go:70","msg":"Upgrading database ","database":"telegraf"}
{"level":"info","ts":1604964432.400656,"caller":"upgrade/database.go:89","msg":"Creating bucket ","Bucket":"telegraf/autogen"}
{"level":"info","ts":1604964432.414589,"caller":"upgrade/database.go:100","msg":"Creating database with retention policy","database":"ed882bb8cfeb5df8"}
{"level":"info","ts":1604964432.428616,"caller":"upgrade/database.go:118","msg":"Creating mapping","database":"telegraf","retention policy":"autogen","orgID":"8689ee2ad682de4a","bucketID":"ed882bb8cfeb5df8"}
{"level":"info","ts":1604964432.442621,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"ed882bb8cfeb5df8","retention policy":"autogen","time":1604275200}
{"level":"info","ts":1604964432.456573,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"ed882bb8cfeb5df8","retention policy":"autogen","time":1604880000}
{"level":"info","ts":1604964432.471543,"caller":"upgrade/database.go:146","msg":"Copying data","source":"/Users/rsavage/workspace/influxdb_testing/.influxdb/data/telegraf/autogen","target":"/Users/rsavage/workspace/influxdb_testing/.influxdbv2/engine/data/ed882bb8cfeb5df8/autogen"}
{"level":"info","ts":1604964432.4879491,"caller":"upgrade/database.go:161","msg":"Copying wal","source":"/Users/rsavage/workspace/influxdb_testing/.influxdb/wal/telegraf/autogen","target":"/Users/rsavage/workspace/influxdb_testing/.influxdbv2/engine/wal/ed882bb8cfeb5df8/autogen"}
{"level":"info","ts":1604964432.510431,"caller":"upgrade/security.go:40","msg":"User is admin and will not be upgraded.","username":"admin"}
{"level":"info","ts":1604964432.547447,"caller":"upgrade/security.go:97","msg":"User upgraded.","username":"russ"}
{"level":"info","ts":1604964432.619276,"caller":"upgrade/security.go:97","msg":"User upgraded.","username":"savage"}
{"level":"info","ts":1604964432.619342,"caller":"upgrade/upgrade.go:436","msg":"Upgrade successfully completed. Start service now"}
➜  darwin git:(master) ✗ INFLUXD_CONFIG_PATH=~/workspace/influxdb_testing/config.toml ./influxd
2020-11-09T23:31:21.810872Z	info	Welcome to InfluxDB	{"log_id": "0QNqRhKW000", "version": "dev", "commit": "cae539055f", "build_date": "2020-11-09T23:31:16Z"}
2020-11-09T23:31:21.829566Z	info	Resources opened	{"log_id": "0QNqRhKW000", "service": "bolt", "path": "/Users/rsavage/workspace/influxdb_testing/.influxdbv2/influxd.bolt"}
2020-11-09T23:31:21.963288Z	info	Checking InfluxDB metadata for prior version.	{"log_id": "0QNqRhKW000", "bolt_path": "/Users/rsavage/workspace/influxdb_testing/.influxdbv2/influxd.bolt"}
2020-11-09T23:31:21.963669Z	info	Using data dir	{"log_id": "0QNqRhKW000", "service": "storage-engine", "path": "/Users/rsavage/workspace/influxdb_testing/.influxdbv2/engine/data"}
2020-11-09T23:31:21.963761Z	info	Compaction settings	{"log_id": "0QNqRhKW000", "service": "storage-engine", "max_concurrent_compactions": 4, "throughput_bytes_per_second": 50331648, "throughput_bytes_per_second_burst": 50331648}
2020-11-09T23:31:21.963795Z	info	Open store (start)	{"log_id": "0QNqRhKW000", "service": "storage-engine", "op_name": "tsdb_open", "op_event": "start"}
2020-11-09T23:31:22.060066Z	info	Opened shard	{"log_id": "0QNqRhKW000", "service": "storage-engine", "op_name": "tsdb_open", "index_version": "inmem", "path": "/Users/rsavage/workspace/influxdb_testing/.influxdbv2/engine/data/ed882bb8cfeb5df8/autogen/5", "duration": "25.568ms"}
2020-11-09T23:31:22.080717Z	info	Opened shard	{"log_id": "0QNqRhKW000", "service": "storage-engine", "op_name": "tsdb_open", "index_version": "inmem", "path": "/Users/rsavage/workspace/influxdb_testing/.influxdbv2/engine/data/ed882bb8cfeb5df8/autogen/2", "duration": "46.211ms"}
2020-11-09T23:31:22.080808Z	info	Open store (end)	{"log_id": "0QNqRhKW000", "service": "storage-engine", "op_name": "tsdb_open", "op_event": "end", "op_elapsed": "117.016ms"}
2020-11-09T23:31:22.080835Z	info	Starting retention policy enforcement service	{"log_id": "0QNqRhKW000", "service": "retention", "check_interval": "30m"}
2020-11-09T23:31:22.080850Z	info	Starting precreation service	{"log_id": "0QNqRhKW000", "service": "shard-precreation", "check_interval": "10m", "advance_period": "30m"}
2020-11-09T23:31:22.080919Z	info	Starting query controller	{"log_id": "0QNqRhKW000", "service": "storage-reads", "concurrency_quota": 9223372036854775807, "initial_memory_bytes_quota_per_query": 9223372036854775807, "memory_bytes_quota_per_query": 9223372036854775807, "max_memory_bytes": 0, "queue_size": 10}
panic: sync: negative WaitGroup counter

goroutine 1 [running]:
sync.(*WaitGroup).Add(0xc0005e0990, 0x7fffffffffffffff)
	/usr/local/Cellar/go/1.15.2/libexec/src/sync/waitgroup.go:74 +0x147
github.com/influxdata/influxdb/v2/query/control.New(0x7fffffffffffffff, 0x0, 0x7fffffffffffffff, 0x0, 0xa, 0xc0011e7ce0, 0x0, 0x0, 0x0, 0xc0008261b0, ...)
	/Users/rsavage/go/src/github.com/influxdata/influxdb/query/control/controller.go:189 +0x7f5
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.(*Launcher).run(0xc0001e6e00, 0x760c700, 0xc0004f8d40, 0x0, 0x0)
	/Users/rsavage/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/launcher.go:876 +0x3998
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1(0x0, 0x0)
	/Users/rsavage/go/src/github.com/influxdata/influxdb/cmd/influxd/launcher/launcher.go:148 +0x14c
github.com/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc00016d080, 0x9b905a0, 0x0, 0x0, 0x0, 0x0)
	/Users/rsavage/go/src/github.com/influxdata/influxdb/kit/cli/viper.go:62 +0x29
github.com/spf13/cobra.(*Command).execute(0xc00016d080, 0xc0001181c0, 0x0, 0x0, 0xc00016d080, 0xc0001181c0)
	/Users/rsavage/go/src/github.com/influxdata/influxdb/vendor/github.com/spf13/cobra/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc00016d080, 0xc000b9ff58, 0x1, 0x1)
	/Users/rsavage/go/src/github.com/influxdata/influxdb/vendor/github.com/spf13/cobra/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/rsavage/go/src/github.com/influxdata/influxdb/vendor/github.com/spf13/cobra/command.go:887
main.main()
	/Users/rsavage/go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:47 +0x19e

here is my config.toml:

bolt-path = "/Users/rsavage/workspace/influxdb_testing/.influxdbv2/influxd.bolt"
engine-path = "/Users/rsavage/workspace/influxdb_testing/.influxdbv2/engine"
http-bind-address = ":8086"
influxql-max-select-buckets = 0
influxql-max-select-point = 0
influxql-max-select-series = 0
log-level = "info"
query-concurrency = 9223372036854775807
reporting-disabled = false
storage-cache-max-memory-size = 1073741824
storage-cache-snapshot-memory-size = 26214400
storage-cache-snapshot-write-cold-duration = "10m0s"
storage-compact-full-write-cold-duration = "4h0m0s"
storage-compact-throughput-burst = 50331648
storage-max-concurrent-compactions = 0
storage-max-index-log-file-size = 1048576
storage-retention-check-interval = "30m0s"
storage-series-file-max-concurrent-snapshot-compactions = 0
storage-series-id-set-cache-size = 100
storage-shard-precreator-advance-period = "30m0s"
storage-shard-precreator-check-interval = "10m0s"
storage-tsm-use-madv-willneed = false
storage-validate-keys = false
storage-wal-fsync-delay = "0s"
tls-cert = "/etc/ssl/influxdb.pem"
tls-key = ""
@russorat
Copy link
Contributor Author

russorat commented Nov 9, 2020

changing query-concurrency = 10 in my config fixes this

@danxmoran danxmoran self-assigned this Nov 9, 2020
@danxmoran danxmoran mentioned this issue Nov 9, 2020
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danxmoran danxmoran

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(upgrade): use 10 instead of MaxInt when rewriting query-concurrency influxdata/influxdb
2 participants
@danxmoran @russorat