InfluxDB starts for 2 hours

[undera]

This is the continuation of the issue #5764 . The machine I use and database I use is the same.

I have upgraded influxdb to 0.12 from debian (0.12.0-1). Now the start takes slightly over 2 hours. 33% increase is not bad, but still unacceptable to have downtime of 2 hours just to start. My dream is to replace MySQL server with InfluxDB, because I see InfluxDB is better to store my timeline data. MySQL starts within several seconds (I will keep memory consumption comparison for separate thread).

Can we improve InfluxDB to beat MySQL? :) I have my feeling that the long startup (and memory consumption also) is the price of being "schemaless", but in fact there is a schema that is kept in the memory. Loading that schema at startup takes long time and a lot of RAM (for certain cases). I might be wrong with my assumptions, sorry for being overconfident.

I've looked into logs, seems there's pretty long shard loads happening, I've grepped from log:
tsm1] 2016/04/06 21:43:56 /var/lib/influxdb/data/Loadosophia/default/512 database index loaded in 11m2.839406822s
[tsm1] 2016/04/06 21:47:00 /var/lib/influxdb/data/Loadosophia/default/513 database index loaded in 14m7.359681031s
[tsm1] 2016/04/06 21:54:48 /var/lib/influxdb/data/Loadosophia/default/525 database index loaded in 21m12.196889089s
[[tsm1] 2016/04/06 22:11:10 /var/lib/influxdb/data/Loadosophia/default/524 database index loaded in 27m14.029874898s
[tsm1] 2016/04/06 22:45:11 /var/lib/influxdb/data/Loadosophia/default/523 database index loaded in 1h12m5.557480759s
[tsm1] 2016/04/06 23:41:10 /var/lib/influxdb/data/Loadosophia/default/527 database index loaded in 1h54m9.795156493s

Attaching full startup log.
influxd.20160407.txt

[jwilder]

Would you be able to run:

influx_inspect dumptsmdev /var/lib/influxdb/data/Loadosophia/default/527/000000058-000000002.tsm?

There is a lot of lock contention related to loading the meta data index still. I suspect you are hitting that.

[undera]

undera@undera-desktop:~$ influx_inspect dumptsmdev /var/lib/influxdb/data/Loadosophia/default/527/000000058-000000002.tsm
Error opening TSM files:  init: read tombstones: invalid argument
panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x518173]

goroutine 1 [running]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*TSMReader).Close(0x0, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:262 +0xd3
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*TSMReader).TimeRange(0x0, 0x27, 0x27)
        /tmp/influxdb/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:306 +0x5d
main.cmdDumpTsm1dev(0xc20800aea0)
        /tmp/influxdb/src/github.com/influxdata/influxdb/cmd/influx_inspect/tsm.go:481 +0x2c3
main.main()
        /tmp/influxdb/src/github.com/influxdata/influxdb/cmd/influx_inspect/main.go:115 +0xf82

undera@undera-desktop:~$ ls -la /var/lib/influxdb/data/Loadosophia/default/527/000000058-000000002.tsm
-rw-r--r-- 1 influxdb influxdb 139593214 Feb 16 10:08 /var/lib/influxdb/data/Loadosophia/default/527/000000058-000000002.tsm

[jwilder]

@undera Can you run ls -la /var/lib/influxdb/data/Loadosophia/default/527/? I've never seen that error before.

[undera]

undera@undera-desktop:~$ ls -la /var/lib/influxdb/data/Loadosophia/default/527/
total 136332
drwxr-xr-x   2 influxdb influxdb      4096 Feb 16 13:42 .
drwxr-xr-x 219 influxdb influxdb      4096 Feb 16 13:54 ..
-rw-------   1 influxdb influxdb         0 Feb 16 13:42 000000058-000000002.tombstone
-rw-r--r--   1 influxdb influxdb 139593214 Feb 16 10:08 000000058-000000002.tsm

[jwilder]

@undera Looks like a permissions issue. The file is owned by influxdb, but you might not be running influx_inspect as influxdb user. Try running it as influxdb user.

[undera]

Good to know. Probably error diagnostics could be improved to hint me the permissions issue.

Here's the output with sudo:

undera@undera-desktop:~$ sudo influx_inspect dumptsmdev /var/lib/influxdb/data/Loadosophia/default/527/000000058-000000002.tsm
[sudo] password for undera: 
Summary:
  File: /var/lib/influxdb/data/Loadosophia/default/527/000000058-000000002.tsm
  Time Range: 2015-03-23T00:00:00Z - 2015-03-29T23:59:59Z
  Duration: 167h59m59s   Series: 355019   File Size: 139593214

Statistics
  Blocks:
    Total: 407136 Size: 113188590 Min: 20 Max: 8426 Avg: 278
  Index:
    Total: 407136 Size: 26404611
  Points:
    Total: 73529019
  Encoding:
    Timestamp:  none: 0 (0%)    s8b: 192803 (47%)       rle: 214333 (52%) 
    Float:      none: 0 (0%)    gor: 41792 (10%) 
    Int:        none: 0 (0%)    s8b: 346613 (85%)       rle: 18731 (4%) 
  Compression:
    Per block: 1.54 bytes/point
    Total: 1.90 bytes/point

[jwilder]

@undera It shouldn't panic like that. That's a separate bug.

How many shards do you have in /var/lib/influxdb/data/Loadosophia/default/?

[undera]

I have 217 of subfolders there

[aderumier]

Hi,
I still have slow start time, even with 0.13-rc2.

https://gist.github.com/aderumier/63921e366ac3eae5cc73965075f04370

around 1h.
I have around 115GB datas on ssd (intel s3610). cpu was aroud 400% of 16cores when starting.