When the messages cached in the stream are empty, kill -9 nats-server and start again, and firstSeq and firstSeq become 0

[GuangchaoDeng]

Defect

When the messages cached in the stream are empty, kill -9 nats-server and start again, and firstSeq and firstSeq become 0

Versions of nats-server and affected client libraries used:

nats-server : 2.2.1
nats : 0.0.22

OS/Container environment:

Red Hat Enterprise Linux Server release 7.2 (Maipo) or other

Steps or code to reproduce the issue:

Describe：

1、Build a leaf node and persist it in file mode

2、Create stream:denggc8, Consumer :denggc8, for subject: denggc8

nats -s nats://leaf_1:leaf_1@localhost:5222 str add denggc8 --subjects="denggc8 " --storage=file --retention=workq --discard=new --max-msgs=-1 --max-bytes=-1 --max-age=-1 --max-msg-size=-1 --dupe-window=5s --replicas=1

nats -s nats://leaf_1:leaf_1@localhost:5222 con add denggc8 denggc8 2 --deliver=all --replay=original --filter="denggc8 " --max-deliver=-1 --max-pending=1 --pull

3、Send 10 messages to Stream Denggc8; And through Consumer: Denggc8 to complete the consumption

stream和consumer info：

4、kill -9 leaf node nats-server

5、start leaf node nats-server again

6、stream和consumer info：

Expected result:

Actual result:

[ripienaar]

The kill -9 will corrupt the index, startup will rebuild the index from scratch.

Because the stream is empty this will result in a reset state for sequences. We might need to do some resetting on consumers or derive the correct sequences some other way. Not sure what our options are

The tl;dr here is:

• WQ stream
• Consume all the messages so consumers advance in sequences
• kill -9 the server
• Start it, it will rebuild the WQ index
• Consumers now report sequence 10 while stream claims its max sequence is 0

/cc @derekcollison

[derekcollison]

There were some improvements to be made but in the case above it seems like the index file may have been non existent completely. In that case we can not at the moment recover the sequences.

The PR does fix a small bug when rebuilding to properly assess the last sequence number.

[GuangchaoDeng]

i see request #2099 merged.I get the latest mater code and build it.I test the above issue，Maybe the result is worse。when kill -9 nats-server ,sometime the stream may become the below：

 Subjects: t3
    Acknowledgements: true
           Retention: File - WorkQueue
            Replicas: 1
      Discard Policy: New
    Duplicate Window: 2m0s
    Maximum Messages: unlimited
       Maximum Bytes: unlimited
         Maximum Age: 0.00s
Maximum Message Size: unlimited
   Maximum Consumers: unlimited

State:

            Messages: 0
               Bytes: 0 B
            FirstSeq: 0
             LastSeq: -1
    Active Consumers: 1

The stream becomes unavailable. When I request the first message, the seq id becomes 0. I can’t get the message anymore, when I create a new consumer。

          Subjects: t3
    Acknowledgements: true
           Retention: File - WorkQueue
            Replicas: 1
      Discard Policy: New
    Duplicate Window: 2m0s
    Maximum Messages: unlimited
       Maximum Bytes: unlimited
         Maximum Age: 0.00s
Maximum Message Size: unlimited
   Maximum Consumers: unlimited

State:
            Messages: 5
               Bytes: 165 B
            FirstSeq: 0 @ 2021-04-12T10:56:52 UTC
             LastSeq: 4 @ 2021-04-12T10:57:01 UTC
    Active Consumers: 0

I don't know if it's correct。
In the past, my response to this situation was to delete the consumer and rebuild; then get the message from the original stream and pass the new consumer。Because I use it at the edge, it will inevitably lose power suddenly。 @derekcollison

[derekcollison]

Will address the -1.

The kill -9 should be fine in general since after purge we write the new index file, so not sure why your index file keeps getting removed or never written in the first place.

[GuangchaoDeng]

ok，I changed to the latest compiled natscli, it is correct；