fix a node can't accept more logs after receiving snapshot

[critical27]

What type of PR is this?

• bug
• feature
• enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:

See the logs below, logs unrelated has been ignored:

1. A node has received a snapshot of committedLogId_ 14958, committedLogTerm_ 27,
2. It append one more log successfully (see line 5). The lastLogId_ is 14959 now
3. Leader send a appendLogRequest with lastLogIdSent = 14965 (line 7), we will return LOG_GAP, and ask leader send logs from committedLogId_ (14958)
4. Leader do send logs from 14958, however, the weird things happened:
   Since out first log id of wal is 14959, we don't have the log of 14958, so the condition below always is true:
   wal_->getLogTerm(req.get_last_log_id_sent()) != req.get_last_log_term_sent(), so we ask leader send logs from committedLogId_(14958) again
5. Repeat step 4 forever

1 RaftPart.cpp:1795] [Port: 9780, Space: 1, Part: 170] Begin to receive the snapshot
2 Part.cpp:464] [Port: 9780, Space: 1, Part: 170] Clean rocksdb part data
3 RaftPart.cpp:1823] [Port: 9780, Space: 1, Part: 170] Receive all snapshot, committedLogId_ 14958, committedLogTerm_ 27, lastLodId 14958, lastLogTermId 27
4 Part.cpp:206] [Port: 9780, Space: 1, Part: 170] Find the new leader "store1":9780
5 RaftPart.cpp:1471] [Port: 9780, Space: 1, Part: 170] Received logAppend: GraphSpaceId = 1, partition = 170, leaderIp = store1, leaderPort = 9780, current_term = 27, committedLogId = 14958, lastLogIdSent = 14958, lastLogTermSent = 27, num_logs = 1, local lastLogId = 14958, local lastLogTerm = 27, local committedLogId = 14958, local current term = 27, wal lastLogId = 0
6 RaftPart.cpp:1677] [Port: 9780, Space: 1, Part: 170] The current role is Follower. Will follow the new leader "store1":9780 on term 28
7 RaftPart.cpp:1471] [Port: 9780, Space: 1, Part: 170] Received logAppend: GraphSpaceId = 1, partition = 170, leaderIp = store1, leaderPort = 9780, current_term = 28, committedLogId = 14975, lastLogIdSent = 14965, lastLogTermSent = 27, num_logs = 11, local lastLogId = 14959, local lastLogTerm = 27, local committedLogId = 14958, local current term = 28, wal lastLogId = 14959
8 RaftPart.cpp:1471] [Port: 9780, Space: 1, Part: 170] Received logAppend: GraphSpaceId = 1, partition = 170, leaderIp = store1, leaderPort = 9780, current_term = 28, committedLogId = 14975, lastLogIdSent = 14958, lastLogTermSent = 27, num_logs = 18, local lastLogId = 14959, local lastLogTerm = 27, local committedLogId = 14958, local current term = 28, wal lastLogId = 14959
9 WalFileIterator.cpp:24] [Port: 9780, Space: 1, Part: 170] The given log id 14958 is out of the range, the wal firstLogId is 14959
10 RaftPart.cpp:1471] [Port: 9780, Space: 1, Part: 170] Received logAppend: GraphSpaceId = 1, partition = 170, leaderIp = store1, leaderPort = 9780, current_term = 28, committedLogId = 14975, lastLogIdSent = 14958, lastLogTermSent = 27, num_logs = 18, local lastLogId = 14959, local lastLogTerm = 27, local committedLogId = 14958, local current term = 28, wal lastLogId = 14959
11 WalFileIterator.cpp:24] [Port: 9780, Space: 1, Part: 170] The given log id 14958 is out of the range, the wal firstLogId is 14959
12 RaftPart.cpp:1471] [Port: 9780, Space: 1, Part: 170] Received logAppend: GraphSpaceId = 1, partition = 170, leaderIp = store1, leaderPort = 9780, current_term = 28, committedLogId = 14975, lastLogIdSent = 14958, lastLogTermSent = 27, num_logs = 18, local lastLogId = 14959, local lastLogTerm = 27, local committedLogId = 14958, local current term = 28, wal lastLogId = 14959
13 WalFileIterator.cpp:24] [Port: 9780, Space: 1, Part: 170] The given log id 14958 is out of the range, the wal firstLogId is 14959
...

How do you solve it?

When we failed to get a log from wal (for example, we pull snapshot), we could use committedLogId_ and committedLogTerm_ to do raft log matching.

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

• standalone test environment

Affects:

• Documentation affected (Please add the label if documentation needs to be modified.)
• Incompatibility (If it breaks the compatibility, please describe it and add the label.）
• If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
• Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:
Fix a node can't accept more logs after receiving snapshot

[darionyaphet]

so case 1 is last_log_id < committedLogId_ and case 2 is wal_->lastLogId() < get_last_log_id_sent ?

[critical27]

Yeah, see the comment above in line 1539

[critical27]

Don't merge plz, test is still on-going

[critical27]

Ready to go