Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable index range scan of string data type in graph layer #2283

Merged
merged 4 commits into from
Sep 17, 2020
Merged

Disable index range scan of string data type in graph layer #2283

merged 4 commits into from
Sep 17, 2020

Conversation

bright-starry-sky
Copy link
Contributor

It was splitted from PR #2277 . Only disable string range scan and contains

liuyu85cn
liuyu85cn previously approved these changes Aug 17, 2020
Copy link
Contributor

@liuyu85cn liuyu85cn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

return Status::OK();
}

bool LookupExecutor::dataTypeCheckForRange(nebula::cpp2::SupportedType type) const {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe "rangeSupportedDataType" is a better name ?

@jude-zhu jude-zhu modified the milestones: R201910_RC3, v1.0.2 Aug 24, 2020
@bright-starry-sky bright-starry-sky changed the title Disable string range scan in graph lyear Disable create index when field type is string Aug 25, 2020
@bright-starry-sky
Copy link
Contributor Author

String type index is completely rejected

@bright-starry-sky bright-starry-sky added the doc affected PR: improvements or additions to documentation label Aug 25, 2020
@codecov-commenter
Copy link

Codecov Report

Merging #2283 into master will decrease coverage by 0.35%.
The diff coverage is 73.89%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2283      +/-   ##
==========================================
- Coverage   86.46%   86.11%   -0.36%     
==========================================
  Files         649      648       -1     
  Lines       64374    64478     +104     
==========================================
- Hits        55662    55523     -139     
- Misses       8712     8955     +243     
Impacted Files Coverage Δ
src/common/hdfs/HdfsCommandHelper.cpp 0.00% <0.00%> (-75.00%) ⬇️
src/graph/DownloadExecutor.cpp 0.00% <0.00%> (ø)
src/graph/DropSpaceExecutor.h 0.00% <ø> (ø)
src/graph/FetchEdgesExecutor.cpp 79.32% <0.00%> (-0.27%) ⬇️
src/graph/FetchVerticesExecutor.cpp 78.21% <0.00%> (-0.18%) ⬇️
src/graph/FindPathExecutor.cpp 81.42% <0.00%> (-0.17%) ⬇️
src/graph/GoExecutor.h 91.30% <ø> (-1.56%) ⬇️
src/graph/IngestExecutor.cpp 0.00% <0.00%> (ø)
src/graph/LookupExecutor.h 0.00% <ø> (ø)
src/graph/test/TestBase.h 70.00% <ø> (ø)
... and 92 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8390267...487f359. Read the comment docs.

@bright-starry-sky
Copy link
Contributor Author

CI failed of test case log_cas_test (Timeout). retry again.

@bright-starry-sky
Copy link
Contributor Author

After offline discussion, we have other way to solve the performance problem of string types “==” condition.
So revert the commit for disable create index for string type.

@bright-starry-sky bright-starry-sky changed the title Disable create index when field type is string Disable index range scan of string data type in graph layer Aug 27, 2020
@CLAassistant
Copy link

CLAassistant commented Sep 2, 2020

CLA assistant check
All committers have signed the CLA.

critical27
critical27 previously approved these changes Sep 15, 2020
liuyu85cn
liuyu85cn previously approved these changes Sep 15, 2020
Copy link
Contributor

@liuyu85cn liuyu85cn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dangleptr
Copy link
Contributor

Please help check the failed UT. @critical27

@dangleptr
Copy link
Contributor

Please resolve the conflicts

@darionyaphet
Copy link
Contributor

LGTM

@bright-starry-sky
Copy link
Contributor Author

resolved conflict.

@bright-starry-sky
Copy link
Contributor Author

One test case failed in the last CI, seems that the wal is missing . I saved the failing logs. let me analyze it at later.

2020-09-17T03:54:53.6522502Z I0917 11:54:43.104784 20385 RaftPart.cpp:1292] [Port: 2534, Space: 0, Part: 2] Recieved a VOTING request: space = 0, partition = 2, candidateAddr = 127.0.0.1:12895, term = 1, lastLogId = 0, lastLogTerm = 0
2020-09-17T03:54:53.6523437Z I0917 11:54:43.104830 20385 RaftPart.cpp:1325] [Port: 2534, Space: 0, Part: 2] The partition currently is a Follower, lastLogId 0, lastLogTerm 0, committedLogId 0, term 0
2020-09-17T03:54:53.6524223Z I0917 11:54:43.104851 20385 RaftPart.cpp:1394] [Port: 2534, Space: 0, Part: 2] The partition will vote for the candidate
2020-09-17T03:54:53.6525091Z I0917 11:54:43.104842 20422 RaftPart.cpp:1292] [Port: 15264, Space: 0, Part: 2] Recieved a VOTING request: space = 0, partition = 2, candidateAddr = 127.0.0.1:12895, term = 1, lastLogId = 0, lastLogTerm = 0
2020-09-17T03:54:53.6526170Z I0917 11:54:43.104869 20422 RaftPart.cpp:1325] [Port: 15264, Space: 0, Part: 2] The partition currently is a Follower, lastLogId 0, lastLogTerm 0, committedLogId 0, term 0
2020-09-17T03:54:53.6526897Z I0917 11:54:43.104884 20422 RaftPart.cpp:1394] [Port: 15264, Space: 0, Part: 2] The partition will vote for the candidate
2020-09-17T03:54:53.6527466Z I0917 11:54:43.105144 20353 RaftPart.cpp:1084] [Port: 12895, Space: 0, Part: 2] Partition is elected as the new leader for term 1
2020-09-17T03:54:53.6528030Z I0917 11:54:43.105162 20353 RaftPart.cpp:1179] [Port: 12895, Space: 0, Part: 2] The partition is elected as the leader
2020-09-17T03:54:53.6528688Z I0917 11:54:43.105407 20353 InMemoryLogBuffer.h:23] [Port: 12895, Space: 0, Part: 2] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6529533Z I0917 11:54:43.105492 20357 Host.cpp:149] [Port: 12895, Space: 0, Part: 2] [Host: 127.0.0.1:2534] This is the first time to send the logs to this host, lastLogIdSent = 0, lastLogTermSent = 0
2020-09-17T03:54:53.6530355Z I0917 11:54:43.105854 20357 Host.cpp:149] [Port: 12895, Space: 0, Part: 2] [Host: 127.0.0.1:15264] This is the first time to send the logs to this host, lastLogIdSent = 0, lastLogTermSent = 0
2020-09-17T03:54:53.6531093Z I0917 11:54:43.106472 20385 RaftPart.cpp:1679] [Port: 2534, Space: 0, Part: 2] The current role is Follower. Will follow the new leader 127.0.0.1:12895 [Term: 1]
2020-09-17T03:54:53.6531840Z I0917 11:54:43.106488 20422 RaftPart.cpp:1679] [Port: 15264, Space: 0, Part: 2] The current role is Follower. Will follow the new leader 127.0.0.1:12895 [Term: 1]
2020-09-17T03:54:53.6532380Z I0917 11:54:43.106587 20388 Part.cpp:182] [Port: 2534, Space: 0, Part: 2] Find the new leader [127.0.0.1:12895]
2020-09-17T03:54:53.6532805Z I0917 11:54:43.106640 20425 Part.cpp:182] [Port: 15264, Space: 0, Part: 2] Find the new leader [127.0.0.1:12895]
2020-09-17T03:54:53.6533398Z I0917 11:54:43.106673 20422 InMemoryLogBuffer.h:23] [Port: 15264, Space: 0, Part: 2] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6534152Z I0917 11:54:43.106693 20385 InMemoryLogBuffer.h:23] [Port: 2534, Space: 0, Part: 2] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6534842Z I0917 11:54:43.219772 20389 RaftPart.cpp:991] [Port: 2534, Space: 0, Part: 1] Start leader election, reason: lastMsgDur 287, term 0
2020-09-17T03:54:53.6535773Z I0917 11:54:43.219899 20389 RaftPart.cpp:1124] [Port: 2534, Space: 0, Part: 1] Sending out an election request (space = 0, part = 1, term = 1, lastLogId = 0, lastLogTerm = 0, candidateIP = 127.0.0.1, candidatePort = 2534)
2020-09-17T03:54:53.6536733Z I0917 11:54:43.221312 20348 RaftPart.cpp:1292] [Port: 12895, Space: 0, Part: 1] Recieved a VOTING request: space = 0, partition = 1, candidateAddr = 127.0.0.1:2534, term = 1, lastLogId = 0, lastLogTerm = 0
2020-09-17T03:54:53.6537640Z I0917 11:54:43.221351 20348 RaftPart.cpp:1325] [Port: 12895, Space: 0, Part: 1] The partition currently is a Follower, lastLogId 0, lastLogTerm 0, committedLogId 0, term 0
2020-09-17T03:54:53.6538359Z I0917 11:54:43.221369 20348 RaftPart.cpp:1394] [Port: 12895, Space: 0, Part: 1] The partition will vote for the candidate
2020-09-17T03:54:53.6539085Z I0917 11:54:43.221465 20422 RaftPart.cpp:1292] [Port: 15264, Space: 0, Part: 1] Recieved a VOTING request: space = 0, partition = 1, candidateAddr = 127.0.0.1:2534, term = 1, lastLogId = 0, lastLogTerm = 0
2020-09-17T03:54:53.6539980Z I0917 11:54:43.221509 20422 RaftPart.cpp:1325] [Port: 15264, Space: 0, Part: 1] The partition currently is a Follower, lastLogId 0, lastLogTerm 0, committedLogId 0, term 0
2020-09-17T03:54:53.6540702Z I0917 11:54:43.221531 20422 RaftPart.cpp:1394] [Port: 15264, Space: 0, Part: 1] The partition will vote for the candidate
2020-09-17T03:54:53.6541272Z I0917 11:54:43.221678 20389 RaftPart.cpp:1084] [Port: 2534, Space: 0, Part: 1] Partition is elected as the new leader for term 1
2020-09-17T03:54:53.6541919Z I0917 11:54:43.221709 20389 RaftPart.cpp:1179] [Port: 2534, Space: 0, Part: 1] The partition is elected as the leader
2020-09-17T03:54:53.6542566Z I0917 11:54:43.221936 20389 InMemoryLogBuffer.h:23] [Port: 2534, Space: 0, Part: 1] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6543354Z I0917 11:54:43.222041 20394 Host.cpp:149] [Port: 2534, Space: 0, Part: 1] [Host: 127.0.0.1:12895] This is the first time to send the logs to this host, lastLogIdSent = 0, lastLogTermSent = 0
2020-09-17T03:54:53.6544174Z I0917 11:54:43.222401 20394 Host.cpp:149] [Port: 2534, Space: 0, Part: 1] [Host: 127.0.0.1:15264] This is the first time to send the logs to this host, lastLogIdSent = 0, lastLogTermSent = 0
2020-09-17T03:54:53.6544916Z I0917 11:54:43.222977 20348 RaftPart.cpp:1679] [Port: 12895, Space: 0, Part: 1] The current role is Follower. Will follow the new leader 127.0.0.1:2534 [Term: 1]
2020-09-17T03:54:53.6545534Z I0917 11:54:43.223064 20353 Part.cpp:182] [Port: 12895, Space: 0, Part: 1] Find the new leader [127.0.0.1:2534]
2020-09-17T03:54:53.6546067Z I0917 11:54:43.223085 20422 RaftPart.cpp:1679] [Port: 15264, Space: 0, Part: 1] The current role is Follower. Will follow the new leader 127.0.0.1:2534 [Term: 1]
2020-09-17T03:54:53.6546782Z I0917 11:54:43.223160 20348 InMemoryLogBuffer.h:23] [Port: 12895, Space: 0, Part: 1] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6547364Z I0917 11:54:43.223183 20426 Part.cpp:182] [Port: 15264, Space: 0, Part: 1] Find the new leader [127.0.0.1:2534]
2020-09-17T03:54:53.6547963Z I0917 11:54:43.223290 20422 InMemoryLogBuffer.h:23] [Port: 15264, Space: 0, Part: 1] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6548840Z I0917 11:54:43.383102 20425 RaftPart.cpp:991] [Port: 15264, Space: 0, Part: 0] Start leader election, reason: lastMsgDur 438, term 0
2020-09-17T03:54:53.6549641Z I0917 11:54:43.383167 20425 RaftPart.cpp:1124] [Port: 15264, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 1, lastLogId = 0, lastLogTerm = 0, candidateIP = 127.0.0.1, candidatePort = 15264)
2020-09-17T03:54:53.6550598Z I0917 11:54:43.384413 20348 RaftPart.cpp:1292] [Port: 12895, Space: 0, Part: 0] Recieved a VOTING request: space = 0, partition = 0, candidateAddr = 127.0.0.1:15264, term = 1, lastLogId = 0, lastLogTerm = 0
2020-09-17T03:54:53.6551499Z I0917 11:54:43.384444 20348 RaftPart.cpp:1325] [Port: 12895, Space: 0, Part: 0] The partition currently is a Follower, lastLogId 0, lastLogTerm 0, committedLogId 0, term 0
2020-09-17T03:54:53.6552391Z I0917 11:54:43.384446 20385 RaftPart.cpp:1292] [Port: 2534, Space: 0, Part: 0] Recieved a VOTING request: space = 0, partition = 0, candidateAddr = 127.0.0.1:15264, term = 1, lastLogId = 0, lastLogTerm = 0
2020-09-17T03:54:53.6553136Z I0917 11:54:43.384461 20348 RaftPart.cpp:1394] [Port: 12895, Space: 0, Part: 0] The partition will vote for the candidate
2020-09-17T03:54:53.6553857Z I0917 11:54:43.384487 20385 RaftPart.cpp:1325] [Port: 2534, Space: 0, Part: 0] The partition currently is a Follower, lastLogId 0, lastLogTerm 0, committedLogId 0, term 0
2020-09-17T03:54:53.6554588Z I0917 11:54:43.384577 20385 RaftPart.cpp:1394] [Port: 2534, Space: 0, Part: 0] The partition will vote for the candidate
2020-09-17T03:54:53.6555179Z I0917 11:54:43.384593 20351 RaftPart.cpp:991] [Port: 12895, Space: 0, Part: 0] Start leader election, reason: lastMsgDur 0, term 0
2020-09-17T03:54:53.6556062Z I0917 11:54:43.384647 20351 RaftPart.cpp:1124] [Port: 12895, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 2, lastLogId = 0, lastLogTerm = 0, candidateIP = 127.0.0.1, candidatePort = 12895)
2020-09-17T03:54:53.6556853Z I0917 11:54:43.384800 20425 RaftPart.cpp:1084] [Port: 15264, Space: 0, Part: 0] Partition is elected as the new leader for term 1
2020-09-17T03:54:53.6557532Z I0917 11:54:43.384817 20425 RaftPart.cpp:1179] [Port: 15264, Space: 0, Part: 0] The partition is elected as the leader
2020-09-17T03:54:53.6558214Z I0917 11:54:43.385079 20425 InMemoryLogBuffer.h:23] [Port: 15264, Space: 0, Part: 0] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6559016Z I0917 11:54:43.385176 20431 Host.cpp:149] [Port: 15264, Space: 0, Part: 0] [Host: 127.0.0.1:12895] This is the first time to send the logs to this host, lastLogIdSent = 0, lastLogTermSent = 0
2020-09-17T03:54:53.6559793Z I0917 11:54:43.385567 20431 Host.cpp:149] [Port: 15264, Space: 0, Part: 0] [Host: 127.0.0.1:2534] This is the first time to send the logs to this host, lastLogIdSent = 0, lastLogTermSent = 0
2020-09-17T03:54:53.6560648Z I0917 11:54:43.385700 20385 RaftPart.cpp:1292] [Port: 2534, Space: 0, Part: 0] Recieved a VOTING request: space = 0, partition = 0, candidateAddr = 127.0.0.1:12895, term = 2, lastLogId = 0, lastLogTerm = 0
2020-09-17T03:54:53.6561502Z I0917 11:54:43.385728 20385 RaftPart.cpp:1325] [Port: 2534, Space: 0, Part: 0] The partition currently is a Follower, lastLogId 0, lastLogTerm 0, committedLogId 0, term 0
2020-09-17T03:54:53.6562186Z I0917 11:54:43.385744 20385 RaftPart.cpp:1394] [Port: 2534, Space: 0, Part: 0] The partition will vote for the candidate
2020-09-17T03:54:53.6562923Z I0917 11:54:43.385761 20422 RaftPart.cpp:1292] [Port: 15264, Space: 0, Part: 0] Recieved a VOTING request: space = 0, partition = 0, candidateAddr = 127.0.0.1:12895, term = 2, lastLogId = 0, lastLogTerm = 0
2020-09-17T03:54:53.6563821Z I0917 11:54:43.385797 20422 RaftPart.cpp:1325] [Port: 15264, Space: 0, Part: 0] The partition currently is a Leader, lastLogId 0, lastLogTerm 0, committedLogId 0, term 1
2020-09-17T03:54:53.6564487Z I0917 11:54:43.385813 20422 RaftPart.cpp:1394] [Port: 15264, Space: 0, Part: 0] The partition will vote for the candidate
2020-09-17T03:54:53.6565121Z I0917 11:54:43.386107 20351 RaftPart.cpp:1084] [Port: 12895, Space: 0, Part: 0] Partition is elected as the new leader for term 2
2020-09-17T03:54:53.6565742Z I0917 11:54:43.386135 20351 RaftPart.cpp:1179] [Port: 12895, Space: 0, Part: 0] The partition is elected as the leader
2020-09-17T03:54:53.6566366Z I0917 11:54:43.386358 20351 InMemoryLogBuffer.h:23] [Port: 12895, Space: 0, Part: 0] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6567042Z I0917 11:54:43.386412 20385 RaftPart.cpp:1679] [Port: 2534, Space: 0, Part: 0] The current role is Follower. Will follow the new leader 127.0.0.1:15264 [Term: 1]
2020-09-17T03:54:53.6567676Z I0917 11:54:43.386422 20348 RaftPart.cpp:1637] [Port: 12895, Space: 0, Part: 0] The current role is Leader. The local term is 2. The remote term is not newer
2020-09-17T03:54:53.6568377Z I0917 11:54:43.386458 20359 Host.cpp:149] [Port: 12895, Space: 0, Part: 0] [Host: 127.0.0.1:2534] This is the first time to send the logs to this host, lastLogIdSent = 0, lastLogTermSent = 0
2020-09-17T03:54:53.6569151Z I0917 11:54:43.386497 20391 Part.cpp:182] [Port: 2534, Space: 0, Part: 0] Find the new leader [127.0.0.1:15264]
2020-09-17T03:54:53.6569744Z I0917 11:54:43.386612 20385 InMemoryLogBuffer.h:23] [Port: 2534, Space: 0, Part: 0] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6570531Z I0917 11:54:43.386906 20359 Host.cpp:149] [Port: 12895, Space: 0, Part: 0] [Host: 127.0.0.1:15264] This is the first time to send the logs to this host, lastLogIdSent = 0, lastLogTermSent = 0
2020-09-17T03:54:53.6571175Z I0917 11:54:43.386993 20422 RaftPart.cpp:876] [Port: 15264, Space: 0, Part: 0] The leader has changed
2020-09-17T03:54:53.6571771Z E0917 11:54:43.387034 20422 RaftPart.cpp:909] [Port: 15264, Space: 0, Part: 0] processAppendLogResponses failed!
2020-09-17T03:54:53.6573125Z E0917 11:54:43.387271 20431 Host.cpp:390] [Port: 15264, Space: 0, Part: 0] [Host: 127.0.0.1:12895] Failed to append logs to the host (Err: -6)
2020-09-17T03:54:53.6573715Z I0917 11:54:43.387501 20385 RaftPart.cpp:1648] [Port: 2534, Space: 0, Part: 0] I believe the leader [127.0.0.1:15264] exists. Refuse to append logs of [127.0.0.1:12895]
2020-09-17T03:54:53.6574381Z I0917 11:54:43.387589 20422 RaftPart.cpp:1679] [Port: 15264, Space: 0, Part: 0] The current role is Follower. Will follow the new leader 127.0.0.1:12895 [Term: 2]
2020-09-17T03:54:53.6575122Z I0917 11:54:43.387643 20422 RaftPart.cpp:1694] [Port: 15264, Space: 0, Part: 0] There is one log 1 i did not commit when i was leader, rollback to 0
2020-09-17T03:54:53.6575893Z I0917 11:54:43.388758 20422 InMemoryLogBuffer.h:27] [Port: 15264, Space: 0, Part: 0] InMemoryLogBuffer dtor, firstLogId 1
2020-09-17T03:54:53.6576466Z I0917 11:54:43.388826 20426 Part.cpp:182] [Port: 15264, Space: 0, Part: 0] Find the new leader [127.0.0.1:12895]
2020-09-17T03:54:53.6577070Z I0917 11:54:43.388881 20422 InMemoryLogBuffer.h:23] [Port: 15264, Space: 0, Part: 0] InMemoryLogBuffer ctor, firstLogId 1
2020-09-17T03:54:53.6578195Z I0917 11:54:43.446902 14810 NebulaStoreTest.cpp:638] Let's write some logs...
2020-09-17T03:54:53.6578788Z I0917 11:54:43.446969 14810 NebulaStoreTest.cpp:649] Put some data then read them...
2020-09-17T03:54:53.6579456Z I0917 11:54:43.447340 20356 Host.cpp:149] [Port: 12895, Space: 0, Part: 0] [Host: 127.0.0.1:2534] This is the first time to send the logs to this host, lastLogIdSent = 1, lastLogTermSent = 2
2020-09-17T03:54:53.6580131Z I0917 11:54:48.449075 14810 NebulaStoreTest.cpp:680] Check the data on [127.0.0.1:12895] for part 0
2020-09-17T03:54:53.6580641Z I0917 11:54:48.449230 14810 NebulaStoreTest.cpp:680] Check the data on [127.0.0.1:2534] for part 0
2020-09-17T03:54:53.6581172Z /__w/nebula/nebula/src/kvstore/test/NebulaStoreTest.cpp:696: Failure
2020-09-17T03:54:53.6581563Z       Expected: 100
2020-09-17T03:54:53.6581788Z To be equal to: num
2020-09-17T03:54:53.6582004Z       Which is: 0
2020-09-17T03:54:53.6582362Z I0917 11:54:48.449360 14810 NebulaStore.cpp:31] Cut off the relationship with meta client

@dangleptr dangleptr merged commit 9154dde into vesoft-inc:master Sep 17, 2020
@bright-starry-sky bright-starry-sky deleted the disable_string_range_in_graph branch September 21, 2020 01:54
tong-hao pushed a commit to tong-hao/nebula that referenced this pull request Jun 1, 2021
…nc#2283)

* Disable string range scan in graph lyear

* rename mothod dataTypeCheckForRange to supportedDataTypeForRange

* fixed typo error

* Does not support left expr and right expr are both kAliasProp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc affected PR: improvements or additions to documentation ready-for-testing PR: ready for the CI test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants