HBASE-24387 TableSnapshotInputFormatImpl support row limit on each InputSplit #1731
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nyl3532016
changed the title
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
nyl3532016
force-pushed
the
HBASE-24387
branch
2 times, most recently
from
2e413d5 to
c4c3e4c
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@HorizonNet @virajjasani mind help review this PR? |
HorizonNet
approved these changes
May 19, 2020
| } | ||
| } | ||
|
|
||
|
|
There was a problem hiding this comment.
NIT: Can you please remove this additional empty line?
|
Oh, was wrong. Jenkins already reports it as green. |
infraio
reviewed
May 20, 2020
| @@ -274,6 +274,7 @@ message Scan { | |||
| } | |||
| optional ReadType readType = 23 [default = DEFAULT]; | |||
| optional bool need_cursor_result = 24 [default = false]; | |||
| optional uint32 limit = 25; | |||
There was a problem hiding this comment.
I thought this is duplicate with ScanRequest protobuf? We already have one field to means "limit".
There was a problem hiding this comment.
Do we need to add a new filed to means "scan limit"?
There was a problem hiding this comment.
yes, duplicate field may confuse user
nyl3532016
force-pushed
the
HBASE-24387
branch
2 times, most recently
from
b4e1f97 to
3dcefa2
Compare
|
🎊 +1 overall
This message was automatically generated. |
nyl3532016
changed the title
infraio
reviewed
May 20, 2020
| Job job = new Job(UTIL.getConfiguration()); | ||
| Path tmpTableDir = UTIL.getDataTestDirOnTestFS(snapshotName); | ||
| // limit the scan | ||
| Scan scan = new Scan().setLimit(10); |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
HorizonNet
reviewed
May 20, 2020
| @@ -304,6 +305,56 @@ public void testWithMockedMapReduceWithNoStartRowStopRow() throws Exception { | |||
| } | |||
| } | |||
|
|
|||
| @Test | |||
| public void testScanLimit() throws Exception { | |||
| setupCluster(); | |||
There was a problem hiding this comment.
Does it make sense to move this one to a test initializer method? I see it used in several tests in here.
There was a problem hiding this comment.
It is better to submit another PR to fix?after this one merged, I will amend it
There was a problem hiding this comment.
It's up to you. We can also do it in a separate ticket.
There was a problem hiding this comment.
Would you like to help merge this PR? Thanks
| UTIL.getConfiguration().unset(SNAPSHOT_INPUTFORMAT_ROW_LIMIT_PER_INPUTSPLIT); | ||
| UTIL.getAdmin().deleteSnapshot(snapshotName); | ||
| UTIL.deleteTable(tableName); | ||
| tearDownCluster(); |
asfgit
pushed a commit
that referenced
this pull request
May 25, 2020
…putSplit (#1731) Signed-off-by: Guanghao Zhang <zghao@apache.org>
asfgit
pushed a commit
that referenced
this pull request
May 25, 2020
…putSplit (#1731) Signed-off-by: Guanghao Zhang <zghao@apache.org>
asfgit
pushed a commit
that referenced
this pull request
May 25, 2020
…putSplit (#1731) Signed-off-by: Guanghao Zhang <zghao@apache.org>
HomminLee
pushed a commit
to HomminLee/hbase
that referenced
this pull request
Jun 12, 2020
…putSplit (apache#1731) Signed-off-by: Guanghao Zhang <zghao@apache.org>
clarax
pushed a commit
to clarax/hbase
that referenced
this pull request
Nov 15, 2020
…putSplit (apache#1731) Signed-off-by: Guanghao Zhang <zghao@apache.org>
wchevreuil
pushed a commit
to wchevreuil/hbase
that referenced
this pull request
May 24, 2021
…putSplit (apache#1731) Signed-off-by: Guanghao Zhang <zghao@apache.org> (cherry picked from commit fff59f4) Change-Id: I48ee436d0daa357af0d3d324ae80e334db450ddd
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In some scenario , We want to scan limited rows on each InputSplit for sampling data extraction