HBASE-29722 (backport portion of HBASE-27558) Coproc - data integrity issues for scan with heavy filters #7470

virajjasani · 2025-11-19T10:48:46Z

…ckport portion of HBASE-27558)

virajjasani · 2025-11-19T11:02:12Z

Test that reproduce this issue apache/phoenix#2319

It fails with

java.lang.AssertionError: Should find exactly 900 records with amount in range 
Expected :900
Actual   :0

	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.apache.phoenix.end2end.IndexWithScanSizeIT.testScanWithBetweenCondition(IndexWithScanSizeIT.java:279)

sanjeet006py

Nice find!!

apurtell

Please add a unit test.

apurtell · 2025-11-20T18:26:00Z

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java

          limitReached = sizeLimitReached || timeLimitReached || resultsLimitReached;

          if (limitReached || !moreRows) {
+            // With block size limit, we may exceed size limit without collecting any results.


Please improve this comment.

This comment is from backport

apurtell · 2025-11-20T18:27:23Z

I will be on vacation for two weeks starting tomorrow. Another committer should feel free to merge after my comment is addressed.

virajjasani · 2025-11-21T02:12:23Z

@apurtell, @sanjeet006py and I have been trying to reproduce the issue in hbase directly but it seems we are not able to. This does not impact direct scan on the table.

virajjasani · 2025-11-21T02:14:10Z

@rmdmattingly @bbeaudreault any suggestions on whether we should merge this directly based on branch-2.6 coverage? Although i am not sure if 2.6+ also have coverage for this.

Apache-HBase · 2025-11-21T07:41:03Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ branch-2.5 Compile Tests _
+1 💚	mvninstall	3m 51s		branch-2.5 passed
+1 💚	compile	2m 59s		branch-2.5 passed
+1 💚	checkstyle	0m 38s		branch-2.5 passed
+1 💚	spotbugs	1m 37s		branch-2.5 passed
+1 💚	spotless	0m 47s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 2s		the patch passed
+1 💚	compile	2m 56s		the patch passed
+1 💚	javac	2m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		the patch passed
+1 💚	spotbugs	1m 44s		the patch passed
+1 💚	hadoopcheck	22m 55s		Patch does not cause any errors with Hadoop 2.10.2 or 3.2.4 3.3.6 3.4.1.
+1 💚	spotless	0m 46s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 12s		The patch does not generate ASF License warnings.
		44m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7470
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux e8cf45359f64 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.5 / `aab0cde`
Default Java	Eclipse Adoptium-11.0.23+9
Max. process+thread count	77 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

stoty · 2025-11-21T08:58:20Z

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java

          boolean sizeLimitReached = scannerContext.checkSizeLimit(LimitScope.BETWEEN_ROWS);
          boolean timeLimitReached = scannerContext.checkTimeLimit(LimitScope.BETWEEN_ROWS);
          boolean resultsLimitReached = numOfResults >= maxResults;
          limitReached = sizeLimitReached || timeLimitReached || resultsLimitReached;


Not in the scope of this backport ticket, but why do we check only those two limits ?
is batch size not applicable here ? Why do we ignore the heap size limit ?

Maybe worth looking into this in another JIRA.

stoty

+1 LGTM

Apache-HBase · 2025-11-21T10:28:04Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
-0 ⚠️	yetus	0m 6s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2.5 Compile Tests _
+1 💚	mvninstall	2m 51s		branch-2.5 passed
+1 💚	compile	0m 40s		branch-2.5 passed
+1 💚	javadoc	0m 24s		branch-2.5 passed
+1 💚	shadedjars	5m 22s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 20s		the patch passed
+1 💚	compile	0m 40s		the patch passed
+1 💚	javac	0m 40s		the patch passed
+1 💚	javadoc	0m 23s		the patch passed
+1 💚	shadedjars	5m 17s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	188m 38s		hbase-server in the patch passed.
		211m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#7470
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux f02acab66d0d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.5 / `aab0cde`
Default Java	Temurin-1.8.0_412-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/testReport/
Max. process+thread count	2736 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-11-21T10:34:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
-0 ⚠️	yetus	0m 6s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2.5 Compile Tests _
+1 💚	mvninstall	3m 52s		branch-2.5 passed
+1 💚	compile	0m 59s		branch-2.5 passed
+1 💚	javadoc	0m 28s		branch-2.5 passed
+1 💚	shadedjars	6m 28s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 18s		the patch passed
+1 💚	compile	1m 0s		the patch passed
+1 💚	javac	1m 0s		the patch passed
+1 💚	javadoc	0m 29s		the patch passed
+1 💚	shadedjars	6m 27s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	190m 39s		hbase-server in the patch passed.
		218m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7470
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 86a9ef0f38b0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.5 / `aab0cde`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/testReport/
Max. process+thread count	3225 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-11-21T11:58:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
-0 ⚠️	yetus	0m 5s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2.5 Compile Tests _
+1 💚	mvninstall	4m 36s		branch-2.5 passed
+1 💚	compile	0m 54s		branch-2.5 passed
+1 💚	javadoc	0m 36s		branch-2.5 passed
+1 💚	shadedjars	8m 5s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 2s		the patch passed
+1 💚	compile	1m 9s		the patch passed
+1 💚	javac	1m 9s		the patch passed
+1 💚	javadoc	0m 33s		the patch passed
+1 💚	shadedjars	8m 14s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	265m 35s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
		302m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#7470
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 984059fd17ba 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.5 / `aab0cde`
Default Java	Eclipse Adoptium-11.0.23+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/testReport/
Max. process+thread count	3755 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7470/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

HBASE-29722 Fix data integrity issues for scan with heavy filters (ba…

76f6ac0

…ckport portion of HBASE-27558)

virajjasani requested review from Apache9 and bbeaudreault November 19, 2025 10:48

This comment has been minimized.

Sign in to view

virajjasani requested a review from apurtell November 19, 2025 16:44

sanjeet006py approved these changes Nov 19, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

Umeshkumar9414 approved these changes Nov 20, 2025

View reviewed changes

mnpoonia approved these changes Nov 20, 2025

View reviewed changes

apurtell requested changes Nov 20, 2025

View reviewed changes

virajjasani requested a review from rmdmattingly November 21, 2025 02:10

add test

aab0cde

virajjasani changed the title ~~HBASE-29722 Fix data integrity issues for scan with heavy filters (backport portion of HBASE-27558)~~ HBASE-29722 (backport portion of HBASE-27558) Coproc - data integrity issues for scan with heavy filters Nov 21, 2025

virajjasani requested a review from stoty November 21, 2025 08:47

stoty reviewed Nov 21, 2025

View reviewed changes

stoty approved these changes Nov 21, 2025

View reviewed changes

virajjasani merged commit 94fd568 into apache:branch-2.5 Nov 21, 2025
1 check failed

HBASE-29722 (backport portion of HBASE-27558) Coproc - data integrity issues for scan with heavy filters #7470

HBASE-29722 (backport portion of HBASE-27558) Coproc - data integrity issues for scan with heavy filters #7470

Conversation

virajjasani commented Nov 19, 2025

Uh oh!

virajjasani commented Nov 19, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sanjeet006py left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

apurtell left a comment

Choose a reason for hiding this comment

Uh oh!

apurtell Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

virajjasani Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

apurtell commented Nov 20, 2025

Uh oh!

virajjasani commented Nov 21, 2025

Uh oh!

virajjasani commented Nov 21, 2025

Uh oh!

Apache-HBase commented Nov 21, 2025

Uh oh!

stoty Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

stoty left a comment

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Nov 21, 2025

Uh oh!

Apache-HBase commented Nov 21, 2025

Uh oh!

Apache-HBase commented Nov 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants