HBASE-28287 MOB HFiles are expired earlier than their reference data #5599

liuwenjing17 · 2023-12-28T07:29:57Z

No description provided.

Apache-HBase · 2023-12-28T08:21:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 11s	master passed
+1 💚	compile	2m 48s	master passed
+1 💚	checkstyle	0m 51s	master passed
+1 💚	spotless	0m 54s	branch has no errors when running spotless:check.
+1 💚	spotbugs	2m 1s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 36s	the patch passed
+1 💚	compile	2m 45s	the patch passed
+1 💚	javac	2m 45s	the patch passed
+1 💚	checkstyle	0m 37s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 21s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 55s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 9s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		40m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5599
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 7bdbfef3d261 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `75801a5`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-12-28T11:38:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 5s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 2s	master passed
+1 💚	compile	0m 43s	master passed
+1 💚	shadedjars	5m 15s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 41s	the patch passed
+1 💚	compile	0m 44s	the patch passed
+1 💚	javac	0m 44s	the patch passed
+1 💚	shadedjars	5m 11s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	the patch passed
		_ Other Tests _
+1 💚	unit	214m 39s	hbase-server in the patch passed.
		237m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5599
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0a3959e0c838 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `75801a5`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/1/testReport/
Max. process+thread count	4555 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-12-28T12:33:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 42s	master passed
+1 💚	compile	0m 37s	master passed
+1 💚	shadedjars	5m 50s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 57s	the patch passed
+1 💚	compile	0m 43s	the patch passed
+1 💚	javac	0m 43s	the patch passed
+1 💚	shadedjars	5m 55s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	the patch passed
		_ Other Tests _
-1 ❌	unit	268m 53s	hbase-server in the patch failed.
		293m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5599
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 18a5cf57db08 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `75801a5`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/1/testReport/
Max. process+thread count	4582 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestExpiredMobFileCleaner.java

Apache9 · 2024-01-04T13:39:54Z

I still do not fully understand the problem here...

If we do not set millis to zero, it will only affect the life time of a MOB file for less than 1 second, how could it make the MOB file expire 2 hours earlier?

liuwenjing17 · 2024-01-05T03:25:39Z

I still do not fully understand the problem here...

If we do not set millis to zero, it will only affect the life time of a MOB file for less than 1 second, how could it make the MOB file expire 2 hours earlier?

Because in org.apache.hadoop.hbase.mob.MobUtils, the creation time of mob files is obtained by parsing their names from fileName using the statement (Date fileDate = parseDate(MobFileName.getDateFromName(fileName));). For instance, data created on 20240105, their timestamps will be parsed as 1704384000000 (2024-01-05 00:00:00). In this way, when the master expired mob thread starts, it may affect the life time of a MOB file for less than 1 day.

Apache9 · 2024-01-05T10:03:01Z

I still do not fully understand the problem here...
If we do not set millis to zero, it will only affect the life time of a MOB file for less than 1 second, how could it make the MOB file expire 2 hours earlier?

Because in org.apache.hadoop.hbase.mob.MobUtils, the creation time of mob files is obtained by parsing their names from fileName using the statement (Date fileDate = parseDate(MobFileName.getDateFromName(fileName));). For instance, data created on 20240105, their timestamps will be parsed as 1704384000000 (2024-01-05 00:00:00). In this way, when the master expired mob thread starts, it may affect the life time of a MOB file for less than 1 day.

Then the problem is we should use a timestamp instead of '20240105' in the mob file name? I still do not understand why setting MILLISECOND to 0 can solve the problem...

liuwenjing17 · 2024-01-05T11:33:33Z

I still do not fully understand the problem here...
If we do not set millis to zero, it will only affect the life time of a MOB file for less than 1 second, how could it make the MOB file expire 2 hours earlier?

Because in org.apache.hadoop.hbase.mob.MobUtils, the creation time of mob files is obtained by parsing their names from fileName using the statement (Date fileDate = parseDate(MobFileName.getDateFromName(fileName));). For instance, data created on 20240105, their timestamps will be parsed as 1704384000000 (2024-01-05 00:00:00). In this way, when the master expired mob thread starts, it may affect the life time of a MOB file for less than 1 day.

Then the problem is we should use a timestamp instead of '20240105' in the mob file name? I still do not understand why setting MILLISECOND to 0 can solve the problem...

Here is an example:

Assume the Time-To-Live (TTL) for the mob data is set to 1 day.
We write mob data at 18:33 on 01/04/2023, and the data is flushed to a mob file named xxxx20230104xxxx.
The mob expiration thread starts within 1 day, at 10:45 on 01/05/2023.
When checking ts, the standard expired timestamp is calculated as (currentTS - 1day parsed by Calendar) : 1704297600720 (2024-01-04 00:00:00)(3 random digits when only set to SECOND level)
The mob file's ts, parsed from its name, is : 1704297600000 (2024-01-04 00:00:00)
if (fileDate.getTime() < expireDate.getTime()) {/* expired */}
if statement is true, indicating that the mob file has expired, and it will be cleaned. These 3 random digits cause the mob files to be cleaned earlier than expected.
But if we set to MILLISECOND level, the expireDate.getTime() will be 1704297600000, and if statement will be false. In this case, the mob file will be retained as intended.

And here is the link to jira: https://issues.apache.org/jira/browse/HBASE-28287

Apache9 · 2024-01-05T13:33:44Z

OK. Got it. Thanks for the explaination.

Apache-HBase · 2024-01-10T11:25:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 18s	master passed
+1 💚	compile	2m 35s	master passed
+1 💚	checkstyle	0m 34s	master passed
+1 💚	spotless	0m 45s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 33s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 55s	the patch passed
+1 💚	compile	2m 35s	the patch passed
+1 💚	javac	2m 35s	the patch passed
+1 💚	checkstyle	0m 35s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	5m 29s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 42s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 39s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		30m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5599
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 0b3bbb291ffd 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `a09305d`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	80 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-01-10T14:58:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 41s	master passed
+1 💚	compile	0m 36s	master passed
+1 💚	shadedjars	5m 21s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 20s	the patch passed
+1 💚	compile	0m 37s	the patch passed
+1 💚	javac	0m 37s	the patch passed
+1 💚	shadedjars	5m 18s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
+1 💚	unit	221m 13s	hbase-server in the patch passed.
		243m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5599
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux d6d3aff6000e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `a09305d`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/2/testReport/
Max. process+thread count	4607 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-01-10T14:59:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 4s	master passed
+1 💚	compile	0m 45s	master passed
+1 💚	shadedjars	5m 27s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 27s	the patch passed
+1 💚	compile	0m 51s	the patch passed
+1 💚	javac	0m 51s	the patch passed
+1 💚	shadedjars	6m 19s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	218m 49s	hbase-server in the patch passed.
		243m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5599
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 62079c1caf4b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `a09305d`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/2/testReport/
Max. process+thread count	5122 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5599/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…5599) Co-authored-by: liuwenjing3 <liuwenjing3@360.cn> Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 4d2b8f8)

Fix the inaccurate expired time of MOB HFiles

aa5965a

Apache9 changed the title ~~Fix the inaccurate expired time of MOB HFiles~~ HBASE-28287 Fix the inaccurate expired time of MOB HFiles Jan 3, 2024

Apache9 reviewed Jan 3, 2024

View reviewed changes

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestExpiredMobFileCleaner.java Outdated Show resolved Hide resolved

wchevreuil approved these changes Jan 5, 2024

View reviewed changes

Fix the inaccurate expired time of MOB HFiles (comment)

a7803a9

Apache9 approved these changes Jan 10, 2024

View reviewed changes

Apache9 changed the title ~~HBASE-28287 Fix the inaccurate expired time of MOB HFiles~~ HBASE-28287 MOB HFiles are expired earlier than their reference data Jan 13, 2024

Apache9 merged commit 4d2b8f8 into apache:master Jan 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-28287 MOB HFiles are expired earlier than their reference data #5599

HBASE-28287 MOB HFiles are expired earlier than their reference data #5599

Uh oh!

liuwenjing17 commented Dec 28, 2023

Uh oh!

Apache-HBase commented Dec 28, 2023

Uh oh!

Apache-HBase commented Dec 28, 2023

Uh oh!

Apache-HBase commented Dec 28, 2023

Uh oh!

Uh oh!

Apache9 commented Jan 4, 2024

Uh oh!

liuwenjing17 commented Jan 5, 2024

Uh oh!

Apache9 commented Jan 5, 2024 •

edited

Loading

Uh oh!

liuwenjing17 commented Jan 5, 2024

Uh oh!

Apache9 commented Jan 5, 2024

Uh oh!

Apache-HBase commented Jan 10, 2024

Uh oh!

Apache-HBase commented Jan 10, 2024

Uh oh!

Apache-HBase commented Jan 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HBASE-28287 MOB HFiles are expired earlier than their reference data #5599

HBASE-28287 MOB HFiles are expired earlier than their reference data #5599

Uh oh!

Conversation

liuwenjing17 commented Dec 28, 2023

Uh oh!

Apache-HBase commented Dec 28, 2023

Uh oh!

Apache-HBase commented Dec 28, 2023

Uh oh!

Apache-HBase commented Dec 28, 2023

Uh oh!

Uh oh!

Apache9 commented Jan 4, 2024

Uh oh!

liuwenjing17 commented Jan 5, 2024

Uh oh!

Apache9 commented Jan 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liuwenjing17 commented Jan 5, 2024

Uh oh!

Apache9 commented Jan 5, 2024

Uh oh!

Apache-HBase commented Jan 10, 2024

Uh oh!

Apache-HBase commented Jan 10, 2024

Uh oh!

Apache-HBase commented Jan 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Apache9 commented Jan 5, 2024 •

edited

Loading