Skip to content

Commit 773d1c8

Browse files
sunhellybbeaudreault
authored andcommitted
HBASE-27152 Under compaction mark may leak (apache#4725)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
1 parent c77c8fe commit 773d1c8

File tree

2 files changed

+13
-8
lines changed

2 files changed

+13
-8
lines changed

RELEASENOTES.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,7 @@ After this feature, there are two implementations of StoreFileTrackers. The firs
191191

192192
This feature is notable in that it better enables HBase to function on storage systems which do not provide the typical posix filesystem semantics, most importantly, those which do not implement a file rename operation which is atomic. Storage systems which do not implement atomic renames often implement a rename as a copy and delete operation which amplifies the I/O costs by 2x.
193193

194-
At scale, this feature should have a 2x reduction in I/O costs when using storage systems that do not provide atomic renames, most importantly in HBase compactions and memstore flushes. See the corresponding section, "Store File Tracking", in the HBase book for more information on how to use this feature.
194+
At scale, this feature should have a 2x reduction in I/O costs when using storage systems that do not provide atomic renames, most importantly in HBase compactions and memstore flushes. See the corresponding section, "Store File Tracking", in the HBase book for more information on how to use this feature.
195195

196196
The file based StoreFileTracker, FileBasedStoreFileTracker, is currently incompatible with the Medium Objects (MOB) feature. Do not enable them together.
197197

@@ -359,12 +359,12 @@ If not present, master local region will use the cluster level store file tracke
359359

360360
Introduced two shell commands for change table's or family's sft:
361361

362-
change\_sft:
362+
change\_sft:
363363
Change table's or table column family's sft. Examples:
364364
hbase\> change\_sft 't1','FILE'
365365
hbase\> change\_sft 't2','cf1','FILE'
366366

367-
change\_sft\_all:
367+
change\_sft\_all:
368368
Change all of the tables's sft matching the given regex:
369369
hbase\> change\_sft\_all 't.\*','FILE'
370370
hbase\> change\_sft\_all 'ns:.\*','FILE'
@@ -375,7 +375,7 @@ change\_sft\_all:
375375

376376
* [HBASE-26742](https://issues.apache.org/jira/browse/HBASE-26742) | *Major* | **Comparator of NOT\_EQUAL NULL is invalid for checkAndMutate**
377377

378-
The semantics of checkAndPut for null(or empty) value comparator is changed, the old match is always true.
378+
The semantics of checkAndPut for null(or empty) value comparator is changed, the old match is always true.
379379
But we should consider that EQUAL or NOT\_EQUAL for null check is a common usage, so the semantics of checkAndPut for matching null is correct now.
380380
There is rare use of LESS or GREATER null, so keep the semantics for them.
381381

@@ -471,7 +471,7 @@ Now we will upload the site artifacts to nightlies for nightly build as well as
471471

472472
* [HBASE-26316](https://issues.apache.org/jira/browse/HBASE-26316) | *Minor* | **Per-table or per-CF compression codec setting overrides**
473473

474-
It is now possible to specify codec configuration options as part of table or column family schema definitions. The configuration options will only apply to the defined scope. For example:
474+
It is now possible to specify codec configuration options as part of table or column family schema definitions. The configuration options will only apply to the defined scope. For example:
475475

476476
hbase\> create 'sometable', \\
477477
{ NAME =\> 'somefamily', COMPRESSION =\> 'ZSTD' }, \\
@@ -781,7 +781,7 @@ belong to system RSGroup only.
781781

782782
* [HBASE-25902](https://issues.apache.org/jira/browse/HBASE-25902) | *Critical* | **Add missing CFs in meta during HBase 1 to 2.3+ Upgrade**
783783

784-
While upgrading cluster from 1.x to 2.3+ versions, after the active master is done setting it's status as 'Initialized', it attempts to add 'table' and 'repl\_barrier' CFs in meta. Once CFs are added successfully, master is aborted with PleaseRestartMasterException because master has missed certain initialization events (e.g ClusterSchemaService is not initialized and tableStateManager fails to migrate table states from ZK to meta due to missing CFs). Subsequent active master initialization is expected to be smooth.
784+
While upgrading cluster from 1.x to 2.3+ versions, after the active master is done setting it's status as 'Initialized', it attempts to add 'table' and 'repl\_barrier' CFs in meta. Once CFs are added successfully, master is aborted with PleaseRestartMasterException because master has missed certain initialization events (e.g ClusterSchemaService is not initialized and tableStateManager fails to migrate table states from ZK to meta due to missing CFs). Subsequent active master initialization is expected to be smooth.
785785
In the presence of multi masters, when one of them becomes active for the first time after upgrading to HBase 2.3+, it is aborted after fixing CFs in meta and one of the other backup masters will take over and become active soon. Hence, overall this is expected to be smooth upgrade if we have backup masters configured. If not, operator is expected to restart same master again manually.
786786

787787

@@ -1053,7 +1053,7 @@ Expose HBCK repost results in metrics, includes: "orphanRegionsOnRS", "orphanReg
10531053

10541054
* [HBASE-25582](https://issues.apache.org/jira/browse/HBASE-25582) | *Major* | **Support setting scan ReadType to be STREAM at cluster level**
10551055

1056-
Adding a new meaning for the config 'hbase.storescanner.pread.max.bytes' when configured with a value \<0.
1056+
Adding a new meaning for the config 'hbase.storescanner.pread.max.bytes' when configured with a value \<0.
10571057
In HBase 2.x we allow the Scan op to specify a ReadType (PREAD / STREAM/ DEFAULT). When Scan comes with DEFAULT read type, we will start scan with preads and later switch to stream read once we see we are scanning a total data size \> value of hbase.storescanner.pread.max.bytes. (This is calculated for data per region:cf). This config defaults to 4 x of HFile block size = 256 KB by default.
10581058
This jira added a new meaning for this config when configured with a -ve value. In such case, for all scans with DEFAULT read type, we will start with STREAM read itself. (Switch at begin of the scan itself)
10591059

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -382,6 +382,12 @@ protected void requestCompactionInternal(HRegion region, HStore store, String wh
382382
// pool; we will do selection there, and move to large pool if necessary.
383383
pool = shortCompactions;
384384
}
385+
386+
// A simple implementation for under compaction marks.
387+
// Since this method is always called in the synchronized methods, we do not need to use the
388+
// boolean result to make sure that exactly the one that added here will be removed
389+
// in the next steps.
390+
underCompactionStores.add(getStoreNameForUnderCompaction(store));
385391
pool.execute(
386392
new CompactionRunner(store, region, compaction, tracker, completeTracker, pool, user));
387393
if (LOG.isDebugEnabled()) {
@@ -390,7 +396,6 @@ protected void requestCompactionInternal(HRegion region, HStore store, String wh
390396
+ "store size is {}",
391397
getStoreNameForUnderCompaction(store), priority, underCompactionStores.size());
392398
}
393-
underCompactionStores.add(getStoreNameForUnderCompaction(store));
394399
region.incrementCompactionsQueuedCount();
395400
if (LOG.isDebugEnabled()) {
396401
String type = (pool == shortCompactions) ? "Small " : "Large ";

0 commit comments

Comments
 (0)