Skip to content

HubSpot Backport: HBASE-28697 Don't clean bulk load system entries until backup is comp… #146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 3, 2025
Merged

HubSpot Backport: HBASE-28697 Don't clean bulk load system entries until backup is comp… #146

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions ...ackup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,13 +104,14 @@ protected static int getIndex(TableName tbl, List<TableName> sTableList) {

/*
* Reads bulk load records from backup table, iterates through the records and forms the paths for
* bulk loaded hfiles. Copies the bulk loaded hfiles to backup destination
* bulk loaded hfiles. Copies the bulk loaded hfiles to backup destination. This method does NOT
* clean up the entries in the bulk load system table. Those entries should not be cleaned until
* the backup is marked as complete.
* @param sTableList list of tables to be backed up
* @return map of table to List of files
* @return the rowkeys of bulk loaded files
*/
@SuppressWarnings("unchecked")
protected Map<byte[], List<Path>>[] handleBulkLoad(List<TableName> sTableList)
throws IOException {
protected List<byte[]> handleBulkLoad(List<TableName> sTableList) throws IOException {
Map<byte[], List<Path>>[] mapForSrc = new Map[sTableList.size()];
List<String> activeFiles = new ArrayList<>();
List<String> archiveFiles = new ArrayList<>();
Expand Down Expand Up @@ -192,8 +193,8 @@ protected Map<byte[], List<Path>>[] handleBulkLoad(List<TableName> sTableList)
}

copyBulkLoadedFiles(activeFiles, archiveFiles);
backupManager.deleteBulkLoadedRows(pair.getSecond());
return mapForSrc;

return pair.getSecond();
}

private void copyBulkLoadedFiles(List<String> activeFiles, List<String> archiveFiles)
Expand Down Expand Up @@ -309,10 +310,12 @@ public void execute() throws IOException {
BackupUtils.getMinValue(BackupUtils.getRSLogTimestampMins(newTableSetTimestampMap));
backupManager.writeBackupStartCode(newStartCode);

handleBulkLoad(backupInfo.getTableNames());
List<byte[]> bulkLoadedRows = handleBulkLoad(backupInfo.getTableNames());

// backup complete
completeBackup(conn, backupInfo, BackupType.INCREMENTAL, conf);

backupManager.deleteBulkLoadedRows(bulkLoadedRows);
} catch (IOException e) {
failBackup(conn, backupInfo, backupManager, e, "Unexpected Exception : ",
BackupType.INCREMENTAL, conf);
Expand Down