Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix comments for pr #1

Merged
merged 2 commits into from
Aug 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -172,8 +172,7 @@ default RewriteDataFiles zOrder(String... columns) {
RewriteDataFiles filter(Expression expression);

/**
* A user provided branch which the rewrite happens on the target branch can also be controlled
* via WAP(write-audit-publish)
* Specify the branch on which the rewrite will be performed. via WAP(write-audit-publish)
*
* @param branch the branch where the rewrite happens
* @return this for chaining
Expand Down
4 changes: 2 additions & 2 deletions docs/docs/spark-procedures.md
Original file line number Diff line number Diff line change
Expand Up @@ -421,14 +421,14 @@ Rewrite the data files in table `db.sample` and select the files that may contai
CALL catalog_name.system.rewrite_data_files(table => 'db.sample', where => 'id = 3 and name = "foo"');
```

Rewrite the data files in table `db.sample` specifying target branch rewrite happens on using WAP
Rewrite the data files in table `db.sample` specifying target branch using WAP
```sql
ALTER TABLE db.table SET TBLPROPERTIES ('write.wap.enabled'='true')
SET spark.wap.branch = audit
CALL catalog_name.system.rewrite_data_files('db.sample');
```

Rewrite the data files in table `db.sample` specifying target branch rewrite happens on using branch identifier
Rewrite the data files in table `db.sample` specifying target branch using branch identifier
```sql
CALL catalog_name.system.rewrite_data_files('db.sample.branch_audit');
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ public void testRewriteDataFilesInEmptyTable() {
}

@TestTemplate
public void testRewriteOnBranchWAP() throws Exception {
public void testRewriteOnBranchWap() throws Exception {
createPartitionTable();
// create 5 files for each partition (c2 = 'foo' and c2 = 'bar')
insertData(10);
Expand All @@ -113,7 +113,7 @@ public void testRewriteOnBranchWAP() throws Exception {
.as("branch ref should have changed")
.isNotEqualTo(branchSnapshotId);
assertThat(table.currentSnapshot().snapshotId())
.as("rewrite should happen on branch")
.as("main branch ref should not have changed")
.isEqualTo(lastSnapshotId);
}

Expand Down Expand Up @@ -143,7 +143,7 @@ public void testRewriteOnBranch() throws Exception {
.as("branch ref should have changed")
.isNotEqualTo(branchSnapshotId);
assertThat(table.currentSnapshot().snapshotId())
.as("rewrite should happen on branch")
.as("main branch ref should not have changed")
.isEqualTo(lastSnapshotId);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -141,16 +141,12 @@ private RewriteDataFiles checkAndApplyFilter(
private RewriteDataFiles checkAndApplyBranch(
Table table, Identifier ident, RewriteDataFiles action) {
String branchIdent = Spark3Util.extractBranch(ident);
if (branchIdent != null) {
return action.targetBranch(branchIdent);
}
SparkWriteConf writeConf = new SparkWriteConf(spark(), table, Maps.newHashMap());
SparkWriteConf writeConf = new SparkWriteConf(spark(), table, branchIdent, Maps.newHashMap());
String targetBranch = writeConf.branch();
if (targetBranch != null) {
return action.targetBranch(targetBranch);
} else {
return action;
action.targetBranch(targetBranch);
}
return action;
}

private RewriteDataFiles checkAndApplyStrategy(
Expand Down