Skip to content

HIVE-29436: Iceberg materialized view rebuild locking issue #6330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
kasakrisz wants to merge 1 commit into
base: master
Choose a base branch
from
+53 −2
Draft

HIVE-29436: Iceberg materialized view rebuild locking issue #6330

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_orc2.q
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,11 @@ select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
create materialized view mat1 stored by iceberg stored as orc tblproperties ('format-version'='1') as
select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;

create materialized view mat2 stored by iceberg stored as orc tblproperties ('format-version'='1') as
select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c <= 52;

select * from mat1;
select * from mat2;

alter materialized view mat1 disable rewrite;

Expand All @@ -39,5 +43,7 @@ explain cbo
alter materialized view mat1 rebuild;

alter materialized view mat1 rebuild;
alter materialized view mat2 rebuild;

select * from mat1;
select * from mat2;
46 changes: 46 additions & 0 deletions iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc2.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,22 @@ POSTHOOK: Output: default@mat1
POSTHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: Lineage: mat1.b SIMPLE [(tbl_ice)tbl_ice.FieldSchema(name:b, type:string, comment:null), ]
POSTHOOK: Lineage: mat1.c SIMPLE [(tbl_ice)tbl_ice.FieldSchema(name:c, type:int, comment:null), ]
PREHOOK: query: create materialized view mat2 stored by iceberg stored as orc tblproperties ('format-version'='1') as
select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c <= 52
PREHOOK: type: CREATE_MATERIALIZED_VIEW
PREHOOK: Input: default@tbl_ice
PREHOOK: Output: database:default
PREHOOK: Output: default@mat2
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: create materialized view mat2 stored by iceberg stored as orc tblproperties ('format-version'='1') as
select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c <= 52
POSTHOOK: type: CREATE_MATERIALIZED_VIEW
POSTHOOK: Input: default@tbl_ice
POSTHOOK: Output: database:default
POSTHOOK: Output: default@mat2
POSTHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: Lineage: mat2.b SIMPLE [(tbl_ice)tbl_ice.FieldSchema(name:b, type:string, comment:null), ]
POSTHOOK: Lineage: mat2.c SIMPLE [(tbl_ice)tbl_ice.FieldSchema(name:c, type:int, comment:null), ]
PREHOOK: query: select * from mat1
PREHOOK: type: QUERY
PREHOOK: Input: default@mat1
Expand All @@ -178,6 +194,17 @@ POSTHOOK: Input: default@mat1
POSTHOOK: Output: hdfs://### HDFS PATH ###
five 54
four 53
PREHOOK: query: select * from mat2
PREHOOK: type: QUERY
PREHOOK: Input: default@mat2
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: select * from mat2
POSTHOOK: type: QUERY
POSTHOOK: Input: default@mat2
POSTHOOK: Output: hdfs://### HDFS PATH ###
one 50
three 52
two 51
PREHOOK: query: alter materialized view mat1 disable rewrite
PREHOOK: type: ALTER_MATERIALIZED_VIEW_REWRITE
PREHOOK: Input: default@mat1
Expand Down Expand Up @@ -256,6 +283,14 @@ POSTHOOK: query: alter materialized view mat1 rebuild
POSTHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
POSTHOOK: Input: default@tbl_ice
POSTHOOK: Output: default@mat1
PREHOOK: query: alter materialized view mat2 rebuild
PREHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
PREHOOK: Input: default@tbl_ice
PREHOOK: Output: default@mat2
POSTHOOK: query: alter materialized view mat2 rebuild
POSTHOOK: type: ALTER_MATERIALIZED_VIEW_REBUILD
POSTHOOK: Input: default@tbl_ice
POSTHOOK: Output: default@mat2
PREHOOK: query: select * from mat1
PREHOOK: type: QUERY
PREHOOK: Input: default@mat1
Expand All @@ -267,3 +302,14 @@ POSTHOOK: Output: hdfs://### HDFS PATH ###
five 54
four 53
ten 60
PREHOOK: query: select * from mat2
PREHOOK: type: QUERY
PREHOOK: Input: default@mat2
PREHOOK: Output: hdfs://### HDFS PATH ###
POSTHOOK: query: select * from mat2
POSTHOOK: type: QUERY
POSTHOOK: Input: default@mat2
POSTHOOK: Output: hdfs://### HDFS PATH ###
one 50
three 52
two 51
3 changes: 1 addition & 2 deletions ...oop/hive/ql/ddl/view/materialized/alter/rebuild/AlterMaterializedViewRebuildAnalyzer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,8 +206,7 @@ private ASTNode getRewrittenAST(TableName tableName) throws SemanticException {
rewrittenAST = ParseUtils.parse(rewrittenInsertStatement, ctx);
this.ctx.addSubContext(ctx);

if (!this.ctx.isExplainPlan() && (AcidUtils.isTransactionalTable(table) ||
table.isNonNative() && table.getStorageHandler().areSnapshotsSupported())) {
if (!this.ctx.isExplainPlan() && (AcidUtils.isTransactionalTable(table))) {
// Acquire lock for the given materialized view. Only one rebuild per materialized view can be triggered at a
// given time, as otherwise we might produce incorrect results if incremental maintenance is triggered.
HiveTxnManager txnManager = getTxnMgr();
Expand Down