HIVE-24553. Exclude calcite from test-jar dependency of hive-exec #1794

viirya · 2020-12-18T19:39:08Z

What changes were proposed in this pull request?

This patch proposes to exclude calcite dependencies from test-jar dependency of hive-exec.

Why are the changes needed?

The test "org.apache.hive.beeline.cli.TestHiveCli - testSetHeaderValue" we found the exception:

Caused by: java.lang.NoSuchMethodError: org.apache.calcite.rel.RelCollationImpl.<init>(Lorg/apache/hive/com/google/common/collect/ImmutableList;)V
	at org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelCollation.<init>(HiveRelCollation.java:29) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getCollationList(RelOptHiveTable.java:181) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.calcite.rel.metadata.RelMdCollation.table(RelMdCollation.java:175) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.calcite.rel.metadata.RelMdCollation.collations(RelMdCollation.java:122) ~[calcite-core-1.10.0.jar:1.10.0]
	at GeneratedMetadataHandler_Collation.collations_$(Unknown Source) ~[?:?]
	at GeneratedMetadataHandler_Collation.collations(Unknown Source) ~[?:?]
	at org.apache.calcite.rel.metadata.RelMetadataQuery.collations(RelMetadataQuery.java:482) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:189) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:374) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelFieldTrimmer.trimFields(HiveRelFieldTrimmer.java:273) ~[hive-exec-2.3.8.jar:2.3.8]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_262]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_262]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_262]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262]
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:270) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:160) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1331) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1261) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106) ~[calcite-core-1.10.0.jar:1.10.0]
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1069) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1085) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:364) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1295) ~[hive-exec-2.3.8.jar:2.3.8]
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204) ~[hive-service-2.3.8.jar:2.3.8]
	... 50 more

This looks like adding dependency of test-jar of hive-exec pulls in the unshaded calcite dependencies which conflicts with shaded calcite. We should exclude unshaded calcite when adding test-jar of hive-exec.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test.

viirya · 2020-12-18T22:41:10Z

@sunchao Looks like "testSetHeaderValue – org.apache.hive.beeline.cli.TestHiveCli" was passed now in CI.

viirya · 2020-12-19T06:59:43Z

@sunchao Please let me know if you think this is good to merge. I will create a JIRA for this.

sunchao

Thanks @viirya ! This looks good to me with one comment. BTW I'm looking at the org.apache.hadoop.hive.cli.TestAccumuloCliDriver being stuck during test and it seems started happening since #1356 (verified locally)

sunchao · 2020-12-19T18:39:59Z

beeline/pom.xml

@@ -99,6 +99,20 @@
      <version>${project.version}</version>
      <classifier>tests</classifier>
      <scope>test</scope>
+      <exclusions>


Looks good as I no longer see the test failure 👍 . BTW do you know why we need to exclude calcite here but not in other places where hive-exec is also used?

I think the test-jar won't include shaded dependencies. Then un-shaded dependencies will be pull. So I think we need to exclude them.

If there are other places using hive-exec in test scope, I think they might also need to exclude these.

Ah I see. I think there are a few other places with similar pattern, such as hive-webhcat-java-client. Should we exclude in all those places?

Yea, I think so. I will update this later.

I updated all hive-exec test-jar usage in all pom.xml files.

+1. Can you create a JIRA for this? I'm not sure if master branch also have this issue.

Created a JIRA. Updated the title and description.

viirya · 2020-12-19T19:32:34Z

Thanks @viirya ! This looks good to me with one comment. BTW I'm looking at the org.apache.hadoop.hive.cli.TestAccumuloCliDriver being stuck during test and it seems started happening since #1356 (verified locally)

Hmm, okay. Just stuck without any error/exception? Then it seems harder to figure out the reason.

sunchao · 2020-12-19T19:42:43Z

Thanks @viirya ! This looks good to me with one comment. BTW I'm looking at the org.apache.hadoop.hive.cli.TestAccumuloCliDriver being stuck during test and it seems started happening since #1356 (verified locally)

Hmm, okay. Just stuck without any error/exception? Then it seems harder to figure out the reason.

Yes just stuck without any error/exception. Thread dump also doesn't give much information which is interesting.

sunchao · 2020-12-19T20:22:11Z

OK. I tried to bump Accumulo version to 1.7.3 (the one used by master branch) and I got the exception:

select * from accumulo_table_1
[ERROR]   TestAccumuloCliDriver.testCliDriver:59 Unexpected exception java.lang.NoClassDefFoundError: org/eigenbase/util/property/BooleanProperty
        at org.apache.calcite.util.SaffronProperties.<init>(SaffronProperties.java:66)
        at org.apache.calcite.util.SaffronProperties.instance(SaffronProperties.java:134)
        at org.apache.calcite.util.Util.getDefaultCharset(Util.java:769)
        at org.apache.calcite.rel.type.RelDataTypeFactoryImpl.getDefaultCharset(RelDataTypeFactoryImpl.java:565)
        at org.apache.calcite.sql.type.SqlTypeUtil.addCharsetAndCollation(SqlTypeUtil.java:1070)
        at org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:65)
        at org.apache.calcite.rex.RexBuilder.<init>(RexBuilder.java:114)
        at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:991)
        at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
        at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1069)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1085)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:364)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)

as in #1792. So perhaps with Accumulo 1.6.0 the exception got swallowed silently and therefore test got stuck. After applying patch there together with the version bump the issue seems gone, but without version bump it will still stuck for unknown reason. I'll update #1792 with this change.

viirya · 2020-12-19T20:50:56Z

Thanks. @sunchao

viirya · 2020-12-19T20:52:23Z

OK. I tried to bump Accumulo version to 1.7.3 (the one used by master branch) and I got the exception:

select * from accumulo_table_1
[ERROR]   TestAccumuloCliDriver.testCliDriver:59 Unexpected exception java.lang.NoClassDefFoundError: org/eigenbase/util/property/BooleanProperty
        at org.apache.calcite.util.SaffronProperties.<init>(SaffronProperties.java:66)
        at org.apache.calcite.util.SaffronProperties.instance(SaffronProperties.java:134)
        at org.apache.calcite.util.Util.getDefaultCharset(Util.java:769)
        at org.apache.calcite.rel.type.RelDataTypeFactoryImpl.getDefaultCharset(RelDataTypeFactoryImpl.java:565)
        at org.apache.calcite.sql.type.SqlTypeUtil.addCharsetAndCollation(SqlTypeUtil.java:1070)
        at org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:65)
        at org.apache.calcite.rex.RexBuilder.<init>(RexBuilder.java:114)
        at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:991)
        at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
        at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1069)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1085)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:364)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)

as in #1792. So perhaps with Accumulo 1.6.0 the exception got swallowed silently and therefore test got stuck. After applying patch there together with the version bump the issue seems gone, but without version bump it will still stuck for unknown reason. I'll update #1792 with this change.

So sounds like the stuck seems related to Accumulo version?

sunchao · 2020-12-19T20:56:15Z

Yes seems so although I don't know the exact reason yet. There is also another JIRA mentioning a similar test stuck issue when upgrading Accumulo from 1.7.3 to 1.8.0

sunchao

LGTM. Will merge shortly.

viirya · 2020-12-21T18:18:26Z

Thanks @sunchao

Exclude calcite in beeline.

cfecf99

kgyrtkirk added the tests pending label Dec 18, 2020

sunchao reviewed Dec 19, 2020

View reviewed changes

kgyrtkirk added tests failed and removed tests pending labels Dec 19, 2020

Exclude calcite when adding hive-exec test classifier dependency.

b4d05d0

kgyrtkirk added tests pending and removed tests failed labels Dec 19, 2020

viirya changed the title ~~[DO-NOT-MERGE] Test exclude calcite in beeline~~ HIVE-24553. Exclude calcite from test-jar dependency of hive-exec Dec 20, 2020

kgyrtkirk added tests failed tests pending and removed tests pending tests failed labels Dec 20, 2020

sunchao approved these changes Dec 21, 2020

View reviewed changes

sunchao merged commit d7caeaf into apache:branch-2.3 Dec 21, 2020

HIVE-24553. Exclude calcite from test-jar dependency of hive-exec #1794

HIVE-24553. Exclude calcite from test-jar dependency of hive-exec #1794

Uh oh!

Conversation

viirya commented Dec 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

viirya commented Dec 18, 2020

Uh oh!

viirya commented Dec 19, 2020

Uh oh!

sunchao left a comment

Choose a reason for hiding this comment

Uh oh!

sunchao Dec 19, 2020

Choose a reason for hiding this comment

Uh oh!

viirya Dec 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunchao Dec 19, 2020

Choose a reason for hiding this comment

Uh oh!

viirya Dec 19, 2020

Choose a reason for hiding this comment

Uh oh!

viirya Dec 20, 2020

Choose a reason for hiding this comment

Uh oh!

sunchao Dec 20, 2020

Choose a reason for hiding this comment

Uh oh!

viirya Dec 20, 2020

Choose a reason for hiding this comment

Uh oh!

viirya commented Dec 19, 2020

Uh oh!

sunchao commented Dec 19, 2020

Uh oh!

sunchao commented Dec 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viirya commented Dec 19, 2020

Uh oh!

viirya commented Dec 19, 2020

Uh oh!

sunchao commented Dec 19, 2020

Uh oh!

sunchao left a comment

Choose a reason for hiding this comment

Uh oh!

viirya commented Dec 21, 2020

Uh oh!

Uh oh!

viirya commented Dec 18, 2020 •

edited

Loading

viirya Dec 19, 2020 •

edited

Loading

sunchao commented Dec 19, 2020 •

edited

Loading