Skip to content

HIVE-21961: Upgrade Hadoop to 3.1.4, Guava to 27.0-jre and Jetty to 9.4.20.v20190813 #1821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

wangyum
Copy link
Member

@wangyum wangyum commented Dec 31, 2020

What changes were proposed in this pull request?

Upgrade Hadoop to 3.1.4, Guava to 27.0-jre and Jetty to 9.4.20.v20190813.

Why are the changes needed?

Fix compatibility issues: #1812 (comment)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing test.

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
@wangyum
Copy link
Member Author

wangyum commented Dec 31, 2020

All the failed tests:

 testCliDriver[druid_timestamptz2] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_expressions] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_extractTime] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_floorTime] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_joins] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_masking] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_mv] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_semijoin_reduction_all_types] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_test1] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_test_alter] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_test_insert] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 testCliDriver[druidmini_test_ts] – org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
 
 testCliDriver[druidkafkamini_avro] – org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver
 testCliDriver[druidkafkamini_csv] – org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver
 
 testCliDriver[insert_into_default_keyword] – org.apache.hadoop.hive.cli.split11.TestMiniLlapLocalCliDriver
 testCliDriver[lineage3] – org.apache.hadoop.hive.cli.split12.TestMiniLlapLocalCliDriver
 testCliDriver[multi_insert_with_join2] – org.apache.hadoop.hive.cli.split15.TestMiniLlapLocalCliDriver
 testCliDriver[results_cache_invalidation2] – org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver
 testCliDriver[sort_acid] – org.apache.hadoop.hive.cli.split21.TestMiniLlapLocalCliDriver
 testCliDriver[sqlmerge] – org.apache.hadoop.hive.cli.split21.TestMiniLlapLocalCliDriver
 testCliDriver[sqlmerge_stats] – org.apache.hadoop.hive.cli.split21.TestMiniLlapLocalCliDriver
 testCliDriver[check_constraint] – org.apache.hadoop.hive.cli.split4.TestMiniLlapLocalCliDriver
 testCliDriver[dynamic_semijoin_reduction_3] – org.apache.hadoop.hive.cli.split7.TestMiniLlapLocalCliDriver
 testCliDriver[enforce_constraint_notnull] – org.apache.hadoop.hive.cli.split7.TestMiniLlapLocalCliDriver
 
 testCommitInsertTable – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitCreateEmptyTablePlusCommitDropTableWithoutPurge – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitInsertIntoWithNonExtendableSegment – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitEmptyInsertTable – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testDeleteSegment – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitInsertOverwriteTable – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testPreCreateTableWhenDataSourceExists – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testPreCreateTableWillCreateSegmentsTable – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitMultiInsertOverwriteTable – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitCreateEmptyTablePlusCommitDropTableWithPurge – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitInsertIntoWithConflictingIntervalSegment – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testInsertIntoAppendOneMorePartition – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitInsertIntoWhenDestinationSegmentFileExist – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitInsertIntoTable – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitCreateTablePlusCommitDropTableWithoutPurge – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 testCommitCreateTablePlusCommitDropTableWithPurge – org.apache.hadoop.hive.druid.TestDruidStorageHandler
 
 testTimeZone – org.apache.hadoop.hive.druid.TestHiveDruidQueryBasedInputFormat
 
 testEncryptDecrypt – org.apache.hive.spark.client.rpc.TestKryoMessageCodec
 testAutoRegistration – org.apache.hive.spark.client.rpc.TestKryoMessageCodec
 testNegativeMessageSize – org.apache.hive.spark.client.rpc.TestKryoMessageCodec
 testMaxMessageSize – org.apache.hive.spark.client.rpc.TestKryoMessageCodec
 testEmbeddedChannel – org.apache.hive.spark.client.rpc.TestKryoMessageCodec
 testKryoCodec – org.apache.hive.spark.client.rpc.TestKryoMessageCodec
 testDecryptionOnly – org.apache.hive.spark.client.rpc.TestKryoMessageCodec
 testFragmentation – org.apache.hive.spark.client.rpc.TestKryoMessageCodec
 testEncryptionOnly – org.apache.hive.spark.client.rpc.TestKryoMessageCodec
 
 testBadHello – org.apache.hive.spark.client.rpc.TestRpc
 testRpcDispatcher – org.apache.hive.spark.client.rpc.TestRpc
 testClientServer – org.apache.hive.spark.client.rpc.TestRpc
 testRpcServerMultiThread – org.apache.hive.spark.client.rpc.TestRpc
 testCloseListener – org.apache.hive.spark.client.rpc.TestRpc
 testNotDeserializableRpc – org.apache.hive.spark.client.rpc.TestRpc
 testClientTimeout – org.apache.hive.spark.client.rpc.TestRpc
 testExtendClientTimeout – org.apache.hive.spark.client.rpc.TestRpc
 testServerAddress – org.apache.hive.spark.client.rpc.TestRpc
 testServerPort – org.apache.hive.spark.client.rpc.TestRpc
 testEncryption – org.apache.hive.spark.client.rpc.TestRpc

@wangyum
Copy link
Member Author

wangyum commented Dec 31, 2020

Test Name Duration Age
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2] 3 min 52 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions] 3 min 20 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime] 3 min 48 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime] 3 min 45 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins] 4 min 14 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking] 4 min 27 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv] 3 min 43 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_semijoin_reduction_all_types] 4 min 8 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1] 3 min 56 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter] 4 min 35 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert] 4 min 5 sec 2
Testing / split-01 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_ts] 3 min 50 sec 2
Testing / split-11 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_avro] 4 min 3 sec 2
Testing / split-11 / PostProcess / org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_csv] 4 min 38 sec 2

@wangyum
Copy link
Member Author

wangyum commented Jan 1, 2021

@sunchao The remaining failed tests can be resolved by upgrading the guava version of Druid: apache/druid#10726

@belugabehr
Copy link
Contributor

Hey, this is bit more complicated perhaps. I have spent a good chunk of time on this trying to upgrade Hive to 3.2.0 and hit a lot of road blocks, check out:

HIVE-24484
(and Druid PR: 10683)

@wangyum wangyum closed this Jan 2, 2021
@wangyum
Copy link
Member Author

wangyum commented Jan 2, 2021

close it.

@wangyum wangyum deleted the HIVE-21961 branch January 2, 2021 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants