Skip to content

HIVE-23998: Upgrade guava to 27 for Hive branch-2 #1395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

viirya
Copy link
Member

@viirya viirya commented Aug 12, 2020

What changes were proposed in this pull request?

This PR proposes to upgrade Guava to 27 in Hive branch-2. This is basically used to trigger test for #1394.

Why are the changes needed?

When trying to upgrade Guava in Spark, found the following error. A Guava method became package-private since Guava version 20. So there is incompatibility with Guava versions > 19.0.

sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator
	at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:108)
	at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)

Does this PR introduce any user-facing change?

Yes. This upgrades Guava to 27.

How was this patch tested?

Built Hive locally.

@viirya
Copy link
Member Author

viirya commented Aug 12, 2020

cc @sunchao

@viirya
Copy link
Member Author

viirya commented Aug 12, 2020

@sunchao Yeah, seems the tests was triggered.

@kgyrtkirk
Copy link
Member

TestAccumuloCliDriver was running for 3 days - I stopped it...maybe this change also needs an accululo upgrade?

@sunchao
Copy link
Member

sunchao commented Aug 15, 2020

@kgyrtkirk I see there is a timeout of 6 hours in Jenkinsfile, curious how the test can still run for so long.

@kgyrtkirk
Copy link
Member

on branch-2 I don't see any timeout set...that's why it could run indefinetly

@sunchao
Copy link
Member

sunchao commented Aug 15, 2020

Ah I see. @viirya do you see the Accumulo test failures related?

@viirya
Copy link
Member Author

viirya commented Aug 15, 2020

Not check it yet, I will look at it later today.

@viirya
Copy link
Member Author

viirya commented Aug 15, 2020

Hm, I don't see Accumulo test failure there http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1395/1/tests.

Tried to run org.apache.hadoop.hive.cli.TestAccumuloCliDriver locally, but seems it was stuck.

I also ran it in current branch-2, but it's also stuck.

Is this correct way to run this test?

mvn clean -Dtest=org.apache.hadoop.hive.cli.TestAccumuloCliDriver -q -Pitests test

@sunchao
Copy link
Member

sunchao commented Aug 15, 2020

@viirya I think just

cd itests
mvn clean test -Dtest=org.apache.hadoop.hive.cli.TestAccumuloCliDriver

is suffice. You also need to build the itests module first.

For me, the test on branch-2 didn't stuck but failed with the following error:

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
Running org.apache.hadoop.hive.cli.TestAccumuloCliDriver
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 163.852 sec <<< FAILURE! - in org.apache.hadoop.hive.cli.TestAccumuloCliDriver
testCliDriver[accumulo_queries](org.apache.hadoop.hive.cli.TestAccumuloCliDriver)  Time elapsed: 62.901 sec  <<< FAILURE!
java.lang.AssertionError:
Unexpected exception java.lang.AssertionError: Client Execution results failed with error code = 1 while executing fname=accumulo_queries.q
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.hadoop.hive.ql.QTestUtil.failedDiff(QTestUtil.java:2201)
        at org.apache.hadoop.hive.cli.control.CoreAccumuloCliDriver.runTest(CoreAccumuloCliDriver.java:97)
        at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
        at org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver(TestAccumuloCliDriver.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.junit.runners.Suite.runChild(Suite.java:127)
        at org.junit.runners.Suite.runChild(Suite.java:26)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)

So don't understand why it stuck for so long in the CI.

@sunchao
Copy link
Member

sunchao commented Aug 15, 2020

@viirya Regardless of this, like you mentioned, the biggest blocker for this is Hadoop 2.7.2 which depends on a lower version of Guava, and I don't see a way to solve this other than upgrading the Hadoop version (Hadoop only start shading Guava from 3.0.0 I think).

@viirya
Copy link
Member Author

viirya commented Aug 16, 2020

@sunchao Technically, if we shade Guava in Hive, we should be able to work with Hadoop 2.7.2 without conflict with its older Guava.

@sunchao
Copy link
Member

sunchao commented Aug 16, 2020

we should be able to work with Hadoop 2.7.2 ...

@viirya You mean Spark, right? I think #1356 is promising but upgrading Guava to 27 is much harder.

@viirya
Copy link
Member Author

viirya commented Aug 16, 2020

I mean Hive. Not sure if it is totally correct or I may miss anything, but I think if we shade Guava in Hive, is Hadoop 2.7.2 which depends on older Guava still a problem here? By shading Guava in Hive, I think Hive can use newer Guava, and let Hadoop uses older Guava?

@sunchao
Copy link
Member

sunchao commented Aug 16, 2020

You can try. I think the hive-exec JAR produced will have no issue working with older version of Hadoop/Guava, but I don't know others like unit tests work in this case.

@viirya
Copy link
Member Author

viirya commented Aug 16, 2020

Hm, ok, let me try to shade Guava too in this PR, and see how it goes.

@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.

@github-actions github-actions bot added the stale label Oct 31, 2020
@github-actions github-actions bot closed this Nov 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants