HIVE-24138. Llap external client flow is broken due to netty shading. #1491
Conversation
kgyrtkirk
added
tests pending
tests unstable
tests failed
and removed
tests pending
tests unstable
labels
ayushtkn
force-pushed
the
HIVE-24138
branch
from
ea87cb4 to
c940eab
Compare
kgyrtkirk
added
tests pending
tests failed
and removed
tests failed
tests pending
labels
ayushtkn
force-pushed
the
HIVE-24138
branch
from
c940eab to
5eb7ff3
Compare
kgyrtkirk
added
tests pending
tests failed
tests unstable
and removed
tests failed
tests pending
tests unstable
labels
ayushtkn
force-pushed
the
HIVE-24138
branch
from
74755f8 to
432443f
Compare
kgyrtkirk
added
tests pending
tests passed
and removed
tests unstable
tests pending
labels
There was a problem hiding this comment.
discussed on internal ticket, this looks good to me, +1
details:
by @abstractdog
" I'm absolutely fine with reverting netty shading, especially because you achieved a green test run upstream...I think, for upstream, all of the shading magic was because of testing failures, and of the suspected benefit that we would have fewer problems with netty later, but unfortunately this hasn't turned out to be the case.
More or less the upstream and downstream patch contains the same changes as far as I can see.
Can I have some minor questions/notes about the result of the patch? Otherwise, I'm about to give a +1 on this patch.
-
UPSTREAM: Could you just confirm how did you manage to get rid of upstream test failures? I think when you removed "netty.hadoop.version" stuff from itests/qtest/pom.xml, and removed netty relocation, you made both hadoop's 4.0.52.Final and hive's 4.1.48.Final available on classpath, which was the source of all problems I had upstream. Again, I'm fine with that as long as the tests pass (as we don't release from upstream, so I only care of tests), just asking.
If you run a mini llap local qtest, you can refer to this log message (not surprisingly, I added this line during my netty work...) -
What's the content of packaging/target/apache-hive-3.1.3000.7.2.3.0-SNAPSHOT-bin.tar.gz/lib/ after this patch?
UPDATE: I've checked that currently (without the patch, on cdpd-master), there is only netty-all-4.1.48.Final.jar, and with the patch, it's the same, which is fine.
(I'm not sure how we use the dist package in our distributions, but this way, we still won't ship double netty-all jar from hive build package, which is promising.)"
by @ayushtkn
"I ran your test on the upstream code and checked the classpath, there is all one netty-all version, which I think is same as what I wanted to achieve.
the source of having netty-all is one from hive, as it is declared as a direct dependency for hive, second is through hadoop, since it is a transitive dependency, which has a different version as that of hive, So if we let it run ideally, it should lead to two netty-versions which I guess led to failures in your case.
Now, presently while including hadoop dependencies I excluded netty-all, so the transitive dependency of netty-all isn't fetched, so instead of that version the hive dependency version is used for hadoop.
This doesn't bother Hadoop jars, as I told I confirmed it, there was no code change in hadoop for migration from 4.0 line 4.1 line and since there is only one netty-all jar, so no conflict, hence no failures.
For the 2 you already confirmed, I too confirmed it, the packaging has only one netty-all, that would be always from the hive direct dependency,"
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
|
@abstractdog can you help merge this |
…apache#1491) (Ayush Saxena reviewed by Laszlo Bodor) * HIVE-24138. Llap external client flow is broken due to netty shading. * Cleanup: Try Remove Unwanted Exclusion.
…apache#1491) (Ayush Saxena reviewed by Laszlo Bodor) * HIVE-24138. Llap external client flow is broken due to netty shading. * Cleanup: Try Remove Unwanted Exclusion. (cherry picked from commit c345cdf) # Conflicts: # pom.xml # ql/pom.xml
…apache#1491) (Ayush Saxena reviewed by Laszlo Bodor) * HIVE-24138. Llap external client flow is broken due to netty shading. * Cleanup: Try Remove Unwanted Exclusion. (cherry picked from commit c345cdf) # Conflicts: # pom.xml # ql/pom.xml
…apache#1491) (Ayush Saxena reviewed by Laszlo Bodor) * HIVE-24138. Llap external client flow is broken due to netty shading. * Cleanup: Try Remove Unwanted Exclusion.
…apache#1491) (Ayush Saxena reviewed by Laszlo Bodor)
https://issues.apache.org/jira/browse/HIVE-24138