Core: bump Hadoop version to 3.2.0 #5024
Conversation
This is the first release to support java 11, among other improvements. Includes a fix for AliyunOSSMockLocalStore to compile by changing to a different implementation of encodeHex on the classpath and adding commons-io as a test-time dependency.
|
for anyone wondering "why 3.2.0 and not 3.2.3?": you need to upgrade hive from 3.2.1+ just to get guava versions lined up. |
|
TEZ-3853 Binary incompatibility caused by DEFAULT_LOG_LEVEL it looks like hive/tez will need some version update, because the version of tez used in the tests isn't compatible with hadoop branch 3. This is going to make for a more traumatic change |
| @@ -37,8 +37,8 @@ | |||
| import java.util.List; | |||
| import java.util.Locale; | |||
| import java.util.Map; | |||
| import org.apache.commons.codec.binary.Hex; | |||
There was a problem hiding this comment.
We try not to rely on Apache Commons for anything to avoid a direct dependency, even in tests (though it looks like that was already violated). Is there an easy way to replace this? We should be able to replace FileUtils with a readFully method in IOUtil
There was a problem hiding this comment.
commons code it was on the classpath already and i just needed any hex.encode call. There's a couple more...let me find them and see which is preferable
Iceberg shades Guava to avoid conflicts. Do we need to worry about this? |
|
hive and tez turn out to have problems in the public branches; ASF hive is currently only building against 3.1 until apache/hive#3279 is in. I'm going to avoid touching this PR for a bit while I see about getting parquet building against 3.2 first |
|
is there possible to use hadoop shaded client? |
|
@pan3793, Iceberg should declare its dependencies correctly. If you want to use a shaded version downstream, you can exclude and change the dependency tree. |
|
@steveloughran Parquet 1.13 has been released recently and compiles against Hadoop 3.2.3 |
|
@Fokko i really am not active on this...if you want to take on the "bump things up" project, that'd be wonderful for me. |
|
giving up on the eternal movign target here. focusing on reflection based access to the cloud-friendly APIs |
Hadoop 3.2.0 is the first release to support java 11, among
other improvements.
Includes a fix for AliyunOSSMockLocalStore to compile by changing
to a different implementation of encodeHex on the classpath
and adding commons-io as a test-time dependency.