Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BIGTOP-4084. Upgrade Tez to 0.10.4. #1246

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

iwasakims
Copy link
Member

https://issues.apache.org/jira/browse/BIGTOP-4084

This PR bumps the verison of Tez to 0.10.3. We can remove most of the patch needed for Tez 0.10.2.

  • TEZ-4492, TEZ-4493, TEZ-4520 are contained in Tez 0.10.3.
  • We do not need TEZ-4319 since snappy-java-1.1.8.2 (supporting aarch64) is pulled as transitive dependency of hadoop now.

@iwasakims
Copy link
Member Author

I got protobuf related error on smoke-tests of Tez. I'm looking into the cause.

java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto,

@iwasakims
Copy link
Member Author

While hadoop uses shaded protobuf now, Hadoop 3.3.6 retains dependency on protobuf-java-2.5.0 for compatibility and the protobuf-java-2.5.0.jar is in the classpath. Since TEZ-4363 upgraded the protobuf to 3.21., Tez must use its own protobuf-java-3.21.1.jar.

@iwasakims iwasakims changed the title BIGTOP-4084. Upgrade Tez to 0.10.3. BIGTOP-4084. Upgrade Tez to 0.10.4. Oct 17, 2024
@iwasakims
Copy link
Member Author

Since init-hcfs.groovy is contained in hadoop package, smoke-tests need hadoop-pkg with the patch applied. You can manually put the tez.tar.gz on HDFS otherwise.

$ ./docker-hadoop.sh --create 1 ... --stack hdfs,yarn,mapreduce,tez
$ ./docker-hadoop.sh --exec 1 sudo -u hdfs hdfs dfs -put /usr/lib/tez/lib/tez.tar.gz /apps/tez/lib/
$ ./docker-hadoop.sh --smoke-tests tez

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant