HDFS-15949. Fix integer overflow #2857

GauthamBanasandra · 2021-04-03T06:17:33Z

There are some instance where
the overflow is deliberately
done in order to get the lower
bound. This results in an
overflow warning.
We fix this by using the min
value directly.

GauthamBanasandra · 2021-04-03T06:18:14Z

This PR fixes some warnings reported as part of #2792.

[ 63%] Building CXX object main/native/libhdfspp/tests/CMakeFiles/hdfs_ext_hdfspp_test_shim_static.dir/hdfs_ext_test.cc.o
In file included from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/googletest-src/googletest/include/gtest/gtest.h:375,
                 from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/googletest-src/googlemock/include/gmock/internal/gmock-internal-utils.h:47,
                 from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/googletest-src/googlemock/include/gmock/gmock-actions.h:51,
                 from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/googletest-src/googlemock/include/gmock/gmock.h:59,
                 from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h:24,
                 from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc:19:
/mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc: In member function ‘virtual void hdfs::HdfsExtTest_TestHosts_Test::TestBody()’:
/mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc:456:95: warning: integer overflow in expression of type ‘long int’ results in ‘-9223372036854775808’ [-Woverflow]
  456 |   EXPECT_EQ(nullptr, hdfsGetHosts(fs, filename.c_str(), 0, std::numeric_limits<int64_t>::max()+1));
      |                                                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
/mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc:460:92: warning: integer overflow in expression of type ‘long int’ results in ‘-9223372036854775808’ [-Woverflow]
  460 |   EXPECT_EQ(nullptr, hdfsGetHosts(fs, filename.c_str(), std::numeric_limits<int64_t>::max()+1, std::numeric_limits<int64_t>::max()));
      |                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~

GauthamBanasandra · 2021-04-03T06:19:50Z

I was able to verify this change using the following standalone C++ program -

#include <iostream>
#include <limits>
#include <ostream>

int main(int argc, char *argv[]) {
  std::cout << "Max int64_t = " << std::numeric_limits<int64_t>::max()
            << std::endl;
  std::cout << "Max int64_t + 1 = " << std::numeric_limits<int64_t>::max() + 1
            << std::endl;
  std::cout << "Min int64_t = " << std::numeric_limits<int64_t>::min()
            << std::endl;
  return 0;
}

Output -

Max int64_t = 9223372036854775807
Max int64_t + 1 = -9223372036854775808
Min int64_t = -9223372036854775808

GauthamBanasandra · 2021-04-03T06:20:47Z

We can observe that Max int64_t + 1 and Min int64_t are equal.

hadoop-yetus · 2021-04-03T08:18:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	13m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 51s		trunk passed
+1 💚	compile	2m 41s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	3m 14s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	shadedclient	52m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	2m 32s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	cc	2m 32s		hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 8 unchanged - 3 fixed = 8 total (was 11)
+1 💚	golang	2m 32s		the patch passed
+1 💚	javac	2m 32s		the patch passed
+1 💚	compile	2m 38s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	cc	2m 38s		hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~~20.04-b08 generated 0 new + 8 unchanged - 3 fixed = 8 total (was 11)
+1 💚	golang	2m 38s		the patch passed
+1 💚	javac	2m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	shadedclient	13m 8s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	31m 55s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		119m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2857/1/artifact/out/Dockerfile
GITHUB PR	#2857
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux b47fa8d047cc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / e93e361f757fe8b8ddb9b3c42c225f8ce5783dc3
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2857/1/testReport/
Max. process+thread count	609 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2857/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* There are some instance where the overflow is deliberately done in order to get the lower bound. This results in an overflow warning. * We fix this by using the min value directly.

hadoop-yetus · 2021-04-05T18:51:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 45s		trunk passed
+1 💚	compile	2m 38s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	2m 41s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	shadedclient	51m 46s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	2m 30s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	cc	2m 30s		hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 8 unchanged - 3 fixed = 8 total (was 11)
+1 💚	golang	2m 30s		the patch passed
+1 💚	javac	2m 30s		the patch passed
+1 💚	compile	2m 31s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	cc	2m 31s		hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~~20.04-b08 generated 0 new + 8 unchanged - 3 fixed = 8 total (was 11)
+1 💚	golang	2m 31s		the patch passed
+1 💚	javac	2m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	shadedclient	13m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 1s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		106m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2857/2/artifact/out/Dockerfile
GITHUB PR	#2857
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 287e7cb0e4d6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d3edf67`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2857/2/testReport/
Max. process+thread count	621 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2857/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org> (cherry picked from commit 36014b8)

Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

goiri approved these changes Apr 5, 2021

View reviewed changes

HDFS-15949. Fix integer overflow

d3edf67

* There are some instance where the overflow is deliberately done in order to get the lower bound. This results in an overflow warning. * We fix this by using the min value directly.

GauthamBanasandra force-pushed the fix-overflow branch from e93e361 to d3edf67 Compare April 5, 2021 17:04

aajisaka approved these changes Apr 6, 2021

View reviewed changes

aajisaka merged commit 36014b8 into apache:trunk Apr 6, 2021

aajisaka pushed a commit that referenced this pull request Apr 6, 2021

HDFS-15949. Fix integer overflow (#2857)

d7b31ac

Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org> (cherry picked from commit 36014b8)

aajisaka pushed a commit that referenced this pull request Apr 6, 2021

HDFS-15949. Fix integer overflow (#2857)

48a2288

Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org> (cherry picked from commit 36014b8)

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15949. Fix integer overflow (apache#2857)

1ba4fa5

Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15949. Fix integer overflow #2857

HDFS-15949. Fix integer overflow #2857

Uh oh!

GauthamBanasandra commented Apr 3, 2021

Uh oh!

GauthamBanasandra commented Apr 3, 2021

Uh oh!

GauthamBanasandra commented Apr 3, 2021

Uh oh!

GauthamBanasandra commented Apr 3, 2021

Uh oh!

hadoop-yetus commented Apr 3, 2021

Uh oh!

hadoop-yetus commented Apr 5, 2021

Uh oh!

Uh oh!

HDFS-15949. Fix integer overflow #2857

HDFS-15949. Fix integer overflow #2857

Uh oh!

Conversation

GauthamBanasandra commented Apr 3, 2021

Uh oh!

GauthamBanasandra commented Apr 3, 2021

Uh oh!

GauthamBanasandra commented Apr 3, 2021

Uh oh!

GauthamBanasandra commented Apr 3, 2021

Uh oh!

hadoop-yetus commented Apr 3, 2021

Uh oh!

hadoop-yetus commented Apr 5, 2021

Uh oh!

Uh oh!