HADOOP-18383. Codecs with @DoNotPool annotation are not closed causing memory leak

[kevins-29]

Description of PR

Explicitly call end() when returning Decompressor implementations with DoNotPool annotation to the CodecPool.

How was this patch tested?

I created the following project to demo the leak. You can run the demo with

./gradlew run --args=1024

and then monitor the memory usage using

while true; do echo \"$(date +%Y-%m-%d' '%H:%M:%S)\",$(pmap -x <PID> | grep "total kB" | awk '{print $4}'); sleep 10; done;

Results - Before Patch

"2022-08-13 10:57:29",1377812
"2022-08-13 10:57:39",1379596
"2022-08-13 10:57:49",1396284
"2022-08-13 10:57:59",1364696
"2022-08-13 10:58:09",1387352
"2022-08-13 10:58:19",1392296
"2022-08-13 10:58:29",1387392
"2022-08-13 10:58:39",1396184
"2022-08-13 10:58:49",1397320
"2022-08-13 10:58:59",1397348
"2022-08-13 10:59:09",1397392
"2022-08-13 10:59:19",1397412
"2022-08-13 10:59:29",1397436
"2022-08-13 10:59:39",1397452
"2022-08-13 10:59:49",1397480
"2022-08-13 10:59:59",1397504
"2022-08-13 11:00:09",1397536
"2022-08-13 11:00:19",1397536
"2022-08-13 11:00:29",1397568
"2022-08-13 11:00:39",1397600
"2022-08-13 11:00:49",1397648
"2022-08-13 11:00:59",1397676
"2022-08-13 11:01:09",1398232
"2022-08-13 11:01:19",1398232
"2022-08-13 11:01:29",1398816
"2022-08-13 11:01:39",1398816
"2022-08-13 11:01:49",1398816
"2022-08-13 11:01:59",1398816
"2022-08-13 11:02:09",1398816
"2022-08-13 11:02:19",1398852
"2022-08-13 11:02:29",1398852

Results - After Patch

"2022-08-13 11:03:31",1102440
"2022-08-13 11:03:41",1102496
"2022-08-13 11:03:51",1102600
"2022-08-13 11:04:01",1102600
"2022-08-13 11:04:11",1102600
"2022-08-13 11:04:21",1102532
"2022-08-13 11:04:31",1102532
"2022-08-13 11:04:41",1102532
"2022-08-13 11:04:51",1102532
"2022-08-13 11:05:01",1102532
"2022-08-13 11:05:11",1102532
"2022-08-13 11:05:21",1102532
"2022-08-13 11:05:31",1102532
"2022-08-13 11:05:41",1102532
"2022-08-13 11:05:51",1102532
"2022-08-13 11:06:01",1102532
"2022-08-13 11:06:11",1102532
"2022-08-13 11:06:21",1102532
"2022-08-13 11:06:31",1102532
"2022-08-13 11:06:41",1102532
"2022-08-13 11:06:51",1102532
"2022-08-13 11:07:01",1102532
"2022-08-13 11:07:11",1102532
"2022-08-13 11:07:21",1102532
"2022-08-13 11:07:31",1102532
"2022-08-13 11:07:41",1102532
"2022-08-13 11:07:51",1102532
"2022-08-13 11:08:01",1102532
"2022-08-13 11:08:11",1102532
"2022-08-13 11:08:21",1102532
"2022-08-13 11:08:31",1102532

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	10m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	36m 24s		branch-3.3 passed
+1 💚	compile	18m 12s		branch-3.3 passed
+1 💚	checkstyle	1m 31s		branch-3.3 passed
+1 💚	mvnsite	2m 7s		branch-3.3 passed
+1 💚	javadoc	1m 27s		branch-3.3 passed
+1 💚	spotbugs	3m 13s		branch-3.3 passed
+1 💚	shadedclient	27m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 4s		the patch passed
+1 💚	compile	17m 32s		the patch passed
+1 💚	javac	17m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 25s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	mvnsite	2m 4s		the patch passed
+1 💚	javadoc	1m 20s		the patch passed
+1 💚	spotbugs	3m 4s		the patch passed
+1 💚	shadedclient	27m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 22s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 33s		The patch does not generate ASF License warnings.
		174m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4739/1/artifact/out/Dockerfile
GITHUB PR	#4739
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 197a7d0c6929 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / f830615
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4739/1/testReport/
Max. process+thread count	1268 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4739/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	36m 15s		branch-3.3 passed
+1 💚	compile	18m 19s		branch-3.3 passed
+1 💚	checkstyle	1m 32s		branch-3.3 passed
+1 💚	mvnsite	2m 9s		branch-3.3 passed
+1 💚	javadoc	1m 26s		branch-3.3 passed
+1 💚	spotbugs	3m 2s		branch-3.3 passed
-1 ❌	shadedclient	24m 26s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 5s		the patch passed
+1 💚	compile	17m 38s		the patch passed
+1 💚	javac	17m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 25s		the patch passed
+1 💚	mvnsite	2m 6s		the patch passed
+1 💚	javadoc	1m 19s		the patch passed
+1 💚	spotbugs	3m 5s		the patch passed
+1 💚	shadedclient	27m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 21s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 34s		The patch does not generate ASF License warnings.
		161m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4739/2/artifact/out/Dockerfile
GITHUB PR	#4739
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 9405d1359731 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / e6a60d1
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4739/2/testReport/
Max. process+thread count	2160 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4739/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[sunchao]

LGTM

[sunchao]

Merged to branch-3.3. Thanks @kevins-29 .