HBASE-26772 Shell suspended in background

[petersomogyi]

No description provided.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 6s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 41s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 22s	the patch passed
-0 ⚠️	rubocop	0m 5s	The patch generated 3 new + 47 unchanged - 1 fixed = 50 total (was 48)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		7m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4137
Optional Tests	dupname asflicense javac rubocop
uname	Linux c06bb1738b1b 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 5dc663e
Default Java	AdoptOpenJDK-1.8.0_282-b08
rubocop	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt
Max. process+thread count	45 (vs. ulimit of 30000)
modules	C: hbase-shell U: hbase-shell
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/console
versions	git=2.17.1 maven=3.6.3 rubocop=0.80.0
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 4s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 38s	master passed
+1 💚	javadoc	0m 11s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 27s	the patch passed
+1 💚	javadoc	0m 11s	the patch passed
		_ Other Tests _
+1 💚	unit	8m 39s	hbase-shell in the patch passed.
		15m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4137
Optional Tests	javac javadoc unit
uname	Linux 45a78edd3ab1 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 5dc663e
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/testReport/
Max. process+thread count	794 (vs. ulimit of 30000)
modules	C: hbase-shell U: hbase-shell
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 6s	master passed
+1 💚	javadoc	0m 12s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 49s	the patch passed
+1 💚	javadoc	0m 11s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 44s	hbase-shell in the patch passed.
		17m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4137
Optional Tests	javac javadoc unit
uname	Linux 57f0c3660a38 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 5dc663e
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/testReport/
Max. process+thread count	849 (vs. ulimit of 30000)
modules	C: hbase-shell U: hbase-shell
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[busbey]

Change looks reasonable to me. Holding off on an approval until I can test run things locally. Feel free to proceed if someone else comes along before I get to finish.

[busbey]

okay so before this change, I can replicate the problem described in the jira:

(base) sbusbey@seans-mbp hbase-3.0.0-alpha-3-SNAPSHOT % (echo "list" | ./bin/hbase shell -n)
2022-02-26T11:48:32,236 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
reading from ~/.irbrc
hbase:001:0> list
TABLE                                                                                                                                                                                                                                                         
0 row(s)
Took 0.7729 seconds                                                                                                                                                                                                                                           
=> []
hbase:002:0> 
(base) sbusbey@seans-mbp hbase-3.0.0-alpha-3-SNAPSHOT % (echo "list" | ./bin/hbase shell -n) &
[1] 53337
(base) sbusbey@seans-mbp hbase-3.0.0-alpha-3-SNAPSHOT % 
[1]  + suspended (tty output)  ( echo "list" | ./bin/hbase shell -n; )
(base) sbusbey@seans-mbp hbase-3.0.0-alpha-3-SNAPSHOT % fg
[1]  + continued  ( echo "list" | ./bin/hbase shell -n; )
2022-02-26T11:49:07,536 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
reading from ~/.irbrc
hbase:001:0> list
TABLE                                                                                                                                                                                                                                                         
0 row(s)
Took 0.7193 seconds                                                                                                                                                                                                                                           
=> []
hbase:002:0> 
(base) sbusbey@seans-mbp hbase-3.0.0-alpha-3-SNAPSHOT %

but after the change I still have things hang, just a bit later.

(base) sbusbey@seans-mbp hbase-3.0.0-alpha-3-SNAPSHOT % (echo "list" | ./bin/hbase shell -n) &
[1] 54168
(base) sbusbey@seans-mbp hbase-3.0.0-alpha-3-SNAPSHOT % 2022-02-26T11:56:54,053 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
reading from ~/.irbrc
hbase:001:0> list

[1]  + suspended (tty output)  ( echo "list" | ./bin/hbase shell -n; )
(base) sbusbey@seans-mbp hbase-3.0.0-alpha-3-SNAPSHOT % fg
[1]  + continued  ( echo "list" | ./bin/hbase shell -n; )
TABLE                                                                                                                                                                                                                                                         
0 row(s)
Took 155.7721 seconds                                                                                                                                                                                                                                         
=> []
hbase:002:0> 
(base) sbusbey@seans-mbp hbase-3.0.0-alpha-3-SNAPSHOT %

Am I messing up the test conditions somehow?

[petersomogyi]

Thanks for taking a look @busbey! The difference seems to be with redirecting the output to a file or not. I'm also able to reproduce the problem you've noticed.

The process is suspended now on this line: https://github.com/apache/hbase/blob/master/hbase-shell/src/main/ruby/shell/formatter.rb#L37

[petersomogyi]

I'm not able to figure out clean a solution for this when the shell is running in the background but the output is stdout instead of a file. My assumption is operators running shell commands in the background would also redirect the output to a file instead of keeping everything on stdout.

One option would be to add a flag to HBase shell which does not call the refresh_width method.

[busbey]

huh. and we already have a tty guard. maybe a bug in checking that stdout is a tty, whereas the guards that are working are checks on stdin?

in any case this seems like a different, even if related, rabbit hole than the one you were chasing here.

[petersomogyi]

huh. and we already have a tty guard. maybe a bug in checking that stdout is a tty, whereas the guards that are working are checks on stdin?

I tried to include stderr to the if statement but in your test case stdout is a tty, however, since the process is in the background the Java.jline.TerminalFactory.get hangs.