Skip to content

HBASE-26543 correct parsing of shell args with GetoptLong #4000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

busbey
Copy link
Contributor

@busbey busbey commented Jan 4, 2022

Since Ruby's getoptlong library can't handle config definitions of the form -Dkey=value, we preprocess all args to pull out config definitions.

After that preprocessing, the remaining changes correct our use of the getoptlong library so we have the same cli handling as prior to using the getoptlong library.

Local testing, see HBASE-26543 for previous behavior:

(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % cat ~/.irbrc 
puts "reading from ~/.irbrc"
(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % cat ../tmp/rely_on_argv.rb 
if ARGV.length > 0
  puts "Pulled out cli option for use in script: #{ARGV.shift}"
end
(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % ./bin/hbase shell ../tmp/rely_on_argv.rb foobar
2022-01-04T17:35:26,106 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pulled out cli option for use in script: foobar

HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/book.html#shell
Version 3.0.0-alpha-3-SNAPSHOT, r898b1695e92e89d22d60da69f3c662555ec1eb0a, Mon Jan  3 19:56:35 CST 2022
Took 0.0010 seconds                                                                                                                                                                                                                             
reading from ~/.irbrc
hbase:001:0> exit
(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % ./bin/hbase shell ../tmp/rely_on_argv.rb foobar -f
2022-01-04T17:36:06,137 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pulled out cli option for use in script: foobar

HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/book.html#shell
Version 3.0.0-alpha-3-SNAPSHOT, r898b1695e92e89d22d60da69f3c662555ec1eb0a, Mon Jan  3 19:56:35 CST 2022
Took 0.0011 seconds                                                                                                                                                                                                                             
hbase:001:0> exit
(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % ./bin/hbase shell -D cats=dee -Dfoo=bar  ../tmp/rely_on_argv.rb foobar -f 
2022-01-04T17:36:41,621 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pulled out cli option for use in script: foobar

HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/book.html#shell
Version 3.0.0-alpha-3-SNAPSHOT, r898b1695e92e89d22d60da69f3c662555ec1eb0a, Mon Jan  3 19:56:35 CST 2022
Took 0.0009 seconds                                                                                                                                                                                                                             
hbase:001:0> p @hbase.configuration.get 'foo'
"bar"
=> "bar"
hbase:002:0> p @hbase.configuration.get 'cats'
"dee"
=> "dee"
hbase:003:0> exit

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 4m 51s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 47s the patch passed
+1 💚 rubocop 0m 6s The patch generated 0 new + 23 unchanged - 24 fixed = 23 total (was 47)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
_ Other Tests _
+1 💚 asflicense 0m 11s The patch does not generate ASF License warnings.
12m 52s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4000
Optional Tests dupname asflicense javac rubocop
uname Linux 81a7c86526b9 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / baeb51f
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 65 (vs. ulimit of 30000)
modules C: hbase-shell U: hbase-shell
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/console
versions git=2.17.1 maven=3.6.3 rubocop=0.80.0
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

)
opts.ordering = GetoptLong::REQUIRE_ORDER
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this going to break back compat somewhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, this ordering option should bring us closer to before HBASE-24772 was introduced. I reasonably believe it is compatible with our parsing prior to that change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for those interested in more details here are the docs for Ruby GetoptLong's ordering:

https://ruby-doc.org/stdlib-3.0.3/libdoc/getoptlong/rdoc/GetoptLong.html#method-i-ordering-3D

specifically, this patch expressly has us use REQUIRE_ORDER:

REQUIRE_ORDER :

Options are required to occur before non-options.

Processing of options ends as soon as a word is encountered that has not been preceded by an appropriate option flag.

I believe this matches the behavior of our manual cli argument parsing prior to HBASE-24772.

Note that by default GetoptLong picks an ordering based on an environment variable we don't set. The default in that case is PERMUTE :

PERMUTE :

Options can occur anywhere in the command line parsed. This is the default behavior.

Every sequence of words which can be interpreted as an option (with or without argument) is treated as an option; non-option words are skipped.

This ordering option is why after HBASE-24772, and prior to this patch, passing in IRB's "ignore .irbrc" flag ( -f) would cause our shell to crash with an "I don't know what that option is supposed to mean".

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 27s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 48s master passed
+1 💚 javadoc 0m 16s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 50s the patch passed
+1 💚 javadoc 0m 14s the patch passed
_ Other Tests _
+1 💚 unit 6m 57s hbase-shell in the patch passed.
18m 29s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4000
Optional Tests javac javadoc unit
uname Linux dbb9fb5da527 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / baeb51f
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/testReport/
Max. process+thread count 2502 (vs. ulimit of 30000)
modules C: hbase-shell U: hbase-shell
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 34s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 59s master passed
+1 💚 javadoc 0m 14s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 45s the patch passed
+1 💚 javadoc 0m 16s the patch passed
_ Other Tests _
+1 💚 unit 7m 29s hbase-shell in the patch passed.
19m 23s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4000
Optional Tests javac javadoc unit
uname Linux ccb25d91205b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / baeb51f
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/testReport/
Max. process+thread count 2434 (vs. ulimit of 30000)
modules C: hbase-shell U: hbase-shell
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@madrob
Copy link
Contributor

madrob commented Jan 7, 2022

LGTM

busbey added a commit that referenced this pull request Jan 8, 2022
Signed-off-by: Mike Drob <mdrob@apache.org>
@busbey busbey closed this Jan 8, 2022
@busbey busbey deleted the HBASE-26543 branch January 8, 2022 04:02
busbey added a commit that referenced this pull request Jan 8, 2022
Signed-off-by: Mike Drob <mdrob@apache.org>
(cherry picked from commit dda337f)
busbey added a commit that referenced this pull request Jan 8, 2022
Signed-off-by: Mike Drob <mdrob@apache.org>
(cherry picked from commit dda337f)
szucsvillo pushed a commit to szucsvillo/hbase that referenced this pull request Aug 22, 2024
Signed-off-by: Mike Drob <mdrob@apache.org>
(cherry picked from commit dda337f)
(cherry picked from commit 9048507)
szucsvillo pushed a commit to szucsvillo/hbase that referenced this pull request Sep 9, 2024
…etoptLong (apache#4000)

Signed-off-by: Mike Drob <mdrob@apache.org>
(cherry picked from commit dda337f)
(cherry picked from commit 9048507)
Change-Id: I48f9c38241aa5e546d77f87e2ce3645cba9ce507
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants