Skip to content

HBASE-26543 correct parsing of shell args with GetoptLong - #4000

Closed
busbey wants to merge 2 commits into
apache:masterfrom
busbey:HBASE-26543
Closed

HBASE-26543 correct parsing of shell args with GetoptLong#4000
busbey wants to merge 2 commits into
apache:masterfrom
busbey:HBASE-26543

Conversation

@busbey

Copy link
Copy Markdown
Contributor

Since Ruby's getoptlong library can't handle config definitions of the form -Dkey=value, we preprocess all args to pull out config definitions.

After that preprocessing, the remaining changes correct our use of the getoptlong library so we have the same cli handling as prior to using the getoptlong library.

Local testing, see HBASE-26543 for previous behavior:

(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % cat ~/.irbrc puts "reading from ~/.irbrc"
(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % cat ../tmp/rely_on_argv.rb if ARGV.length > 0
puts "Pulled out cli option for use in script: #{ARGV.shift}"
end
(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % ./bin/hbase shell ../tmp/rely_on_argv.rb foobar
2022-01-04T17:35:26,106 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pulled out cli option for use in script: foobar
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/book.html#shell
Version 3.0.0-alpha-3-SNAPSHOT, r898b1695e92e89d22d60da69f3c662555ec1eb0a, Mon Jan 3 19:56:35 CST 2022
Took 0.0010 seconds reading from ~/.irbrc
hbase:001:0> exit
(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % ./bin/hbase shell ../tmp/rely_on_argv.rb foobar -f
2022-01-04T17:36:06,137 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pulled out cli option for use in script: foobar
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/book.html#shell
Version 3.0.0-alpha-3-SNAPSHOT, r898b1695e92e89d22d60da69f3c662555ec1eb0a, Mon Jan 3 19:56:35 CST 2022
Took 0.0011 seconds hbase:001:0> exit
(base) sbusbey@Seans-MacBook-Pro hbase-3.0.0-alpha-3-SNAPSHOT % ./bin/hbase shell -D cats=dee -Dfoo=bar ../tmp/rely_on_argv.rb foobar -f 2022-01-04T17:36:41,621 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pulled out cli option for use in script: foobar
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/book.html#shell
Version 3.0.0-alpha-3-SNAPSHOT, r898b1695e92e89d22d60da69f3c662555ec1eb0a, Mon Jan 3 19:56:35 CST 2022
Took 0.0009 seconds hbase:001:0> p @hbase.configuration.get 'foo'
"bar"
=> "bar"
hbase:002:0> p @hbase.configuration.get 'cats'
"dee"
=> "dee"
hbase:003:0> exit

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 31sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 51smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 47sthe patch passed
+1 💚rubocop0m 6sThe patch generated 0 new + 23 unchanged - 24 fixed = 23 total (was 47)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
12m 52s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4000
Optional Testsdupname asflicense javac rubocop
unameLinux 81a7c86526b9 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / baeb51f
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count65 (vs. ulimit of 30000)
modulesC: hbase-shell U: hbase-shell
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/console
versionsgit=2.17.1 maven=3.6.3 rubocop=0.80.0
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

['-D', GetoptLong::REQUIRED_ARGUMENT],
['--return-values', '-r', GetoptLong::NO_ARGUMENT]
)
opts.ordering = GetoptLong::REQUIRE_ORDER

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this going to break back compat somewhere?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, this ordering option should bring us closer to before HBASE-24772 was introduced. I reasonably believe it is compatible with our parsing prior to that change.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for those interested in more details here are the docs for Ruby GetoptLong's ordering:

https://ruby-doc.org/stdlib-3.0.3/libdoc/getoptlong/rdoc/GetoptLong.html#method-i-ordering-3D

specifically, this patch expressly has us use REQUIRE_ORDER:

REQUIRE_ORDER :

Options are required to occur before non-options.

Processing of options ends as soon as a word is encountered that has not been preceded by an appropriate option flag.

I believe this matches the behavior of our manual cli argument parsing prior to HBASE-24772.

Note that by default GetoptLong picks an ordering based on an environment variable we don't set. The default in that case is PERMUTE :

PERMUTE :

Options can occur anywhere in the command line parsed. This is the default behavior.

Every sequence of words which can be interpreted as an option (with or without argument) is treated as an option; non-option words are skipped.

This ordering option is why after HBASE-24772, and prior to this patch, passing in IRB's "ignore .irbrc" flag ( -f) would cause our shell to crash with an "I don't know what that option is supposed to mean".

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 48smaster passed
+1 💚javadoc0m 16smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 50sthe patch passed
+1 💚javadoc0m 14sthe patch passed
_ Other Tests _
+1 💚unit6m 57shbase-shell in the patch passed.
18m 29s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4000
Optional Testsjavac javadoc unit
unameLinux dbb9fb5da527 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / baeb51f
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/testReport/
Max. process+thread count2502 (vs. ulimit of 30000)
modulesC: hbase-shell U: hbase-shell
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 59smaster passed
+1 💚javadoc0m 14smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 45sthe patch passed
+1 💚javadoc0m 16sthe patch passed
_ Other Tests _
+1 💚unit7m 29shbase-shell in the patch passed.
19m 23s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4000
Optional Testsjavac javadoc unit
unameLinux ccb25d91205b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / baeb51f
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/testReport/
Max. process+thread count2434 (vs. ulimit of 30000)
modulesC: hbase-shell U: hbase-shell
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4000/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@madrob

Copy link
Copy Markdown
Contributor

LGTM

busbey added a commit that referenced this pull request Jan 8, 2022
@busbeybusbey closed this Jan 8, 2022
@busbey
busbey deleted the HBASE-26543 branch January 8, 2022 04:02
busbey added a commit that referenced this pull request Jan 8, 2022
Signed-off-by: Mike Drob <mdrob@apache.org>
(cherry picked from commit dda337f)
busbey added a commit that referenced this pull request Jan 8, 2022
Signed-off-by: Mike Drob <mdrob@apache.org>
(cherry picked from commit dda337f)
szucsvillo pushed a commit to szucsvillo/hbase that referenced this pull request Aug 22, 2024
Signed-off-by: Mike Drob <mdrob@apache.org>
(cherry picked from commit dda337f)
(cherry picked from commit 9048507)
szucsvillo pushed a commit to szucsvillo/hbase that referenced this pull request Sep 9, 2024
…etoptLong (apache#4000)
Signed-off-by: Mike Drob <mdrob@apache.org>
(cherry picked from commit dda337f)
(cherry picked from commit 9048507)
Change-Id: I48f9c38241aa5e546d77f87e2ce3645cba9ce507
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@busbey@Apache-HBase@madrob