Sub-commands and automatic transcript generation #257

tleonhardt · 2018-01-20T18:19:44Z

Added support for argparse sub-commands when using cmd2 decorators

Modified the do_help() method to behave differently for commands which have been decorated with an argparse ArgumentParser. This is so that the help command will properly deal with sub-command help.

Suppose you have a base command base which has two sub-commands, foo and bar. Then "help base" will provide very different help text than "help base foo".

Slightly tweaked the two argparse decorators to set an attribute in the decorated function's dictionary so that the do_help method can know which functions have an ArgumentParser and which do not.

Added a "subcommands.py" example for demonstrating how to create and use subcommands based on argparse and the cmd2 @with_argument_parser decorator.

Tab-completion of subcommand names is supported out-of-the box - it just requires the user to pass a 2nd argument to the decorators.

Also bumped version from 0.8.0a to 0.8.0

Unless someone has any bright ideas regarding how to improve support for sub-commands, this takes care of it.

Edit:

Fixed a bug where unknown commands were getting saved in the history
Added a -t flag to the history command which automatically generates a transcript file based on selected contents within the history

This closes #71
This closes #254
This closes #258

Modified the do_help() method to behave differently for methods which have been decorated with an argparse ArgumentParser. This is so that help will properly deal with sub-command help. Suppose you have a base command "base" which has two sub-commands, "foo" and "bar". Then "help base" will provide very different help text than "help base foo". Slightly tweaked the two argparse decorators to set an attribute in the decorated function's dictionary so that the do_help method can know which functions have an ArgumentParser and which do not. Added a "subcommands.py" example for demonstrating how to create and use subcommands based on argparse and the cmd2 @with_argument_parser decorator.

Updated unit tests due to changes in how help is output for commands decorated with an argparse ArgumentParser.

Also bumped version to 0.8.0.

codecov · 2018-01-20T18:24:01Z

Codecov Report

Merging #257 into master will decrease coverage by 0.74%.
The diff coverage is 87.09%.

@@            Coverage Diff             @@
##           master     #257      +/-   ##
==========================================
- Coverage   95.93%   95.19%   -0.75%     
==========================================
  Files           1        1              
  Lines        1231     1312      +81     
==========================================
+ Hits         1181     1249      +68     
- Misses         50       63      +13

Impacted Files	Coverage Δ
cmd2.py	`95.19% <87.09%> (-0.75%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b564b4...504e3db. Read the comment docs.

tleonhardt · 2018-01-20T20:52:29Z

@kotfu If you could review the general argparse usage of sub-commands and modifications to the decorators and do_help it would be appreciated.

@kmvanbrunt If you could review the tab completion of sub-commands with an eye towards edge cases I might have missed, I would appreciate that as well.

tleonhardt · 2018-01-20T21:10:59Z

I plan to add a few unit tests, but other than that I'm done with this PR now (pending feedback).

tleonhardt · 2018-01-20T21:52:04Z

I added a number of unit tests for sub-commands.

The overall code coverage will inevitably have dropped due to the fact that I had to override the cmd.complete() method which is a relatively low-level function which provides tab-completion based on readline.

In order to add unit tests of that function, we will have to mock out all of the readline calls.

Also added a section on Sub-commands to the documentation.

tleonhardt · 2018-01-20T23:06:05Z

I added a number of unit tests to to mostly cover the newly overridden complete() method. This should bring the code coverage back up.

tleonhardt · 2018-01-21T00:45:19Z

The one asymmetry I don't like about how help is being handled here is that getting help on a command decorated with an argparse.ArgumentParser sends the help text to sys.stdout instead of to self.stdout.

I suppose we could live with that minor inconsistency, but if there is a way to fix it, it would be welcome.

History changes: - Unknown commands are no longer saved in the history - history command now has a -t option to generate a transcript based on commands in the history Also: - Moved examples transcripts from examples to examples/transcripts - Added a new transcript for use with the pirate.py example

tleonhardt · 2018-01-21T01:06:13Z

history command now has a -t flag which automatically generates a transcript file for regression testing based on the chosen contents of the history. This doesn't always work perfectly, but it does in many cases and gets close in others.

Also fixed a bug where unknown commands were being saved in the history.

kotfu

I wish that there was some way we could make the @with_argument_parser decorator interrogate the argparser instance passed to it and derive the available subcommands. That way we wouldn't have to pass them as a separate argument. I looked carefully (as I'll bet you did) and it doesn't look like it's possible.

A couple of other items to consider:

in subcommands.py the function names for the two subcommands are foo() and bar(). Developers utilizing cmd2 can choose whatever function names they want. Should we change those function names to indicate a convention of something like base_foo() and base_bar()?
We already merged this, but I was looking at our new argparse decorators, and we have one called with_argument_parser() and another called with_argparser_and_unknown_args(). Maybe we should change with_argument_parser() to be called with_argparser(), so those two decorators use the same naming scheme.

tleonhardt · 2018-01-22T03:10:22Z

I did try to find a way to interrogate the argparse ArgumentParser object to find out what sub-commands it had. I was able to determine if there were any subcommands, but not what their names were.

I think your suggestion about changing the sub-command names in subcommand.py to base_foo() and base_bar() to set a good example/convention is a great idea. I'll do that.

In regards to renaming the decorator, we haven't created a release to PyPI yet, so I view that as no big deal. Better to make the improvement now. So I will do that as well.

Also: - Reanamed foo and bar subcommand methods to base_foo and base_bar

tleonhardt · 2018-01-22T03:29:39Z

Ok, I renamed that stuff you suggested.

While we are at renaming the decorators, can you think of a better slightly shorter name for @with_argparser_and_unknown_args? It just feels a little long.

Also, can you think of a good way to make it so that help for commands which use one of the argparse decorators goes to self.stdout instead of sys.stdout? It would be nice to have that consistent. If for no other reason, then doing a history -t transcript.txt won't save help command text to the transcript for commands which use these decorators because their output is coming directly from argparse.

…stdout for argparse commands In order to make "help" behave more consistently for decorated and undecorated commands, argparse output is temporarily redirected to self.stdout. So doing "help history" is similar to "help load". However, when using the "-h" with argparse commands without using the "help" command, the output from argparse isn't redirected to self.stdout. Fixing this would be rather difficult and would essentially involve creating a pyparsing rule to detect it at the parser level.

tleonhardt · 2018-01-22T05:20:39Z

TravisCI is saying that the build works on Python 2.7, 3.5, 3.6, and 3.7; but is broken on Python 3.4. But the error messages don't make any sense to me. So I'm not sure if something is messed up with TravisCI's infrastructure for Python 3.4, if one of the required test dependencies pushed an update that broke on Python 3.4, or if I actually made a change which broke cmd2 on Python 3.4.

I'll try to get a local virtualenv setup for testing.

The error looks like the following:

tox
GLOB sdist-make: /home/travis/build/python-cmd2/cmd2/setup.py
py34 create: /home/travis/build/python-cmd2/cmd2/.tox/py34
py34 installdeps: mock, pyparsing, pyperclip, pytest, pytest-xdist, six
py34 inst: /home/travis/build/python-cmd2/cmd2/.tox/dist/cmd2-0.8.0.zip
py34 installed: apipkg==1.4,attrs==17.4.0,cmd2==0.8.0,execnet==1.5.0,mock==2.0.0,pbr==3.1.1,pluggy==0.6.0,py==1.5.2,pyparsing==2.2.0,pyperclip==1.6.0,pytest==3.3.2,pytest-forked==0.2,pytest-xdist==1.22.0,six==1.11.0
py34 runtests: PYTHONHASHSEED='3132367182'
py34 runtests: commands[0] | py.test -v -n2
Traceback (most recent call last):
  File "/home/travis/build/python-cmd2/cmd2/.tox/py34/lib/python3.4/site-packages/_pytest/config.py", line 328, in _getconftestmodules
    return self._path2confmods[path]
KeyError: local('/home/travis/build/python-cmd2/cmd2/tests')

So it turns out that we need contextlib2 for Python 3.4 and earlier.

…onal dependency on contextlib2

tleonhardt · 2018-01-23T02:06:56Z

I'm going to merge this to master before it gets any bigger. If we find any problems, we can open an Issue and we will wait a little while before creating a release to PyPI so we have some time to use the new and modified features.

tleonhardt added 3 commits January 20, 2018 11:54

Fixed unit tests

4046a69

Updated unit tests due to changes in how help is output for commands decorated with an argparse ArgumentParser.

Added links to docs and examples to CHANGELOG.md

014371f

Also bumped version to 0.8.0.

tleonhardt added enhancement documentation labels Jan 20, 2018

tleonhardt added this to the 0.8.0 milestone Jan 20, 2018

tleonhardt self-assigned this Jan 20, 2018

tleonhardt requested a review from kotfu January 20, 2018 18:19

tleonhardt mentioned this pull request Jan 20, 2018

Consider adding support for nested commands or subcommands #71

Closed

tleonhardt added 3 commits January 20, 2018 13:30

Just improved a few comments in the subcommands.py example

74ad43d

Edited README.md to mention support for sub-commands

86d81c3

Tab-completion of subcommand names is now supported

29ef268

tleonhardt requested a review from kmvanbrunt January 20, 2018 20:49

Added unit tests for sub-commands

5550ab7

Added unit tests for newly-overridden complete() method

bd948d7

Also added a section on Sub-commands to the documentation.

tleonhardt changed the title ~~Added support and an example for argparse sub-commands~~ Added support for sub-commands and automatic transcript generation Jan 21, 2018

tleonhardt changed the title ~~Added support for sub-commands and automatic transcript generation~~ Support for sub-commands and automatic transcript generation Jan 21, 2018

tleonhardt changed the title ~~Support for sub-commands and automatic transcript generation~~ Sub-commands and automatic transcript generation Jan 21, 2018

tleonhardt added bug major labels Jan 21, 2018

Updated the changelog

711d638

kotfu reviewed Jan 22, 2018

View reviewed changes

Renamed @with_argument_parser decorator to @with_argparser

c9f7c01

Also: - Reanamed foo and bar subcommand methods to base_foo and base_bar

tleonhardt added 2 commits January 22, 2018 00:33

redirect_stderr wasn't added to contextlib until Python 3.5

6c2cdf2

So it turns out that we need contextlib2 for Python 3.4 and earlier.

Added documentation about Python 3.4 and earlier now having an additi…

504e3db

…onal dependency on contextlib2

tleonhardt merged commit ddfd3d9 into master Jan 23, 2018

tleonhardt deleted the sub-commands branch January 23, 2018 02:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Sub-commands and automatic transcript generation #257

Sub-commands and automatic transcript generation #257

Uh oh!

tleonhardt commented Jan 20, 2018 •

edited

Loading

Uh oh!

codecov bot commented Jan 20, 2018 •

edited

Loading

Uh oh!

tleonhardt commented Jan 20, 2018 •

edited

Loading

Uh oh!

tleonhardt commented Jan 20, 2018

Uh oh!

tleonhardt commented Jan 20, 2018

Uh oh!

tleonhardt commented Jan 20, 2018

Uh oh!

tleonhardt commented Jan 21, 2018

Uh oh!

tleonhardt commented Jan 21, 2018 •

edited

Loading

Uh oh!

kotfu left a comment

Uh oh!

tleonhardt commented Jan 22, 2018

Uh oh!

tleonhardt commented Jan 22, 2018 •

edited

Loading

Uh oh!

tleonhardt commented Jan 22, 2018 •

edited

Loading

Uh oh!

tleonhardt commented Jan 23, 2018

Uh oh!

Uh oh!

Sub-commands and automatic transcript generation #257

Sub-commands and automatic transcript generation #257

Uh oh!

Conversation

tleonhardt commented Jan 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tleonhardt commented Jan 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tleonhardt commented Jan 20, 2018

Uh oh!

tleonhardt commented Jan 20, 2018

Uh oh!

tleonhardt commented Jan 20, 2018

Uh oh!

tleonhardt commented Jan 21, 2018

Uh oh!

tleonhardt commented Jan 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kotfu left a comment

Choose a reason for hiding this comment

Uh oh!

tleonhardt commented Jan 22, 2018

Uh oh!

tleonhardt commented Jan 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tleonhardt commented Jan 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tleonhardt commented Jan 23, 2018

Uh oh!

Uh oh!

tleonhardt commented Jan 20, 2018 •

edited

Loading

codecov bot commented Jan 20, 2018 •

edited

Loading

tleonhardt commented Jan 20, 2018 •

edited

Loading

tleonhardt commented Jan 21, 2018 •

edited

Loading

tleonhardt commented Jan 22, 2018 •

edited

Loading

tleonhardt commented Jan 22, 2018 •

edited

Loading