Refactor prompt commands to be plugin based

[ixje]

Rationale:
Currently prompt.py over time evolved into a beast of if/else branches for command handling with several issues such as:

• Commands present but missing from the help output
• Commands present but not part of the auto completer
• Some commands have been extracted to live under /neo/Prompt/Commands/ yet still are dependent on proper integration in prompt.py
• Most commands are suboptimally covered in automated testing and/or poorly testable
• Custom commands require patching after every update

Goal:
Turn console commands into a plugin based system that's easily testable without having to run the actual prompt. They can live under /neo/Prompt/Commands/. They should be reusable with another front-end (see the related issue Create more interactive and clean CLI)

Command TODO list

• quit

• help

• show

  • block
  • header {index/hash}
  • tx {hash}
  • mem # returns memory in use and number of buffers
  • nodes # returns connected peers
  • state
  • notifications {block_number or address}
  • account {address} # returns account state
  • asset
  • • {assetId} # returns asset state
  • contract
  • • {contract hash} # returns contract state

• config

  • debug-notify {on/off}
  • sc-events {on/off}
  • maxpeers {num_peers}
  • node-requests {reqsize} {queuesize}
  • output
  • vm_log
  • compiler-nep8 {on/off}

• wallet

  • create
  • open
  • close
  • {verbose}
  • claim (max_coins_to_claim)
  • rebuild {start block}
  • create_addr {number of addresses <= 3}
  • delete_addr {addr}
  • alias {addr} {title}
  • token
  • • delete {token_contract_hash}
  • • send {token symbol} etc
  • • send_from {token symbol} etc
  • • approve {token symbol} etc
  • • allowance {token symbol} etc
  • • mint {token symbol} {mint_to_addr} etc
  • • register {addr} etc
  • • history {token symbol}
  • unspent (neo/gas)
  • split {addr} {asset} etc
  • send {assetId or name} etc
  • sendmany {number of outgoing tx} etc
  • sign {transaction in JSON format}
  • import
  • • nep2
  • • wif
  • • multisig_addr
  • • token
  • • contract_addr
  • • watch_addr
  • export
  • • wif
  • • nep2

• search

  • asset {query}
  • contract {query}

#need a good group command name for development related activities>

• to be named group
  • build {path/to/file.py} etc
  • load_run {path/to/file.avm} etc
  • testinvoke {contract hash} etc
  • debugstorage {on/off/reset}
  • import contract {path/to/file.avm} etc

TODO various

• separate presentation from implementation (move execute() implementations to separate functions such that others can call them without having to go through WalletCommand X)
• after the previous item, make sure console themes still work. This requires "Format Tokens" from the prompt_toolkit package, but it should only be applied when used in the CLI. Meaning; the business logic should not be coupled to a PromptInterface() instance because that's the one who creates the theme tokens. solution idea: supply print function as param.
• Move wallet commands related to address in its own address group ( alias, create, delete, split)
• Wallet is becoming huge. Split file into multiple (e.g. based on subgroup)
• check all command help output and validate consistent use of argument vs parameter.
• Update readthedocs documentation

[LysanderGG]

I'd like to try and do this issue.

I suggest the following:

• Open PR with the basic structure for prompt.py and one or two commands moved to /neo/Prompt/Commands/
• Review ^
• Move the other commands once the structure is ok.

What do you think?

[ixje]

@LysanderGG Sounds good! Looking forward to your PR.

To help you a bit with direction setting, here are a bit more formal requirements:

• As a developer, I want to extend neo-python with custom commands, so that I can tune it to my needs
• As a user, I want my custom commands to survive updates, so that I can continue working without further interruption or required actions
• As a user, I want to get an overview of all available commands, so that I know what I can and can't do
• As a user, I want to get usage instructions if a command is not self explanatory, so that I know how to correctly call the command and know what the expected result is
• As a user, I want commands to auto complete, so that I won't make typing mistakes and so that I can work faster
• As a developer, I want to be able to unittest my custom commands, so that I can verify correct behaviour inside neo-python
• As a maintainer, I want the implementation to be independent from the presentation, so that we can change CLI/GUI with minimal effort

Note that everything is up for discussion. I tried to keep it high level as I don't have a 100% view at this point either, we learn as we go. Feel free to comment/suggest stuff here or reach out to me on Discord (same name)

[ixje]

@LysanderGG just checking in to see if you are in need of some input/ideas. Or if you lost interest, please let us know so we don't duplicate work if somebody else shows interest. Thanks! 👍

[LysanderGG]

@ixje I started thinking about it and writing some code this week, I'll contact you on Discord for a few questions :)

[ixje]

I gave a moment of thought about how we could group commands.

quit
help

show
- block {index/hash} (tx)
- header {index/hash}
- tx {hash}
- mem # returns memory in use and number of buffers
- nodes # returns connected peers
- state
- notifications {block_number or address}
- account {address} # returns account state
- asset
-- {assetId} # returns asset state
-- search {query} <- not ideal. open for ideas how to structure otherwise.
- contract 
-- {contract hash} # returns contract state
-- search {query}

config 
- debug {on/off}
- sc-events {on/off}
- maxpeers {num_peers}
- node-requests {reqsize} {queuesize}
- node-requests {slow/normal/fast}
- compiler-nep8 {on/off}

wallet
- create
- open 
- close
- {verbose}
- claim (max_coins_to_claim)
- migrate
- rebuild {start block}
- create_addr {number of addresses <= 3}
- delete_addr {addr}
- alias {addr} {title}
- token
-- delete {token_contract_hash}
--send {token symbol} {address_from} {address to} {amount}
-- send_from {token symbol} {address_from} {address to} {amount}
-- approve {token symbol} {address_from} {address to} {amount}
-- allowance {token symbol} {address_from} {address to}
-- mint {token symbol} {mint_to_addr} (--attach-neo={amount}, --attach-gas={amount})
-- register {addr} ({addr}...) (--from-addr={addr})
-- history {token symbol}
- unspent (neo/gas)
- split {addr} {asset} {unspent index} {divide into number of vins}
- send {assetId or name} {address} {amount} (--from-addr={addr}) (--fee={priority_fee})
- sendmany {number of outgoing tx} (--change-addr={addr}) (--from-addr={addr}) (--fee={priority_fee})
- sign {transaction in JSON format}
- import
-- nep2
-- wif
-- multisig_addr
-- token
-- contract_addr
- export
-- wif
-- nep2

#need a good group command name for development related activities>
build {path/to/file.py} (test {params} {returntype} {needs_storage} {needs_dynamic_invoke} {is_payable} [{test_params} or --i]) --no-parse-addr (parse address strings to script hash bytearray)
load_run {path/to/file.avm} (test {params} {returntype} {needs_storage} {needs_dynamic_invoke} {is_payable} [{test_params} or --i]) --no-parse-addr (parse address strings to script hash bytearray)
testinvoke {contract hash} [{params} or --i] (--attach-neo={amount}, --attach-gas={amount}) (--from-addr={addr}) --no-parse-addr (parse address strings to script hash bytearray)
debugstorage {on/off/reset}
import contract {path/to/file.avm} {params} {returntype} {needs_storage} {needs_dynamic_invoke} {is_payable}

It's a first idea. I'm open to input, changes. The above would give a base help output of

config
help
show
wallet
quit
// and one yet unnamed for development commands

[ixje]

@jseagrave21 I know you're eager looking for stuff to do. The plugin base is merged into the refactor-prompt branch, so we can start migrating commands to that branch according to the above structure. If you encounter a reason why the above structure doesn't work or does not make sense then let's discuss.

[LysanderGG]

@ixje @jseagrave21
I'd suggest a markdown checklist that we could check when starting to work on a command so that we don't get to be two working on the same.

[ixje]

Good point. I created a todo list in the original issue. I don't think you guys can update it, but I can keep track of that. We'll have to post here in the comments to let each other know what command or command group we picked up to migrate to avoid conflicts/double work.

[jseagrave21]

1. Am I right in assuming from this

   neo-python/neo/Prompt/Commands/Wallet.py

   Lines 32 to 35 in 9b44392

   	self.register_sub_command('create', CommandWalletCreate())
   	self.register_sub_command(['v', '--v', 'verbose'], CommandWalletVerbose())
   	self.register_sub_command('migrate', CommandWalletMigrate())
   	self.register_sub_command('create_addr', CommandWalletCreateAddress())

   
   the corresponding commands are completed?

2. I would like to try to complete send, sendmany, sign, open, and close.

[LysanderGG]

@jseagrave21

1. The implementation is done but I haven't checked if I had some tests to move yet. I will do it.
   quit and help are also done.
   @ixje Can you check these please?

2. 👌

On another note, up to now I put all the subcommands in the same file as the parent command but maybe it's better to split each command / subcommand to its file. What do you think?

[jseagrave21]

@LysanderGG I like keeping them in the same folder. That is the example I have been following.

Also, @LysanderGG I can't get wallet to work it gives me this error:

neo> wallet
Could not execute command: list index out of range
  File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/mnt/c/users/jseag/neo/neo-python-refactor_prompt/venv/lib/python3.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/mnt/c/users/jseag/neo/neo-python-refactor_prompt/venv/lib/python3.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
  File "/mnt/c/users/jseag/neo/neo-python-refactor_prompt/venv/lib/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/mnt/c/users/jseag/neo/neo-python-refactor_prompt/venv/lib/python3.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/mnt/c/users/jseag/neo/neo-python-refactor_prompt/venv/lib/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/mnt/c/users/jseag/neo/neo-python-refactor_prompt/venv/lib/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/mnt/c/users/jseag/neo/neo-python-refactor_prompt/neo/bin/prompt.py", line 1063, in run
    traceback.print_stack()
Traceback (most recent call last):
  File "/mnt/c/users/jseag/neo/neo-python-refactor_prompt/neo/bin/prompt.py", line 989, in run
    if arguments[-1] == 'help':
IndexError: list index out of range