QMK CLI and JSON keymap support

[skullydazed]

Description

This PR adds fleshed out python support to QMK. I tried to keep the changes fairly minimal to start, but it does comprise several features:

• Fleshed out python support- it should now be easier for people to add python tools to qmk
• QMK CLI (Add qmk_firmware/bin to your path to use
• JSON keymap support- you can now use keymap.json instead of keymap.c.
• qmk compile-json - a command that will compile configurator exports
• qmk hello - an example QMK CLI command

Types of Changes

• Core
• New feature
• Keymap/layout/userspace (addition or update)

Issues Fixed or Closed by This PR

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have tested the changes and verified that they work and don't break anything (as well as I can manage).

[yanfali]

I looked through it briefly and it looks reasonable. Haven't tested it yet. Can you also please update .editorconfig for your python settings? It's not currently part of the set.

[yanfali]

I just tried to use the command and it has the following error

can you add some examples to the usage message?

➜  json git:(compile_json) ✗ qmk-compile-json keymap.json                   <<<
☒ JSON file does not exist!
usage: qmk-compile-json [-h] [-V] [-v] [--datetime-fmt GENERAL_DATETIME_FMT]
                        [--log-fmt GENERAL_LOG_FMT]
                        [--log-file-fmt GENERAL_LOG_FILE_FMT]
                        [--log-file GENERAL_LOG_FILE] [--color] [--no-color]
                        [-c GENERAL_CONFIG_FILE] [--save-config]
                        filename
☒ [Errno 2] No such file or directory: 'keymap.json'
Traceback (most recent call last):
  File "/Users/yanfa.li/projects/qmk_firmware/lib/python/milc.py", line 537, in __call__
    self.__call__()
  File "/Users/yanfa.li/projects/qmk_firmware/lib/python/milc.py", line 543, in __call__
    return self._entrypoint(self)
  File "/Users/yanfa.li/projects/qmk_firmware/lib/python/qmk/cli/compile/json.py", line 25, in main
    with open(cli.args.filename, 'r') as fd:
FileNotFoundError: [Errno 2] No such file or directory: 'keymap.json'

[yanfali]

OK, it looks like it only works from the top level. I don't really like that assumption. Anyway you can make it work from any dir? it's not make, it's python, you should be able to find the basedir easily.

[yanfali]

The usage messages are too terse. A couple of lines about what they do or pointing to documentation would be helpful. I can't figure out what qmk-json-keymap does for example. I tried the same command as compile and it just errored out. There's no obvious clue as to what it actually does.

[yanfali]

qmk-json-keymap keyboards/clueboard/66_hotswap/keymaps/json/keymap.json
☒ 'Namespace' object has no attribute 'output'
Traceback (most recent call last):
  File "/Users/yanfa.li/projects/qmk_firmware/lib/python/milc.py", line 537, in __call__
    self.__call__()
  File "/Users/yanfa.li/projects/qmk_firmware/lib/python/milc.py", line 543, in __call__
    return self._entrypoint(self)
  File "/Users/yanfa.li/projects/qmk_firmware/lib/python/qmk/cli/json/keymap.py", line 25, in main
    if cli.args.output == ('-'):
AttributeError: 'Namespace' object has no attribute 'output'
➜  qmk_firmware git:(compile_json) ✗ qmk-json-keymap                        <<<
usage: qmk-json-keymap [-h] [-V] [-v] [--datetime-fmt GENERAL_DATETIME_FMT]
                       [--log-fmt GENERAL_LOG_FMT]
                       [--log-file-fmt GENERAL_LOG_FILE_FMT]
                       [--log-file GENERAL_LOG_FILE] [--color] [--no-color]
                       [-c GENERAL_CONFIG_FILE] [--save-config]
                       [-o GENERAL_OUTPUT]
                       filename
qmk-json-keymap: error: the following arguments are required: filename

[skullydazed]

I fixed up some of the file handling and made the CLI a little more robust in the face of errors. Bugs identified so far should be addressed.

[gwerbin]

What's the intended installation process for this?

If you're already asking users to pip install -r requirements.txt, then you might as well go the extra mile and make the whole thing an installable package. That has the added advantage of setting this code up to become a general-purpose QMK utility library.

I'm experienced with Python application development and packaging, so if you're interested in what I describe below I am happy to contribute the time to making it happen.

Right now you are "installing" scripts into bin/ and manually dispatching thereto from bin/qmk. That certainly works for this particular, case, but I have two specific recommendations:

1. Use the built-in argparse library, which provides the ability to parse -options and --options as well as delegate to subcommands
2. Make the whole thing "installable", and use "entry points" so that the command line program qmk will be accessible from anywhere, as long as Python where it was installed is on the user's PATH

Recommendation 1 means no more manual parsing of sys.argv -- it's optional but nice to have. It also means automatically-generated --help text, automatic/default error messages, et al. Looks like you're already using mlic, my mistake!

Recommendation 2 means you can delete the bin/qmk* files entirely, and replace them with a __main__.py file that implements the whole CLI in pure Python functions, and you never have to worry about the shebang line being incorrect.

If you're interested I can put together a PoC based off this PR.

[yanfali]

Suggested change

	These are good. We should have some one day.
	These are good. We should have some one day.
	# Editor Support
	## vim
	If you use w0rp/ale you can easily add linting and formatting to your `.vimrc` by installing flake8 and yapf, and adding the following lines:
	```vimrc
	let g:ale_linters = {
	\ 'python': ['flake8'],
	\}
	let g:ale_fixers = {
	\ 'python': ['yapf']
	\}

[yanfali]

LGTM

[orngshdw]

Hi sorry for the late comment.
It may be worthwhile to consider not installing (or ask before installing) python packages in requirements.txt globally as it may conflict with people who version control packages installed on their machine (since requirements.txt doesn't specify versions, the latest of each module will be installed).

One suggestion is to use a python virtual environment, such as venv, to install packages in and run our python scripts with. If we do that, we can avoid the issue I mentioned above & have the added benefit of knowing that the python scripts are being run with the modules & versions we specify in the requirements.txt file. It can be something as simple as (for mac at least):

# install venv...may need sudo
python3 -m pip install --user virtualenv

# creating the python3 virtualenv in qmk project root
python3 -m venv qmk_pythonenv

# activate the python environment created
. qmk_pythonenv/bin/activate

# all pip/python commands will run in that virtualenv
which python
>>> ${qmk_firmware_root}/qmk_pythonenv/bin/python
which pip
>>> ${qmk_firmware_root}/qmk_pythonenv/bin/python
####
# run qmk commands or install modules here such as 
# pip install -r requirements.txt
####

# deactivate virtual environment at end of script/work
deactivate

# all python/pip commands after the deactivate command now use global python/pip environment

Note we'll have to add a directory to the .gitignore file to ignore the virtual environment directory we create.

It's something to consider, if not now, then maybe in the future

[yanfali]

@orngshdw good points, but this just makes everything even harder to use. The opposite of the intention of the command :(

[awkannan]

@yanfali @orngshdw as someone who works in python daily, PLEASE use a virtualenv! The fact that the install is hidden inside a shell script makes it even more insidious.

My recommendation is to use pipenv to make it dirt simple and easy.

[skullydazed]

I was hoping to avoid virtualenv at first. :)

You make good points on how we can do what we need without impacting users. The question is whether we should let that work hold up getting this in.

I'd like to note that we install a lot of software during the install process, some of which may conflict with a user's needs. The risk of adding the two packages we currently do is minimal, and if someone needs to pin those to another version it likely won't impact us for now. That's why I've left only abstract package definitions for the moment. We can add a prompt and/or a note to the documentation so that people are aware, but I don't think that particular problem is python specific.

The right way to use virtualenv here is to create the environment and use virtualenv entrypoints so that bin/qmk works as expected without having to activate. As @yanfali says we don't want to increase complexity for the user, and luckily we don't have to. We can create the virtualenv and the proper symlinks during the setup process.

My experience with virtualenv is that getting to that point will take some elbow grease. That will get done faster if someone volunteers to take that project on.

[yanfali]

@awkannan not disagreeing, this is more of a case of python packaging sucking

[yanfali]

So in reality this only really affects python developers. Most end users probably don't care about polluting their global python cache. Does it make sense to use something like pyenv/pipenv to enforce the python environment if we detect it on the runtime path?

[gwerbin]

I strongly, strongly recommend the virtualenv approach. It's just too easy to break a Python installation. I'd estimate that 10 to 20% of all newbie questions I field on chat rooms are related to broken environments. And heaven forbid someone should decide to become a python developer after they have QMK installed. Then what?

python -m venv ./venv
. ./venv/bin/activate
python -m pip install requirements.txt

Is that really so bad for first time set up?

And similarly is

. ./venv/bin/activate

Before running QMK commands that bad?

Take it from the experienced python users here, the alternatives are painful and they are not worth the pain.

In the future, you can put this on PyPI and people can use other, more advanced methods for set up that require a little less magic incantation typing. But I think this is suitable for minimum viability.

[yanfali]

@gwerbin have you seen most of our users, they have a hard enough time installing drivers. Asking them to jump through venv; which incidentally requires more software to install, is even more pain and raises barriers for entry. In principle I'm not disagreeing, it's just python doesn't make this easy at all.

[skullydazed]

20 year python veteran here- yes it's the better way, and long term the way we're going to do it, but please don't oversell the problem and downplay the work involved in the solution.

Anything extra the user has to do is that bad. We're working towards a 1 command QMK install, and forcing the user to interact with virtualenv in that way prevents that. Luckily that's not the only way to use virtualenv.

Things are not going to be perfect on day 1, but we will keep improving them.

[gwerbin]

venv ships with Python. And compared to drivers it's relatively easy to debug, not reliant on unmaintained and closed source software, generally less mysterious to all parties involved, and in an absolute worst case scenario you can just uninstall and start again.

Compared to the overhead of installing Python itself, spinning up an environment I don't think is a big deal.

[skullydazed]

Does that mean you're volunteering to do the venv integration work here?

[orngshdw]

Think my earlier comment may have suggested something a bit unrealistic and/or had high effort to implement; hopefully you all won't view it as the only viable solution. It was just an idea/suggestion

This PR does looks like a good first step and isn't worth the holdup as a final solution is being looked into as @skullydazed has said. I don't feel the two packages (argcomplete & colorama) are likely to be dependancies that will break other tools and are lightweight enough to uninstall via pip uninstall -r requirements.txt (for those of us in the know). Others who don't use python only want qmk working so that their keyboard can be flashed afterall...

@gwerbin Hopefully anyone that becomes a python dev after installing qmk will know to only do python work in a virtualenv :) In new virtualenvs pip freeze doesn't return anything regardless of what is installed globally. lmk if I'm wrong

Note: there are other less impactful & simpler alternatives that will not need any user interaction (such as adding packages to the repo ourselves and then doing a sys.path.insert/append during runtime). It'll take some looking into & test for sure, which I think we can look into perhaps in another PR.

[gwerbin]

@orngshdw I actually didn't know you could uninstall like that. I will shut up now :)

Yes this seems like a great start.