[Contribution] qubes-completion

[jamke]

The problem you're addressing (if any)

The tab completion for qvm-, qubes- and other Qubes OS specific commands is not working in Qubes OS.

The solution you'd like

I made a really advanced bash completion scrcipt for Qubes OS commands, it is available here:
https://github.com/jamke/qubes-completion
Anybody can install it from Releases section of github.

Currently I would like somebody from the Team make some kind of code review or at least provide some feedback on code quality and information how well it works for you.
Feedback from average users will also be appreciated. So, feel free to try.

The proposal is to make it a part of the system out of the box (and add bash-completion that was missed probably by mistake (#7879).

Some examples, more in README of the project:

qvm-ls --run and <tab> => qvm-ls --running
qvm-ls --t and <tab> => qvm-ls --tags
qvm-ls --tags and <tab><tab> => shows the list of existing TAGs if possible
qvm-ls -- and <tab> => shows the list of possible arguments the usual bash-way

And support of the list of names of qubes (VMs) - would be great! Imagine you type in qvm-clone , you press <tab> and get not the list of files in the current directory, but the list of VMs suitable for this place:

qvm-clone and <tab> => shows the list of existing VMs
qvm-clone pe and <tab> => qvm-clone personal
qvm-clone sys- and <tab><tab> => shows the list of existing VMs that start with sys-, because there is more than one.

Problems

The only downside of such improvement - the completion information should be kept updated with the commands, the same way as documentation and --help output now.

The value to a user, and who that user might be

It will significantly improve UX of Qubes OS users who uses terminal at least from time to time (almost everybody, I think).

[jamke]

I can try to make it if no one wants.
But it will take a long time and can require my interaction with somebody from the Team.

[jamke]

There is already an example of implementation (maybe outdated, that was made 7 years ago by @jgriffiths):
https://github.com/jgriffiths/qubes-completion
His project looks very promising and interesting.
I am not sure if this particular code should be used in the package, but some ideas for implementation can be definitely taken.

It would be great to include something like that in the dom0 out of the box because otherwise no one will install or even know that it's possible.

[andrewdavidwong]

Thank you for the proposal and for volunteering to work on it! Let's see what @marmarek thinks of this idea when he has time to comment.

[DemiMarie]

I can try to make it if no one wants. But it will take a long time and can require my interaction with somebody from the Team.

I would be willing to answer any questions you have.

[unman]

I dont think it should take too long.

https://github.com/Qubes-Community/Contents/blob/master/code/productivity/qvm-cmds-bash-completion.bash

[jamke]

@unman thanks for the link, but, as far as I understand, it's not what's needed from bash-completion.
Does it provide proper completion for the first example with qvm-ls --run from the first post of this issue-thread?

[jamke]

@unman or you meant that you would want to implement completion and it will not take you too long? Maybe I misunderstood.

[jamke]

OK, now I'm starting to make an up to date and advanced bash completion for Qubes OS tools as I mentioned above.
I am planning to make it secure and clean enough so it could be packaged to Qubes OS repos and allow everyone to get more comfort when using Qubes OS out of the box.

I would be willing to answer any questions you have.

Thank you @DemiMarie, I will contact you (probably by email) when I have questions to ask!

[jamke]

I am halfway of making a script for amazing qubes completion.

The completion is already working for 29 Qubes OS tools (it's more than a half of total amount of selected ones) and provides completion for everything:

• qube names (VMNAME), taking into account running state and VM type,
• all known arguments,
• all known flags,
• options, properties and their possible values,
• prefs,
• features,
• device IDs,
• and more, respecting argument order and -- argument.

A lot of work, including code cleaning, support of volume and pools completions, still has to be done.
When everthing is ready I will provide the script for testing.

I also found a bunch of inconsistencies and minor errors in mans of tools and --help outputs.
Do you want me to spend time and make the list of problems?

The completion also supports different ways of providing options:

qvm-device block attach --option=read-only=yes
qvm-device block attach --option=read-only="yes"
qvm-device block attach --option="read-only=yes"
qvm-device block attach --option="read-only"="yes"

qvm-device block attach --option read-only=yes
qvm-device block attach --option read-only="yes"
qvm-device block attach --option "read-only=yes"
qvm-device block attach --option "read-only"="yes"

I checked that all of those ways work in Qubes OS properly and the completion script provides options accordingly. It was hard.

The list of Qubes OS tools that I decided to make advanced completion for:

qvm-ls
qvm-tags
qvm-start
qvm-shutdown
qvm-kill
qvm-run

qvm-pause
qvm-unpause

qvm-create
qvm-clone
qvm-remove

qvm-device
qvm-block
qvm-usb
qvm-pci

qvm-prefs
qvm-features

qvm-volume

qvm-template
qvm-template-postprocess
qvm-backup
qvm-backup-restore

qvm-check
qvm-firewall
qvm-service
qvm-console-dispvm
qvm-get-image
qvm-get-tinted-image

qvm-pool
qvm-pool-legacy
qvm-sync-appmenus
qvm-appmenus

qvm-copy-to-vm
qvm-move-to-vm
qvm-copy
qvm-move

qvm-start-gui
qvm-start-daemon

qvm-xkill
qvm-sync-clock

qubes-dom0-update
qubes-prefs
qubes-hcl-report
qubesctl
qubesd-query

If you think I'm missing something or want to provide any help - let me know.

[DemiMarie]

I am halfway of making a script for amazing qubes completion.

The completion is already working for 29 Qubes OS tools (it's more than a half of total amount of selected ones) and provides completion for everything:

Nice!!!

• qube names (VMNAME), taking into account running state and VM type,
• all known arguments,
• all known flags,
• options, properties and their possible values,
• prefs,
• features,
• device IDs,
• and more, respecting argument order and -- argument.

A lot of work, including code cleaning, support of volume and pools completions, still has to be done. When everthing is ready I will provide the script for testing.

That would be great!

I also found a bunch of inconsistencies and minor errors in mans of tools and --help outputs. Do you want me to spend time and make the list of problems?

Yes please!

The completion also supports different ways of providing options:

qvm-device block attach --option=read-only=yes
qvm-device block attach --option=read-only="yes"
qvm-device block attach --option="read-only=yes"
qvm-device block attach --option="read-only"="yes"

qvm-device block attach --option read-only=yes
qvm-device block attach --option read-only="yes"
qvm-device block attach --option "read-only=yes"
qvm-device block attach --option "read-only"="yes"

I checked that all of those ways work in Qubes OS properly and the completion script provides options accordingly. It was hard.

The list of Qubes OS tools that I decided to make advanced completion for:

qvm-ls
qvm-tags
qvm-start
qvm-shutdown
qvm-kill
qvm-run

qvm-pause
qvm-unpause

qvm-create
qvm-clone
qvm-remove

qvm-device
qvm-block
qvm-usb
qvm-pci

qvm-prefs
qvm-features

qvm-volume

qvm-template
qvm-template-postprocess
qvm-backup
qvm-backup-restore

qvm-check
qvm-firewall
qvm-service
qvm-console-dispvm
qvm-get-image
qvm-get-tinted-image

qvm-pool
qvm-pool-legacy
qvm-sync-appmenus
qvm-appmenus

qvm-copy-to-vm
qvm-move-to-vm
qvm-copy
qvm-move

qvm-start-gui
qvm-start-daemon

qvm-xkill
qvm-sync-clock

qubes-dom0-update
qubes-prefs
qubes-hcl-report
qubesctl
qubesd-query

If you think I'm missing something or want to provide any help - let me know.

I’d be willing to help review the code. A few questions:

• What is the latency like? Slow completion can be worse than no completion, as the shell is unresponsive while the completion code runs.
• For qubes-dom0-update, is your code aware of DNF options?

[jamke]

• What is the latency like? Slow completion can be worse than no completion, as the shell is unresponsive while the completion code runs.

I'm developing it in a qube, not in dom0, so I made simple stubs of qvm-ls, qvm-firewall, qvm-block. And it works instantly, so bash completion script by itself does not slow things down at all. But yes, actual qvm-* commands are quite slow (why, btw?), so the latency can be an issue.

For qubes-dom0-update, is your code aware of DNF options?

Not yet. I'm hoping to somehow pass completion to the dnf completion itself (like sudo does) as it's quite complicated already. Or I will have to cover some commands myself, like --enablerepo.

[marmarek]

performance hint: list vms with qvm-ls --raw-list or qvm-ls --raw-data --fields ... (listing only necessary fields) - it's significantly faster; otherwise it fetches a lot of info for the whole table.

[jamke]

performance hint: list vms with qvm-ls --raw-list or qvm-ls --raw-data --fields ... (listing only necessary fields) - it's significantly faster; otherwise it fetches a lot of info for the whole table.

@marmarek yes, thank you, good point. I am already using --raw-data, but dropping unused fields (I use the first three columns) is a good idea, especially if it speeds up.

By the way, another possible bug qvm-ls --help-columns does not provide NAME column, nor is it shown i nqvm-ls --help:

available columns (see --help-columns):
  CLASS, DISK, FLAGS, GATEWAY, MEMORY, PRIV-CURR, PRIV-MAX, PRIV-USED,
  ROOT-CURR, ROOT-MAX, ROOT-USED, STATE

Other issues I see:

1. Not all commands have --raw attributes, and parsing output without such flags is not the best idea in the first place. Like output of ls-pci is not possible to parse in genera except the first column, because it has a second column of device names that can be anything, including spaces, double spaces and etc.
2. Latency can be a factor. qvm-ls runs fast (in --raw-* mode, of course), no problem with that. But running qvm-volume list takes 2.7s on my modern desktop PC (whilst lvs runs in 0.2s, ten times faster and provides a similar list). Unfortunately, qvm-volume has no --raw option, nor I see any way to speed it up as VMNAME is not defined at the point where the completion of volumes is required. That's currently the only place I gave up to complete because of latency is too high for being comfortable.
3. ...

There are many inconsistencies in tools. For example:

• qvm-ls supports --raw-data, whilst qvm-firewall supports --raw, some support nothing,
• qvm-pool always outputs header line whilst qvm-device outputs header depending on output target: terminal and pipe act differently.
• different tools support different sets of default args (--help, --verbose, --quite).

So, I see a room for improvement for all these tools and docs.
I have several proposals how these qvm- and qubes- tools should be addressed to be more consistent and clean. Not sure if the Team would eager to implement it.
Can I modify mans myself and provide PRs? Opening issues for each problem or typo is not a good option, probably.

[DemiMarie]

I have several proposals how these qvm- and qubes- tools should be addressed to be more consistent and clean. Not sure if the Team would eager to implement it.

Please make them!

Can I modify mans myself and provide PRs? Opening issues for each problem or typo is not a good option, probably.

Sure!

[jamke]

Please make them!

On @qubes-dev mailing list or on github as a new enhancement issue, where more people sees and discuss it? Or both.

[andrewdavidwong]

On @qubes-dev mailing list or on github as a new enhancement issue, where more people sees and discuss it? Or both.

Sounds like it would be better to start on qubes-devel, then create issue(s) only once concrete, actionable, and well-scoped tasks have been defined.

It's not a matter of where more people see things. The issue tracker and each mailing list has its own rules and purpose. In particular, the issue tracker is not intended to serve as a discussion venue.

[jamke]

• For qubes-dom0-update, is your code aware of DNF options?

Update on this - I created a wonderful hack that allows to autocomplete all the dnf flags, standalone options and even list packages the same way dnf does.
I reassemble completion command like it's dnf install ... and as completion of _dnf(). Similar to what sudo does. Actually even better: dnf install is replaced with the argument of --action= of qubes-dom0-update, so dnf provides flags and other options accordingly. I hope, it will work well.
Plus I avoided the duplication of code from dnf completion.

[jamke]

Please make them!

I've posted two proposals in qubes-devel, you probably saw.

P.S. Frankly speaking, I consider qubes-devel to be nearly dead and inactive, so I expect no actual discussions there.
It's understandable as mailing lists from the technical point of view are obsolete, too many disadvantages: no way to make corrections, no proper text formatting, plus no way to login to the web version without google account and etc. And no advantages, as other platforms already do mail notifications with full text and formatting and even allow posting from email if one needs it for some reason (like a habit).
I hope development discussions will someday move to github, gitlab or an least forum, to revive. :)

[andrewdavidwong]

@jamke, you are also welcome to post your proposals on the forum, if you like. Unlike qubes-users, qubes-devel is not automatically mirrored on the forum, so you would not be duplicating content that already exists on the forum. (Normally we discourage posting the same thing to multiple mailing lists, but posting to both the qubes-devel and the forum seems fine, as long as the content is appropriate to both venues.)

[jamke]

Sorry for delay.
I started using current version in my own dom0 on daily bases, and hope that when I am free in a month, I will publish the first version.

[jamke]

Still keeping development, currently the project is on stage that I can almost share.

I have a question about dnf inside dom0. I know that dom0 is updated via updater qube (e.g. sys-firewall), but is it still possible to have bash completion for dnf?
Like:

dnf install <TAB><TAB>

It does not work inside dom0 correctly - it provides the list of files in the current directory instead of list of packages (like in any other qubes, even offline ones). How to make it work properly? It would be much more convenient.

[DemiMarie]

@jamke I recommend not completing anything for dnf in the initial version.

[jamke]

I recommend not completing anything for dnf in the initial version.

Maybe I did not get you right. I am not talking about dnf itself, but qubes-dom0-update, of course.
I have implemented a trick with passing completion task from qubes-dom0-update to dnf . It's like what sudo does but more complicated.
And it works like magic, e.g. it provides all flags of dnf that can be passed via qubes-dom0-update by design.

But when I copy the script from a development qube to dom0 the completion of packages is not working. I thinks it is because dnf has no database in dom0. So, I do not know how to fix it, because completion of default --action=install would be great.

By the way, completion of packages for commands like: qubes-dom0-update --action=remove does work as expected even in dom0, at least this is nice.

[jamke]

I’d be willing to help review the code.

@DemiMarie, I am ready to share current source code, maybe some recommendations will initiate further changes.
Are you still in and if yes, then how would you prefer to receive it?

I do not want to make it public at current point for a couple of reasons: 1) it may have problems that I am not aware of and people will struggle, 2) it is not a final version, and if people liked it they may keep this version with possible issues for a while. 3) would like a second opinion before publishing.
In the future I would love to see it as a part of default installation because it really makes live with Qubes OS much more comfortable, especially for advanced users.

@unman @marmarek maybe you are also interested?

[unman]

Of course, interested. Can you not branch off existing repo, or if it entirely de novo, just create your own?

[jamke]

Can you not branch off existing repo, or if it entirely de novo, just

It is completely new, not based on any project's source code, only some ideas.

About publishing current version at github - OK, I will work a bit on README.md and return shortly with the link.

[jamke]

Pushed to github and shared the repo with you. I think it should work.
Looking forward for your feedback, guys.

[jamke]

Maybe some preliminary feedback is available?
Should I post it on forum for testing of Qubes OS users (there will be no way back)?

[jamke]

Feedback will be appreciated. Even preliminary.

[jamke]

Improved completion of qvm-firewall and fixed several minor issues. Removing rules is now very pleasant due to table output of --rule-no= selector.

[jamke]

Preliminary documentation about what the project is able to do is here:
https://forum.qubes-os.org/t/project-qubes-completion-awesome-bash-completion-for-qubes-os-commands/19462

I still would want to make the project public only after preliminary review by somebody from the Team or at least somebody with development skills, but I had no feedback for more than a month, maybe I should publish it as it is.

Nonetheless, anyone who is interested in trying completion and wants to provide feedback - please contact me.

[andrewdavidwong]

@jamke: Sorry to hear you haven't received the feedback you wanted. I suppose everyone with the requisite skills has been too busy to volunteer. Since it looks like you're trying to contribute a package, I've updated this issue to conform to our package contribution guidelines.

(It looks like you got some volunteers in your forum thread, but they said it'd be easier for them to work with a public git repo. My guess is the same might be true of the volunteers you're seeking here, so that's something to consider.)

[jamke]

@andrewdavidwong I am going to open the repository and hope that it has no major flaws for people who decided to download and use it without reviews.
Preliminary review by somebody with expertise from the Team before making it public is still desired, but it does not look like happening, so be it that way.

[unman]

I dont believe that you told me what repo was when soliciting review.

[jamke]

OK, now anybody can try it!
I made an improved version public, anybody can install it from Releases section of github:

https://github.com/jamke/qubes-completion

Currently I would like somebody from the Team make some kind of code review or at least provide some feedback on code quality and information how well it works for you.
Feedback from average users will also be appreciated. So, feel free to try.

[jamke]

I dont believe that you told me what repo was when soliciting review.

Maybe I am not getting you right, but I sent invites on github to the project on May 9. I hoped that github provided information about name of the repo (qubes-completion) and the author/owner of repository, so it all would be obvious. Or you mean something else?

[jamke]

I changed the first post to reflect the fact that qubes-completion project already exists and ready for use.

[unman]

On Tue, Jun 27, 2023 at 08:52:47PM -0700, Jamke wrote: > I dont believe that you told me what repo was when soliciting review. Maybe I am not getting you right, but I sent invites on github to the project on May 9. I hoped that github provided information about name of the repo (`qubes-completion`) and the author/owner of repository, so it all will be obvious. Or you mean something else?

Dont have any record of that in email.

[jamke]

Dont have any record of that in email.

I do not know. github shows me that your invite was simply expired after a while. It is strange that you had no email notification for this event. Does not matter now, feel free to try it now.

[jamke]

New release 1.0.2 of qubes-completion was published!

Give it a try: https://github.com/jamke/qubes-completion/releases/tag/v1.0.2

Change log:

• Add support for qvm-backup and qvm-backup-restore
• Update all basic tests
• Remove third-party code
• Avoid qube completion with dom0 for some unsuitable cases
• Add submodules of BATS
• Improve README
• Minor typo fixes and improvements

Installation:

1. install bash-completion package in dom0:
   sudo qubes-dom0-update bash-completion
2. put bash script-file attached to the release qubes-completion.sh to /etc/bash_completion.d/ in dom0.

More details can be found in the README file.

Note:

This the release was made with a help of Qubes OS community!

Thanks to all contributors, here is a list of them on github:
https://github.com/jamke/qubes-completion/graphs/contributors

[jamke]

I am reminding about this contribution.

It works flawlessly, the Qubes OS experience really gets much better for advanced users with this clever completion.
And I would really love other people to have it out of the box in the foreseeable future (R4.2 updates, maybe, or at least R4.3).
Can I somehow further help with it?

[DemiMarie]

@jamke I think the main problem you are running into is that R4.2 is about to be released, so the developers (especially @marmarek) are busy with the upcoming release. Once R4.2 is out, the developers will have way more time.