Completion problems

[rgoldberg]

There are numerous completion problems: some only for certain shells, some for specific CompletionKinds for certain shells, and some that are only for arguments but not for option values.

bash

• Offers candidates for all positional arguments regardless of current argument position
• Repeatedly offers singular options instead of just once
• Offers flags & options after --
• directory not implemented for arguments
• file not implemented for arguments
• directory & file(…) for option values moves to next parameter after selecting the first item, even if you've submitted a directory, instead of allowing you to request candidates from within the selected directories. If you move back to the parameter & append a /, tabbing will then offer properly filtered candidates.

There might not be a way in bash 3.2.57, or even in later bashes, to improve the following:

• outputs to stdout, then opens a new prompt rather than interactive output below command line

fish

• Offers candidates for all positional arguments regardless of current argument position
• Doesn't offer flags or options unless you type a leading -
• Repeatedly offers singular options instead of just once
• Repeats parameter help text for each candidate
• file() offers no candidates when no extensions were specified
• file(extensions:) doesn't offer directories

zsh

• shellCommand does not offer any candidates

There aren't necessarily bugs, they are just potentially suboptimal setups / aren't properly documented in the docs or code:

• If the user hasn't already typed a leading -, doesn't offer flags or options if there are unsupplied positional arguments
• file(extension:) if existing part of the completing parameter is a path to a leaf directory, and if no non-hidden files in that directory match any of the given extensions, then all non-hidden files in that directory are offered as candidates, regardless of their respective extensions. Is that intended? If so, it should be documented.

all

The following @Option doesn't offer true & false. Using a @Flag would probably be better, but if a developer wants to use a Bool @Option, the completion scripts should probably offer true & false without additional setup being required.

@Option(help: "Case Sensitive")
var caseSensitive: Bool

Documentation

The CompletionKind / CompletionKind.Kind documentation is also sometimes wrong or confusing, so I will update that along with the associated code.

PR

I will work on a PR to fix these problems. If I find additional problems, I'll add them to this issue.

Related Issues

• CompletionKind should support multiple options #566: My PR will fix the problems mentioned, but won't implement the proposed solution.
• Add completion for default subcommand to supercommand #488: My PR won't add support for this.
• Custom fish completions do not work #376: It looks like this has already been fixed, but if it somehow resurfaces, I'll fix it.

Versions

ArgumentParser version:
main branch

Swift version:

swift-driver version: 1.62.15 Apple Swift version 5.7.2 (swiftlang-5.7.2.135.5 clang-1400.0.29.51)
Target: x86_64-apple-macosx12.0

Checklist

• If possible, I've reproduced the issue using the main branch of this package
• I've searched for existing GitHub issues

Steps to Reproduce

Generate completion scripts using the aforementioned CompletionKinds.

Try to use them.

Expected behavior

Completion works.

Actual behavior

Some completion either does nothing, or outputs an error.

See per-shell sections above for details.

[rgoldberg]

I am partway through a PR for this, with many improvements for all 3 shells.

What is the oldest version of each shell with which the respective script must maintain compatibility? Here are some guesses:

• bash 3.2.57 (included in macOS 10.13)
• zsh 5.3 (included in macOS 10.13)
• fish 3.7.1 (latest fish for almost a year; fish is not included with any macOS, so we can expect users to know about upgrading it)

I know that SAP can be used on OSes other than macOS (sorry to be Mac-centric, but I don't know shell versions distributed with / popular for other OSes).

Supporting bash 3.2.57 might be annoying because it doesn't support associative arrays. Being able to require some bash 4.x or (better yet) 5.x might simplify things.

[natecook1000]

If we can make things work with the default-installed Bash version, I think we should. I know it's a bit of a pain, but it would feel like a mismatch (and probably generate more issues) to require Bash v4 or v5. The other two shell versions both seem reasonable!

[rgoldberg]

@natecook1000 thanks. Will do.

2 questions:

Overloaded Swift custom completion functions

I think there should be 2 extra arguments to Swift custom completion functions:

1. 0-based index of the word for which completion candidates are being requested. This is necessary because 2 (all 3 after my fixes, as long as it's fish 4+) of the shells include any command line words that are after the word for which completions are being requested; without this index, there's no way of knowing the word for which completions are being requested.

2. 0-based index of the location of the cursor within the word for which completion candidates are being requested. This is useful because a user could request completions from the before any character in the word, or at the end of the word. Maybe a custom Swift completion function would like to know the exact cursor location.

I would leave the existing single-argument function as is but deprecate it, and add & recommend the new three-argument overload.

Completion inconsistencies between shells

There are inconsistencies amongst how the completions behave in the 3 shells.

Is it OK if I make all 3 behave as similarly as possible? I'll try to document all inconsistencies that I couldn't overcome within a reasonable amount of effort.

I will standardize around what I perceive to be the best completion behavior. In most cases, this will be closest to the current zsh behavior. I'll attempt to note all behavior changes.

Most inconsistencies seem to be contained within the shell completion behavior itself, so fixing them should be backwards compatible with existing SAP uses.

The 3 shells, however, pass the command-line words to Swift custom completion functions in varied ways. Making that info consistent, and fixing bugs for which people might already workarounds, could break existing Swift custom completion functions. See below for more info.

[rgoldberg]

@natecook1000 Thanks for merging my 2 other PRs.

Here some of the problems I'm fixing. There are many more issues, but the following are ones about which I have open questions.

Should I include anyone else in this discussion?

Backwards compatibility

The way that Strings are passed from Swift to shell completion scripts is broken in numerous ways in SAP 1.5.0.

Frequently, they aren't properly escaped (so the whole completion script can break; Swift either must not include certain values as completions, or must escape things properly in some places, but not escape them in others, differently for the different shells, etc.), but sometimes they are, so devs never know what they must do.

Some Strings are split into different words on spaces, so values cannot contain spaces, but some other places (or for different shells), spaces are acceptable.

Etc.

Devs might have worked around all these bugs, but many workarounds won't work properly after my PR fixes all these problems.

It will be easiest to not try to be compatible with the old, incredibly broken way of handling Strings. My PR will be binary compatible (with one deprecation that is easy to fix; users can just add 2 arguments to existing custom completion functions, and they can ignore the arguments and let their old code run without deprecation warnings), but String`s will be consistently & correctly handled (barring oversights in my PR). It's probably enough of a change that SAP should be bumped up a major version, just to indicate how existing workarounds could break.

Custom completion command line contents

Simple question

Should the shells pass the full command line to Swift custom completion functions? That would include other commands (separated by ;) before and/or after the commend for which completions are being requested, pipes into and/or of that command, etc. That would simplify things for SAP implementation & provide the user with the most data, but it might be harder for them to parse stuff (& might give tools that SAP shell completion more data than they should get). Or should the shells only send the section of the command line for the command for which completions are being requested? (that's the current behavior).

If Swift shouldn't receive the full command line, what should mark things to excise? I assume ;, |, &, &&, ||, …?

Details

I have found multiple inconsistencies & problems with how the contents of the current command line are supplied to Swift custom completion functions by the 3 supported shells.

Undiscovered inconsistencies & problems might exist.

None of the shells supplies all words from the command line that we should want shells to pass to Swift custom completion functions.

We will unfortunately need to supply verbatim (instead of unquoted) words to Swift; otherwise, e.g., $x & '$x', etc. would incorrectly look the same.

My upcoming PR will support 2 types of custom functions:

1. the existing single-argument SAP 1.5.0 function, which will be deprecated
2. a new three-argument function (with 2 index arguments), which will replace the deprecated function

We could try to keep the broken existing behavior in the deprecated function, but there are so many other problems with completions that will require breaking fixes that we probably want to bump major version, and fix the problems in the deprecated function, too.

In examples:

• ^ is the location of the cursor when TAB is pressed; it is not a literal ^.
• … is a placeholder for the relevant word(s); it is not a literal ….
• Command lines can extend before or after the relevant example, except they cannot extend before a leading com.
• "a ➡ b" indicates that a from the command line becomes b as a Swift argument

Table detailing how the command line is communicated from shells to Swift custom completion functions:

feature	example	bash	fish	zsh
accepts args before current subcommand	com … sub ^	🎉1	✅	✅
includes words before current subcommand	… sub ^	✅	✅	🎉
includes non-empty current word	current^	✅	🎉	✅
includes empty current word not followed by other words	^	✅	🎉	🎉
includes empty current word followed by other words	^ later	🎉	🎉	🎉
includes words after current word	^ …	✅	4+	✅
preserves quoting (i.e. supplies verbatim words)	'$x' ➡ '$x'	✅	4+	✅
does not expand params, etc.	$x ➡ $x	✅	✅	✅
includes redirects (separate symbols & source/target)	<…, >…, >>…, …	✅	✅2	-
includes process substitution (as one word)	<(…)	✅34	✅	✅
includes command substitution as one word	$(…)	✅	✅	✅
omits pipes in	…|	✅	✅	✅
omits pipes out	|…	✅	✅	✅
omits prior commands	…;	✅	✅	✅
omits later commands	;…	✅	✅	✅

• ✅: Already works in SAP 1.5.0.
• 🎉: Fixed in my upcoming PR.
• X+: Fixed in my upcoming PR for shell version X+, but not for earlier versions of the shell.
• -: Does not work in SAP 1.5.0. Haven't yet investigated a solution. Might not be able to fix.
• ?: Must investigate if it already works or is even supported by the shell. Might not be able to fix possible problems.

Footnotes

1. Fixed in my upcoming PR. In SAP 1.5.0, if arguments exist before the current subcommand, Swift custom completion is never called (I haven't yet investigated exactly what arguments cause this, if not all do). ↩

2. redirect symbols (<, >, >>, …) omitted ↩

3. <( & each space-delimited token in …) separate ↩

4. For bash 3.2.57, if a process substitution is on the command line, Swift custom completion is never called; for bash 5.2.37, however, completion is called. Not sure about other versions. Will try 4.0 later, then possibly others even later. ↩

[rgoldberg]

Testing platforms for custom completion command line contents

The above results were obtained on macOS for the following shell versions:

• bash 3.2.57
• bash 5.2.37
• fish 3.7.1
• fish 4.0b1
• zsh 5.8.1
• zsh 5.9

All versions of a shell behaved the same, unless explicitly noted.

Will try to test zsh 5.3, bash 4.0, and any shell versions released from now until when the PR is merged.

Should I test any other shell versions?

I do not have any OSes other than macOS handy. If testing must be performed on other OSes (or on multiple versions of macOS), can you find someone else to handle them?

[rgoldberg]

Questions about custom completion command line contents

1. Do we want input and/or output redirects to be included? What if there is no way to get all the shells to either include or exclude them in identifiable ways?
2. I'd want to make process substitutions, redirects, etc. look the same regardless of the shell. So I'd either concatenate or split them in a standard manner. In the examples below, each separate code fragment is its own element in the [String] argument to a Swift custom completion function:
   1. Constructs with start & end delimiters, e.g.:
      1. Prefer <(proc-sub a b)
      2. Possibly < & (proc-sub a b)
      3. Possibly <( & proc-sub a b & )
      4. Definitely not <( & proc-sub a b)
      5. Definitely not <( & proc-sub & a & b)
      6. Definitely not <( & proc-sub & a & b & )
   2. Constructs with only a start marker, e.g.:
      1. Prefer > & out
3. Do we want parameters, command substitutions, and/or anything else that can be evaluated on the command line passed verbatim as in the command line, or evaluated (i.e. expanded)?
   1. I'd say no to everything besides parameters (because of side effects and/or runtime), but possibly OK to parameters.

[rgoldberg]

Swift completion output

After my upcoming PR, all completion candidates supplied from Swift must be output as the verbatim text for the equivalent of a zsh single-quoted string (i.e. no escape values are supported), except that single quotes may be used within the output values.

For shells (like bash & zsh) that support verbatim single-quoted strings (no escapes, single quotes not allowed as direct content), SAP will properly replace each single quote in the content by splitting the output into multiple single-quoted strings concatenated with escaped single quotes between them.

For shells (like fish) that do not support verbatim single-quoted strings, SAP will properly escape all necessary characters (e.g., for fish shell, \ ➡ \\ & ' ➡ \')

For shellCommand, the commands should be output expecting to be inserted into a string to be passed to eval. From the Swift perspective, the strings will be handled the same way as per completion values (the actual escaping by SAP might be different, but Swift users won't see any difference).

There are many problems with how SAP 1.5.0 parses completions from Swift. Some (if not all) types of completions in the different shells split completions on spaces (so a b becomes 2 separate completions: a & b.)

It's so broken, I think it makes sense to only follow the consistent, fixed rules that will be followed by my PR (and will be considered bugs if not obeyed).

This might technically require SAP to bump by a major version.

[rgoldberg]

Index issues:

fish 3-: words are unquoted, but "cursor index in current word" is based on the verbatim characters, so it could be greater than it should be. Fixed in fish 4+. If someone else wants to improve this for fish 3-, they can.

zsh: "cursor index in current word" is one greater than correct if the cursor is immediately after a backslash used as an escape (i.e. zsh reports the cursor as after the character that is escaped by the backslash, instead of after the backslash itself).

[rgoldberg]

@natecook1000

In what format do you want text to be passed between shells & Swift in the various places where they can communicate?

The new SAP_SHELL & SAP_SHELL_VERSION environment variables will allow Swift code to know the shell & version with which Swift is interacting, so Swift code can properly parse/format verbatim text for the specific shell. It might be annoying/complex to deal with such parsing/formatting, so verbatim text should be avoided as much as possible, but it might be unavoidable in certain circumstances.

The following ignores environment variables as a communication method between shells & Swift. Anyone using them can do whatever they want with them.

It's also what I've gathered going through this all. If anything is incorrect, please let me know.

From shell to Swift

Text is sent from a shell to Swift only for custom completions, which pass the contents of part of the command line from the shell to Swift.

In this case, we must send verbatim text, otherwise there would be no way to distinguish between:

• a variable reference & a string literal that, when unquoted, looks like a variable reference ($x vs. '$x')
• a command substitution & a lookalike string literal (<(…) vs. '<(…)')
• …

Are there any bash, fish, or zsh parsing libraries in Swift that we can recommend to users?

The various shells send command lines differently in SAP 1.5.0, some in broken ways; I am fixing that.

From Swift to shell

Text sent from Swift to a shell should probably be expected to be in different formats depending on the use case.

The only places about which I know where Swift sends text to a shell are in the completion functions, and in the command, subcommand, flag & option names.

Completion functions:

• case list([String])
  • Contents of each element of the Swift [String] are a literal value of a single completion value.
  • Neither escapes nor references are supported.
    • No escapes: \\ will behave as a literal string containing two backslashes (instead of a single escaped backslash).
    • No references: $x will behave as a literal string with a $ followed by an x; it will not reference the value of a variable named x.
  • Characters that must be escaped within the shell to behave properly in the shell will be escaped by SAP (in or by the script).
  • SAP could use one of the following characters as a delimiter between elements from the [String]:
    • NUL
      • harder to implement
      • allows completions to contain newlines
    • newline
      • easier to implement
      • prevents completions from containing newlines
      • easier to debug
      • if completions shouldn't contain newlines, easier to do this
• case custom(@Sendable ([String], Int, Int) -> [String])
  • Same as case list([String])
• case default
  • Same as case list([String])
• case file(extensions: [String])
  • Contents of each element of the returned [String] are a glob, which will require correct escaping for each shell:
    • e.g., in zsh, \1*$x will match any extension that begins with a literal 1 & ends with the value of the variable x; \\1*\$x will match any extension that begins with a literal \1 & ends with a literal $x.
• case shellCommand(String)
  • Contents of Swift String are a verbatim shell script that will be run via the equivalent of zsh eval (so a bad script in the String will not break the completion script)
    • e.g., $x will reference the value of the variable named x

Command, subcommand, flag & option names should behave like case list([String]), but implementation would be much simpler if the allowable characters could be constrained. escaping whitespace, quotes, other shell symbols (e.g., $, ;, …), etc. would be exceedingly cumbersome. Can the supported characters be limited those that do not require escaping or quoting in the shell? Does the following regex specify all the characters I should test (I won't specifically limit supported characters to the following regex, but I won't test Chinese characters or emoji, even if they probably will work, unless you tell me I shoul)?

[-+._:0-9A-Za-z]

I have seen /bin/[ & /usr/local/bin/g[. [ would require escaping everywhere, but I'd really prefer to not have to deal with that possibility everywhere.

[rgoldberg]

In what order should arrays of completions be shown? The two main options would be

1. sorted (presumably lexicographically, but taking into account full numbers instead of sorting by successive digits; e.g., 10 would be sorted after 9)
2. in the original order supplied from Swift

I think the former is mainly (if not exclusively) what is done in 1.5.0, but I prefer the latter. If you want your completions to be sorted a certain way, the latter provides flexibility, while the former is restrictive.