extendedglob pattern with `((` breaks syntax highlighting

[mr-salty]

I ran into this in my own scripts but I found an example from the zsh docs as some proof I'm not writing something completely weird (see the section on s,e which must be enclosed in parentheses)

if [[ "${path}" = *((#s)|/)test((#e)|/)* ]]; then
  echo "match"
fi

this breaks highlighting for the rest of the file (echo) and (fi) should be a red color:

note the *, |, and # do not matter, if [[ "${path}" = ((a)b) ]]is also broken - although if [[ "${path}" = ((a)) ]] is ok (I know those latter two don't make much sense, just to illustrate).

I will try to send a PR, but the syntax file is pretty daunting so I thought I'd see if there was an easy/obvious solution.
I do have the latest version (2020-11-21)

[danielshahaf]

Does it help if you change <=> to <=>* in the syntax file? That's not a fix; only a scaffold for debugging.

The <=> was added in the first commit after the 2021-11-20 timestamping, b01ddc0.

[danielshahaf]

Workaround: add a pair of empty quotes.

% [[ foo == ((#s)f)* ]] && echo yes
yes
% [[ foo == (''(#s)f)* ]] && echo yes
yes

[mr-salty]

Thanks, <=>* does not help. and the '' does not work on my original expression if [[ "${path}" = ''*((#s)|/)test((#e)|/)* ]]; then (although it does work for me using your example). I haven't had time to dig into it yet but since it doesn't sound like an obvious fix I can do that when I get a chance.

[danielshahaf]

Add the two single quotes between each pair of opening parentheses:

if [[ "${path}" = *(''(#s)|/)test(''(#e)|/)* ]]; then :; fi
echo echo

[danielshahaf]

By the way, the reason <=>* didn't help would be that I missed that your example had two pairs of opening parentheses. The asterisk would have excluded the first pair from matching as zshMathSubst, but not the second one.

[danielshahaf]

As to a fix, how about extending the negative lookbehind to also match any character other than interword whitespace, the last characters of command separator tokens (!;&|), and $? (A positive lookbehind may be easier, if ((…)) at the start of a line is allowed.)

((…)) is only valid at command position, and this would exclude zshMathSubst from matching when random stuff is in front of two opening parentheses.

This should DTRT on cases such as if ((…)) and repeat N ((…)), though it won't fix echo ((…)) (which is a glob pattern, or possibly glob qualifiers) or [[ haystack = ((needle|…)…) ]]. It also won't affect the aforementioned ((pwd;); ls) case.

[lacygoill]

((…)) is only valid at command position

Sorry, I don't know the zsh syntax well enough, nor the shell in general, but can a command position be easily described with one regex, or possibly a few?

Can you give a few examples covering all the contexts in which a command can appear?

If you can, maybe we could write a few rules to match this command position. It would not be used to highlight any character; just to match a zero-width position. Then, using nextgroup, we could assert what is allowed, including this (( syntax. The latter could be in turn matched with another rule using the contained argument, to prevent it from matching anywhere.

[danielshahaf]

Can you give a few examples covering all the contexts in which a command can appear?

Sure. Off the top of my head:

• At the very start of a script

• After command separator tokens and after some reserved words and builtins https://github.com/zsh-users/zsh-syntax-highlighting/blob/c7caf57ca805abd54f11f756fda6395dd4187f8a/highlighters/main/main-highlighter.zsh#L392-L417

• After a scalar or array assignment in command position

• After a case pattern

• foo in repeat "42" foo

• foo in function { foo }

• never inside quoted strings or heredocs

• not if one of the following words is ()

There's plenty of examples in upstream's test suite in the A*.ztst section, and in z-sy-h. (For the former, there's a ztst.vim in that repository that highlights *.ztst files and delegates to zsh.vim, but beware of bugs in the delegation.)

Sorry, I don't know the zsh syntax well enough, nor the shell in general, but can a command position be easily described with one regex, or possibly a few?
⋮
If you can, maybe we could write a few rules to match this command position. It would not be used to highlight any character; just to match a zero-width position. Then, using nextgroup, we could assert what is allowed, including this (( syntax. The latter could be in turn matched with another rule using the contained argument, to prevent it from matching anywhere.

This sounds promising.

From a syntax perspective, (( has several different meanings:

• Arithmetic expression: (( 2 + 2 == 4 )) && echo equals

• Part of the syntax of the arithmetic for loop: for (( … ; … ; … ))

• Part of arithmetic substitutions: echo $(( 2 + 2 ))

• Two subshells: zmodload zsh/system; f() echo $sysparams[pid]; ((f;); f;); f

• Start a pattern in various contexts: [[ foo == ((bar|foo)|baz) ]], echo ((.))(N), case foo in …

What further complicates the issue is that case patterns in zsh may take one of two forms: either (foo) or just foo). The former case also allows whitespace, e.g., case foo in ( foo ) echo matched ;; esac.

For all I know, it might be possible to disable command position highlighting after case foo in without implementing the zero-width matcher for all command positions in general. Would this be a lower-hanging fruit?

[danielshahaf]

• not if one of the following words is ()

Sorry, the rules here are not as simple as that. ls() … defines a function called ls, but time() … times the run of an anonymous function. (time is a reserved word, not a builtin.)

[lacygoill]

For all I know, it might be possible to disable command position highlighting after case foo in without implementing the zero-width matcher for all command positions in general. Would this be a lower-hanging fruit?

Yes, although here, it seems the issue is with the [[ command. It's not matched by anything; so anything can match, including something wrong. We need to match it. Ideally, we would only match [[ as a command if it's in the right position, but that's above my zsh level. For the moment, we could assume that [[ is a command if it's immediately followed by a whitespace, and if it's later followed by ]] on the same line (the latter being itself preceded by a whitespace).

We don't need to highlight this; just match it, and recognize it as a syntax item of its own. Nothing will match inside, unless we explicitly allow it to. This way, the problematic (( won't match. It seems that [[ expects an expression. We could allow a cluster (say @zshExpr) of syntax items. Inside, we just need to include any syntax which is valid inside a zsh expression; for a start, zshGlob and zshString.

As a suggestion, here is a patch:

diff --git a/syntax/zsh.vim b/syntax/zsh.vim
index bab89b9..00e3da9 100644
--- a/syntax/zsh.vim
+++ b/syntax/zsh.vim
@@ -175,6 +175,9 @@ syn region  zshSubst            matchgroup=zshSubstDelim start='\${' skip='\\}'
                                 \ end='}' contains=@zshSubst,zshBrackets,zshQuoted,zshString fold
 exe 'syn region  zshOldSubst    matchgroup=zshSubstDelim start=/`/ skip=/\\[\\`]/ end=/`/ contains='.s:contained. ',zshOldSubst fold'
 
+syn region  zshConditionalExpr  start=/\[\[\s/ end=/\s\]\]/ contains=@zshExpr oneline
+syn cluster zshExpr             contains=zshGlob,zshString
+
 syn sync    minlines=50 maxlines=90
 syn sync    match zshHereDocSync    grouphere   NONE '<<-\=\s*\%(\\\=\S\+\|\(["']\)\S\+\1\)'
 syn sync    match zshHereDocEndSync groupthere  NONE '^\s*EO\a\+\>'

[danielshahaf]

Yes, although here, it seems the issue is with the [[ command.

My bad; I meant to post that remark to issue #39.

It's not matched by anything; so anything can match, including something wrong. We need to match it. Ideally, we would only match [[ as a command if it's in the right position, but that's above my zsh level.

[[ is a reserved word (see the output of type '[['), like if and while; it's valid wherever either of these two is valid: at command position.

For the moment, we could assume that [[ is a command if it's immediately followed by a whitespace, and if it's later followed by ]] on the same line (the latter being itself preceded by a whitespace).

We don't need to highlight this; just match it, and recognize it as a syntax item of its own. Nothing will match inside, unless we explicitly allow it to. This way, the problematic (( won't match. It seems that [[ expects an expression.

[[ expects a conditional expression: https://zsh.sourceforge.io/Doc/Release/Conditional-Expressions.html#Conditional-Expressions

For context, [[ and [ are very similar, with one important difference: [ is a builtin and [[ is a reserved word. That means [ is subject to the normal quoting and word-splitting rules, whereas [[ gets to interpret the command line much earlier than ordinary commands, functions, and builtins:

% [ foo < bar ] 
zsh: no such file or directory: bar
% [[ foo < bar ]] ; echo $?
1
% 

You can see that the first case interpreted < as a redirection operator, same as in nl </etc/passwd -ba, whereas the second case interprted it as a comparison operator.

We could allow a cluster (say @zshExpr) of syntax items. Inside, we just need to include any syntax which is valid inside a zsh expression; for a start, zshGlob and zshString.

Be careful here. [[ doesn't interpret globs like other commands do. Look:

% cd $(mktemp -d)
% [[ ?? = '??' ]]
yes
% [ ?? = '??' ]
zsh: no matches found: ??

[[ interpreted the first ?? as a literal string. [ interpreted the first ?? as a glob pattern, the same way ls ?? = '??' ] does.

By the way, feel free to stop by #zsh on libera IRC or zsh-users@zsh.org with any questions you may have about zsh syntax. Do mention you're working on vim.zsh :)

As a suggestion, here is a patch:

Confirmed it fixes #33 (comment). Looking for [[\s seems reasonable. Thanks for looking into this!

[danielshahaf]

I can reproduce this without the EXTENDED_GLOB option. Test case:

[[ "" = ((foo|bar)baz) ]]
integer i

integer should be highlighted, but it's not. Bisected to 907cdc0, the fix for #31.

[arp242]

One solution might be to not highlight (( .. )) at all. Right now [ .. ] and [[ .. ]] isn't highlighted, so it's a bit inconsistent.

Of course, you could also say that the square brackets should get highlighted and that it's a shortcoming that it doesn't; the Bourne shell/bash syntax does highlight them. Personally I'd be fine with just not highlighting it though.

As an aside, the way I'd fix the syntax error is with a comment rather than modifying the pattern:

if [[ "${path}" = *((#s)|/)test((#e)|/)* ]]; then  # Fix Vim syntax: ))))