[Go] preliminary support for Go 1.18

[mitranim]

Features:

• Support for ~.
• Support for any and comparable.
• Support for type unions.
• Support for general interfaces (with type unions).
• Support for type parameters in generic functions.
• Support for type parameters in method receivers of generic types.
• Support for type arguments following type names.
• Support for type parameters in type declarations.
• Support for embedded parametrized types in struct definitions.
• Support for type arguments between function name and function arguments.
• Support for invalid.illegal for dangling )]}.
• Slightly better meta scopes in type definitions.

Known regressions:

• Embedded interfaces are no longer scoped as "inherited class", due to ambiguities.
• "Namespace" idents such as ident. no longer support a comment between identifier and dot.
• Last ident in parens followed by opening paren is no longer scoped as a function call, due to false positives and ambiguities.

Known issues and limitations:

• Detection of type parameter lists and type argument lists is incomplete. In some places it relies on regex lookahead, which doesn't work across multiple lines. There are also edge case ambiguities which are supported by the Go compiler but not by our syntax.

Misc tweaks in comments:

• Convert anon to named contexts for compatibility with overrides.
• Avoid false positives when scoping javadoc bullets.
• Use prototype to reduce repetition.

Closes #3281.

[jrappen]

• It might make sense to move tests to ./Go/tests/ and split them up.
• You're still using syntax v1. Was there any reason not make use of newer v2 syntax engine features?

[mitranim]

• It might make sense to move tests to ./Go/tests/ and split them up.

I have no strong opinion. It has obvious upsides but also some downsides. Search-and-replace opens N files, split-panel becomes slightly more annoying to use.

• You're still using syntax v1. Was there any reason not make use of newer v2 syntax engine features?

No reason. Updated to v2.

[deathaxe]

FWIW There's no hard restriction with regards to used syntax features and syntax version. V2 just gates some backward incompatible changes to the syntax engine, which require a syntax to opt-in. Most of them are related to how meta scopes are applied in push/set/pop scenarios.

[deathaxe]

Here's another set of suggestions.

[jrappen]

Appreciate the effort to name contexts.

[TerminalFi]

Is this also going to support syntax based code folding ?

[jrappen]

@TheSecEng Currently no changes for that, yet.

@mitranim For reference the master branch already has some commits for the new code folding stuff in other languages.

[jrappen]

Maybe address #3228 as well with this PR?

[deathaxe]

Syntax based code folding should work pretty well, once each pushed context receives a meta scope so neiboured parens can be detected as separate folding begin/end markers.

[deathaxe]

The linked issue still contains some open tasks.

Do you plan to integrate them in this PR?

In case those are planned for future PRs, would you mind adding some small reminder issues with existing limitations and possible enhancements to certain code constructs - maybe including small examples?

[mitranim]

Apologies for slow responses. The linked issue had multiple progress reports which are all outdated, and it wasn't meant to be a progress status of this PR. However, there are other known defects. I'm building a list now.

[mitranim]

Known defects / missing features:

• In parameter lists, types with type arguments in square brackets (inline type instantiation) should be treated as types, rather than regular parameters.

interface{
  Method(Type[TypeArg])
}

func(Type, Type[TypeArg])

• In embedded interfaces with type arguments, type arguments should be scoped as variable.other.type rather than variable.other.

interface{
  EmbeddedInterface[TypeArg]
}

• "Function call" detection should work with type arguments of arbitrary complexity, rather than just with identifiers.

FuncName[Type[TypeArg]]()

• Parameter lists should support types of arbitrary complexity such as [][]Ident.

func FuncName(param [][]Type)

• In calls to namespaced functions or methods (which are syntactically identical), when there is a list of multiple type parameters separated by commas, type parameters should be scoped as variable.other.type rather than variable.parameter.type.

namespace.FuncName[TypeArg, TypeArg]()

• Calls to type-parametrized functions should be recognized as function calls even when the function name is namespaced, rather than only when the function name is bare.

namespace.FuncName[TypeArg]()

[deathaxe]

This PR provides many improvements with regards to type support.

Not supporting block quotes everywhere in fully-qualified identifiers is acceptable. Many other syntaxes don't do so at all, even though valid by language specs.

Work on Java syntax teached me required complexity needed to handle all those typing and generics "perfect" especially with regards to multi-line support.

So I think we could add this PR as improvement and add an issue with the open "TODOs" to come back to them later. Maybe fixing them in smaller pieces?

[mitranim]

I've been using this implementation for months, and while it's still incomplete, it's a massive improvement over not having support for Go 1.18. I would suggest merging this sooner, to improve user experience in the next stable build, and then I'll open a new issue for the remaining unresolved problems.

[deathaxe]

Could you probably have a look at the conflicts?

[jrappen]

Maybe you could add to the commit message (the opening comment) that you're addressing RFC issue 2569 with the scopes for doc blocks.

---

Edit: disregard, I meant a different one.

[mitranim]

Maybe you could add to the commit message (the opening comment) that you're addressing RFC issue 2569 with the scopes for doc blocks.

This confuses me. The issue seems to be about syntax highlighting inside docblocks, for stuff like @param{type}. Our Go syntax does not support this and hopefully never will, because Go generates documentation from source code, making Javadoc unnecessary. Not sure if this RFC applies.

[jrappen]

This confuses me.

Sorry, disregard. I meant a different one. I'll look it up.

[jrappen]

Found it, there was some discussion in PR 3291 about the use of punctuation.definition.comment (in JSON). No need to link it, though.

[jrappen]

I'd further like to suggest changing:

  # ...
  set: [one, two]

to:

  # ...
  set:
    - one
    - two

as it's easier to read.

---

Also, what about using linebreaks for vars as in:

  keyword: |-
    (?x:
      \b(?:
        break|
        case|chan|const|continue|
        default|defer|
        else|
        fallthrough|for|func|
        go|goto|
        if|import|interface|
        map|
        package|range|
        return|
        select|struct|switch|
        type|
        var
      )\b
    )

or those longer match patterns for word groups.

---

Maybe you could add comment block begin&end scopes to the folding rules in the fold metadata.

---

Maybe add a test for the indentation rules and symbols. Just a heads up, tests for negative symbols are broken, i.e. using the CI test binary and @@@ none as the github workflow does here.

---

... we could do all this later though, too.

[mitranim]

For set: [one, two] that easily fits under 80 char, I prefer a one-liner, but that's very subjective. I wouldn't mutiny if someone changed it later. Not seeing a strong convention and don't have a strong opinion.

For long regexes that are lists of words such as (?:one|two|three), I subjectively prefer a one-liner for code compactness. They tend to appear in variables, and you can easily scroll past a one-liner, while a multi-liner requires you to "parse" code. However, a flat multi-line structure makes Git diffs more useful:

  predeclared_type: |-
    \b(?x:
-     bool
+     any
+     |bool
      |byte
+     |comparable
      |complex64
      |complex128
      ...
    )\b

...as opposed to:

-  predeclared_type: \b(?:byte|complex64|complex128| ...
+  predeclared_type: \b(?:any|bool|byte|comparable|complex64|complex128 ...

So if we're changing variable lists, it should be by flattening them vertically, IMO.

[deathaxe]

For set: [one, two] that easily fits under 80 char

I agree, one-liners beeing well suited in case they fit into 80 chars boundary.

Multi-line form got a bit more popular just because of many contexts being pushed at once in syntaxes like Java or JavaScript. They may produce more readable code, if context names get longer.

I've tried to keep with one style recently, but there's no strong convention about it, nor a need for one.

a flat multi-line structure makes Git diffs more useful

Agree. With a look at Ruby, which makes already use of it, and some experiments with applying it to PHP, I'd be a bit concerned about the huge amount of lines added - especially for syntaxes with insane long lists of builtin funcs/vars/... (such as PHP).

[TerminalFi]

Been using this for a month or so. Great work!

[michaelblyons]

@patrickrgaffney @jakelucas Any comments?

[jakelucas]

@patrickrgaffney @jakelucas Any comments?

I can only assume I've been pinged here by mistake
Looks good to me