nu --ide-ast bugs

[maxim-uvarov]

Describe the bug

nu --ide-ast doesn't highlight some tokens properly

How to reproduce

Create a .nu file with a content below and open it in vscode:

mkdir test; cd test # cd is not highlighted

def 'abc' [] {} # def and 'abc' are incorrect

def abcd [] {} # all is correct here

Expected behavior

I expected nu to highlight ; cd, and def 'abc' properly

Screenshots

No response

Configuration

key	value
version	0.85.0
branch
commit_hash
build_os	macos-aarch64
build_target	aarch64-apple-darwin
rust_version	rustc 1.70.0 (90c541806 2023-05-31)
rust_channel	stable-aarch64-apple-darwin
cargo_version	cargo 1.70.0 (ec8a8a0ca 2023-04-25)
build_time	2023-09-23 05:58:06 +00:00
build_rust_channel	release
allocator	mimalloc
features	dataframe, default, sqlite, trash, which, zip
installed_plugins

Additional context

No response

[fdncred]

I don't believe --ide-ast controls highlighting. It's more likely the regular expressions in the vscode extension, since it currently doesn't support semantic highlighting. I'll move this issue over to that repo.

[fdncred]

@glcraft When you get a moment, could you look at this issue?

[maxim-uvarov]

@fdncred you are right, nu --ast gives correct results. I thought I saw something similar to this problem analyzing history. But this example doesn't have problems. Thank you!

> "mkdir test; cd test # cd is not highlighted

def 'abc' [] {} # def and 'abc' are incorrect

def abcd [] {} # all is correct here" | save test.txt

> nu --ide-ast test.txt | from json
╭content┬───────shape────────┬──────span───────┬type─╮
│ mkdir │ shape_internalcall │ ╭───────┬───╮   │ ast │
│       │                    │ │ end   │ 5 │   │     │
│       │                    │ │ start │ 0 │   │     │
│       │                    │ ╰───────┴───╯   │     │
│ test  │ shape_directory    │ ╭───────┬────╮  │ ast │
│       │                    │ │ end   │ 10 │  │     │
│       │                    │ │ start │ 6  │  │     │
│       │                    │ ╰───────┴────╯  │     │
│ cd    │ shape_internalcall │ ╭───────┬────╮  │ ast │
│       │                    │ │ end   │ 14 │  │     │
│       │                    │ │ start │ 12 │  │     │
│       │                    │ ╰───────┴────╯  │     │
│ test  │ shape_directory    │ ╭───────┬────╮  │ ast │
│       │                    │ │ end   │ 19 │  │     │
│       │                    │ │ start │ 15 │  │     │
│       │                    │ ╰───────┴────╯  │     │
│ def   │ shape_internalcall │ ╭───────┬────╮  │ ast │
│       │                    │ │ end   │ 48 │  │     │
│       │                    │ │ start │ 45 │  │     │
│       │                    │ ╰───────┴────╯  │     │
│ 'abc' │ shape_string       │ ╭───────┬────╮  │ ast │
│       │                    │ │ end   │ 54 │  │     │
│       │                    │ │ start │ 49 │  │     │
│       │                    │ ╰───────┴────╯  │     │
│ []    │ shape_signature    │ ╭───────┬────╮  │ ast │
│       │                    │ │ end   │ 57 │  │     │
│       │                    │ │ start │ 55 │  │     │
│       │                    │ ╰───────┴────╯  │     │
│ def   │ shape_internalcall │ ╭───────┬────╮  │ ast │
│       │                    │ │ end   │ 95 │  │     │
│       │                    │ │ start │ 92 │  │     │
│       │                    │ ╰───────┴────╯  │     │
│ abcd  │ shape_string       │ ╭───────┬─────╮ │ ast │
│       │                    │ │ end   │ 100 │ │     │
│       │                    │ │ start │ 96  │ │     │
│       │                    │ ╰───────┴─────╯ │     │
│ []    │ shape_signature    │ ╭───────┬─────╮ │ ast │
│       │                    │ │ end   │ 103 │ │     │
│       │                    │ │ start │ 101 │ │     │
│       │                    │ ╰───────┴─────╯ │     │
╰───────┴────────────────────┴─────────────────┴─────╯