[ES|QL] Autocomplete suggestions improvements

## Top priorities (in progress, recently done, or up-next)

```[tasklist]
### Get rid of buggy experiences (@drewdaemon )
- [x] **Improve support for Invoke completion trigger kind** (shows suggestions again when user resumes typing after the menu has closed) — https://github.com/elastic/kibana/issues/188677
- [x] **Disable suggestions when typing values** (https://github.com/elastic/kibana/pull/189228) - Today, when typing a field value between double quotes, the suggest list appears with recommendations filtered, by the string, for functions and variables. This is an interim 'fix' as we await the availability of field values. <img width="420" src="https://github.com/elastic/kibana/assets/446285/64f4346b-4681-4b93-a793-d07f4f8c00db" />
- [x] **Fix syntax highlighting for several ESQL words** — https://github.com/elastic/kibana/pull/190170
- [ ] https://github.com/elastic/kibana/issues/187184
- [x] **Show command suggestions when blank editor is refocused** (relates to https://github.com/elastic/kibana/issues/180839) — https://github.com/elastic/kibana/pull/191142
- [ ] https://github.com/elastic/kibana/issues/187247
- [ ] https://github.com/elastic/kibana/issues/189662
- [ ] **Respect `DROP`/`STATS` when suggesting variables** — Drop here suggests the avg(bytes) which is wrong as with the previous keep I am only keeping one column <img src="https://github.com/elastic/kibana/assets/17003240/ea0d8a37-23d5-45ef-87f4-2cafc67a3aa4" />
- [ ] https://github.com/elastic/kibana/issues/191111
- [ ] https://github.com/elastic/kibana/issues/191866
- [ ] **A second comma loosens suggestion constraints** - https://github.com/elastic/kibana/assets/315764/0e27577e-ea73-4a41-9b12-fe043fb4fd59
- [ ] https://github.com/elastic/kibana/issues/191876
```

```[tasklist]
### Improve key functions (@qn895 )
- [x] **Build argument suggestions based on all available call signatures** ([ref](https://github.com/drewdaemon/kibana/blob/a4b7e5a907530c1a89fc4a4e89b4d92d252e4822/packages/kbn-esql-validation-autocomplete/src/autocomplete/autocomplete.ts#L1217-L1219)) — https://github.com/elastic/kibana/pull/190828
- [x] **`BUCKET`** — In general, we want to guide users toward building date histograms attached to the time picker. https://github.com/elastic/kibana/pull/190828
- [x] **`CASE`** — Decide on an algorithm for suggesting fields and values. Improve suggestion for the first parameter (see https://github.com/elastic/kibana/issues/177258). Technical note: possibly depart from the current function definition structure for this one. It is a really strange function and we don’t really have a way to describe it to the validator and the suggestion engine. - https://github.com/elastic/kibana/pull/192135
- [x] **`COUNT`** — suggest `*` - https://github.com/elastic/kibana/pull/192205
- [ ] https://github.com/elastic/kibana/pull/191881
```

```[tasklist]
### Improve field metadata display (@qn895 )
- [x] **Indicate field type** - When writing a query and doing calculations or bucketing with STATS and BUCKET, it would be helpful to know the field type (date, number, boolean, etc.)  — https://github.com/elastic/kibana/pull/187922 <img src="https://github.com/elastic/kibana/assets/446285/8b6d868b-c1ad-42d2-ac65-a5b219f20110" />
- [x] **Add ECS field descriptions** — https://github.com/elastic/kibana/pull/187922
- [x] https://github.com/elastic/kibana/issues/181036
```

```[tasklist]
### Improve core metadata (@qn895 )
- [x] **Use Elasticsearch data types for suggestions and validation** — For historical reasons we were using simplified types (e.g. `number` instead of `double`, `long`, `int`, etc). This led to occasionally-incorrect suggestions and a lack of precision in our client-side validation. https://github.com/elastic/kibana/issues/174710
```

```[tasklist]
### Improve command suggestions
- [x] **Improve BY suggestion to include pipe** - If you add a `BY` then the +,- is suggested. We should at least suggest the pipe here — https://github.com/elastic/kibana/pull/189458
- [x] **Improve Metadata suggestions with comma** - Metadata autocomplete should use comma when suggesting the rest of the available meta fields, also should suggest the pipe (https://github.com/elastic/kibana/pull/188338)
- [x] **`SORT` command should suggest uppercase options and apply highlighting** —  https://github.com/elastic/kibana/pull/188579
- [x] **`SORT` command should suggest ASC/DESC after subsequent field** — https://github.com/elastic/kibana/pull/189959 and https://github.com/elastic/kibana/pull/193595
- [ ] **`INLINESTATS` support** — https://github.com/elastic/kibana/issues/189356  (@qn895 )
- [ ] **`RENAME`: suggest comma as well as pipe** — after `RENAME message AS var0`... or maybe don’t suggest `var0` after `AS`? (@qn895)
```


## Other ideas (blocked, not currently planned, or just ideas for the future)
```[tasklist]
### Function helpers
- [ ] https://github.com/elastic/kibana/issues/180953
- [ ] https://github.com/elastic/kibana/issues/180737
- [ ] **Constant-only parameter constraints should be inherited by nested functions** - For example, if func1's first parameter is constant-only, any nested functions should inherit that constraint: `func1(func2(shouldBeConstantOnly))`
- [ ] **Improve 'Read more' for functions** - This is contextually relevant and contains useful documentation; however, the order and readability of content could be improved <img src="https://github.com/elastic/kibana/assets/446285/d1575030-d1fc-49ab-b22c-7d6439f24d63" /> _At left: ES|QL ... right: analog example)_
- [ ] **Excel-like experience for functions** (see https://github.com/elastic/kibana/issues/180528) - Leverage the [Parameter Info](https://learn.microsoft.com/en-us/visualstudio/ide/using-intellisense?view=vs-2022#parameter-info) capabilities provided by Intellisense for a more Excel-like experience when adding functions to a query <img src="https://github.com/elastic/kibana/assets/446285/2d3a9fe0-dcf1-4006-b5c8-d02859d9b71b" />
```

```[tasklist]
### Smarter suggest
- [ ] #184679 - This is a known ES limitation at the moment (need issue link if one exists). It's worth capturing as it is a key, missing capability of the current editing experience. As seen below, the suggest lists simply contains a redundant field list. <img width="420" src="https://github.com/elastic/kibana/assets/446285/64c71ea7-4346-41a3-9b3a-a6a38fbfd193" />
- [ ] https://github.com/elastic/kibana/issues/180839
- [x] **Redisplay autocomplete when editing/typing a data source** - After adding a data source, backspace/delete some text, start typing again, autocomplete should redisplay (filtered). In the gif below, when re-typing auditbeat, I expect the suggestion menu to redisplay, filtered to what I have typed thus far. (done in https://github.com/elastic/kibana/issues/188677) <img src="https://github.com/elastic/kibana/assets/446285/1825c447-7a68-4e6b-beb3-53dcda3d52b7" />
- [ ] **Indicate when a _value_ needs to be entered** - In addition to suggesting fields, commands, and functions, it can be equally logical to enter a value such as a number. In the example below, I am desiring to enter a numerical value (bonus: a hint on the range or sampling of values would be extra delightful) <img src="https://github.com/elastic/kibana/assets/446285/8edeb58a-4be4-4abf-b320-30704609340f" />
- [ ] Suggest constant values for functions that accept constants as arguments
- [ ] **Ignore casing when sorting (?)** - When one data source uses camel case, those fields appear first. In the situation where sources use the same casing, the list appears to be combined and sorted as a single list. I sense lowercase is the best practice, but other casing does not seem to be prohibited. Upon further investigation, I don't see indices in obs or esql demo environments with camel case field names. <img src="https://github.com/elastic/kibana/assets/446285/a314d8b3-f03e-464d-9b93-672ae3f60fcd" />
- [ ] **Revisit how we handle remote indices** -  Support for remote indices was recently added. Now that it is in place, we may want to consider some enhancements with regards to scalability and organizational hierarchy. For example, clusters appear alphabetically with local indices. <img src="https://github.com/elastic/kibana/assets/446285/7c35fbc7-4bb4-439e-8aa8-89d84c47cbe7" />
```


```[tasklist]
### Environmental / Authoring helpers
- [ ] **Indicate data source for a field** - Most relevant upon adding more than one data source, the suggest list combines/mixes fields making it difficult to know which source(s) the field is coming from. Fields may exist in both - appear as one entry - or be similar in name on more than one data source. Note that the 'Ability to view field names' recommendation would help here, too. <img src="https://github.com/elastic/kibana/assets/446285/ee6ce178-8fae-4b13-9e32-c91825880cb0" />
- [x] **Remove not-very-useful 'Read more' on fields/variables** - Currently, the same 'Read more' text is shown on every field/variable. It is both redundant and lacking useful information; therefore, we should consider disabling it. <img src="https://github.com/elastic/kibana/assets/446285/82eb535b-8ade-45c5-84f0-c56aa8eb6891" />
```

```[tasklist]
### Syntactical papercuts
- [ ] https://github.com/elastic/kibana/issues/180975
- [ ] https://github.com/elastic/kibana/issues/180840
- [ ] **Fix suggestions after `+`** - In the above, after clicking the + the aggregation is suggested which is wrong. If the user runs it, ES will error out <img src="https://github.com/elastic/kibana/assets/17003240/44ea3e54-d356-410f-bd84-15204542c564" />
- [ ] **Autocomplete shouldn't suggest on `=` when it is part of `<=` or `>=`** - https://github.com/elastic/kibana/assets/315764/4635d08f-5618-4091-b8fd-925baff95b77
```

#### Related, previously issues
- [x] **Support `data_part` as proposed in docs**
These date functions can be very tricky so we would like to help our users a bit with autocomplete. It would be great if we could suggest the `date_part` as proposed on our [official documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html#esql-date_extract
)


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ES|QL] Autocomplete suggestions improvements #176033

stratoula
openedon Feb 1, 2024

Top priorities (in progress, recently done, or up-next)

Get rid of buggy experiences (@drewdaemon )

Improve key functions (@qn895 )

Improve field metadata display (@qn895 )

Improve core metadata (@qn895 )

Improve command suggestions

Other ideas (blocked, not currently planned, or just ideas for the future)

Function helpers

Smarter suggest

Environmental / Authoring helpers

Syntactical papercuts

Related, previously issues

Assignees

Labels

Type

Projects

Milestone

Relationships

Development