[DOCS] Update ES|QL generated docs to consistently use the `applies_to` metadata #128576

craigtaverner · 2025-05-28T13:26:22Z

Summary

This PR updates the documentation generation system for ESQL functions to correctly generate applies_to metadata in DocsV3, namely to support multiple versions and lifecycles.

This metadata is needed to signal version differentiation in the new docs, because we don't publish version-branched docs in 9.x (we don't duplicate every page for every minor).

For example, if something is preview in 9.0 and GA in 9.1 we need all this information on single, live 9.x page, that's what applies_to enables. Importantly, this also enables us to correctly signal serverless feature availability in the new docs.

We need different levels of applies_to discrimination for this:

Important

Once this PR is merged, we will use this new documentation generation logic to properly GA full-text search functions in a follow-up PR.

Key changes

1. Documentation generation code

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/Example.java - Added applies_to() field for version-specific examples
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/DocsV3Support.java - Updated to generate standardized applies_to blocks instead of manual callouts

Note

DocsV3Support.java contains the most significant docs gen code changes:

Total additions: ~57 lines
Total deletions: ~63 lines
Total line edits: ~120 lines
Net change: ~-6 lines (the file got slightly smaller)

2. Function status corrections

All full-text search functions: now correctly marked as preview with proper applies_to syntax
Spatial functions (ST_*): Added missing preview = true flags, and now use proper applies_to syntax
A few other functions weren't correctly identifying inline version changes such as ToUpper supporting MVs in 9.1 only

3. Generated documentation updates

As a result all of the function code adjusted results in updated markdown and Kibana JSON file updates

Result

The documentation system now correctly generates version-aware metadata, enabling proper lifecycle tracking across versions. All full-text search functions are consistently marked as technical preview with the correct applies_to syntax, to enable us flipping them to GA for 9.1.

👀 URL preview

- Add PREVIEW annotations to all preview functions - Update docs generation logic to use annotations instead of preview boolean - Keep existing preview boolean for Kibana

Match.java: Updated the options parameter description MultiMatch.java: Updated the options parameter description ToLower.java: Reformatted parameter description and updated version annotation ToUpper.java: Removed the appliesTo annotation entirely and reformatted parameter description

…averner/elasticsearch into esql_docs_real_applies_to

This reverts commit 183b131.

leemthompo · 2025-06-21T08:40:33Z

Made this PR easier to review:

by removing some irrelevant typo/formatting noise which means less Kibana files to distract reviewer
removing GAing search functions from the scope of this PR, that will be a fast follow
adding much more context and links to the PR description

…' into esql_docs_real_applies_to

nik9000 · 2025-06-21T11:59:51Z

I just merged main in to pick up my changes. Looks like @leemthompo already did. but doesn't hut I guess.

craigtaverner

Overall I think this is heading in the right direction and is an improvement, but I did see a number of preview items that seemed to be wrong. Also, I'm finding the use of preview confusing, sometimes with a version sometimes without, and the choice of version does not match any pattern I can discern.

craigtaverner · 2025-06-23T11:25:09Z

docs/reference/query-languages/esql/_snippets/functions/layout/match.md

-:::
+```{applies_to}
+stack: preview 9.0.0
+serverless: preview


Was the previous serverless:ga incorrect?

we will flip all the search functions GA for both in fast follow up PR

++ to Craig - it'd be nicer if the Serverless: GA wasn't changed in this PR

Also applies to the other full text search functions.

This technically doesn't change the serverless GA, because that note was only about optional named parameters.

The big :::{warning}Do not use on production environments. banner means Match is currently still in preview in the docs

It wasn't actually, the live docs still say tech preview currently

docs/reference/query-languages/esql/_snippets/functions/layout/multi_match.md

docs/reference/query-languages/esql/_snippets/functions/layout/round_to.md

docs/reference/query-languages/esql/_snippets/functions/layout/st_distance.md

craigtaverner · 2025-06-23T11:35:55Z

docs/reference/query-languages/esql/_snippets/functions/parameters/to_upper.md

@@ -3,5 +3,5 @@
 **Parameters**

 `str`
-:   String expression. If `null`, the function returns `null`. The input can be a single- or multi-valued column or an expression.
+:   String expression. If `null`, the function returns `null`. The input can be a single-valued column or expression, or a multi-valued column or expression {applies_to}`stack: ga 9.1.0`. 


Definitely an area for improvement, but I guess this is the best we can get at the moment.

craigtaverner · 2025-06-23T13:23:23Z

...ql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/ToLower.java

-            ) }
+        examples = {
+            @Example(file = "string", tag = "to_lower"),
+            @Example(file = "string", tag = "to_lower_mv", applies_to = "stack: ga 9.1.0") }


I like this approach. I think we could improve the rendering, but this is a good first step.

docs eng will handle rendering for us :)

craigtaverner · 2025-06-23T13:24:31Z

...ql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/string/ToUpper.java

-        appliesTo = {
-            @FunctionAppliesTo(
-                lifeCycle = FunctionAppliesToLifecycle.COMING,
-                version = "9.1.0",
-                description = "Support for multivalued parameters is only available from 9.1.0"
-            ) }


Shouldn't it say serverless, so we know it is not in stateful?

.../esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/DocsV3SupportTests.java

craigtaverner · 2025-06-23T13:28:04Z

.../esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/DocsV3SupportTests.java

-            ###### DISCONTINUED 9.0.0
-            The ability to count the number of emojis in a string has been discontinued since 9.0.0
-            :::
+            ```{applies_to}


This test originally deliberately tried to test a complex example, and the new version is extremely simple. Perhaps we could think of a new complex example? At least where stack and serverless are both mentioned?

💯 in a follow up 👍

💯 — will follow up

craigtaverner · 2025-06-23T13:29:07Z

.../esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/DocsV3SupportTests.java

@@ -418,7 +404,7 @@ public static class TestClass extends Function {
                @Example(description = """
                    To count the number of times an expression returns `TRUE` use a
                    <<esql-where>> command
-                    to remove rows that shouldn’t be included""", file = "stats", tag = "count-where"),
+                    to remove rows that shouldn't be included""", file = "stats", tag = "count-where"),


This is wrong. We took the decision in March to put in the correct typesetting apostrophe because MD does not render it correctly, while asciidoc does. This appears to reverse that decision.

unintended side effect, and unfortunately a lot less important than being able to GA the search functions for 9.1

- Revert CATEGORIZE back to GA (remove preview flags that were incorrectly added) - Revert spatial functions (ST_CONTAINS, ST_DISJOINT, ST_INTERSECTS, ST_WITHIN) back to GA - Revert match operator to show PREVIEW status correctly - Revert MatchOperator appliesTo annotation to PREVIEW lifecycle - Update TO_LOWER/TO_UPPER parameter descriptions for clarity

alex-spies

Thanks for the iteration @leemthompo , much easier to review now.

I have some minor remarks, but please consider this unblocked and merge after handling my comments at your own discretion.

alex-spies · 2025-06-23T15:08:31Z

...lugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/Term.java

@@ -50,6 +52,7 @@ public class Term extends FullTextFunction implements PostAnalysisPlanVerificati
    @FunctionInfo(
        returnType = "boolean",
        preview = true,
+        appliesTo = { @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0") },


How do we make sure we don't forget to update this? To release into tech preview, the workflow right now is to link in the generated docs and move the function out of the snapshot-only list; seems like updating this annotation would be easy to forget.

In contrast, for TO_UNSIGNEDLONG and most other tech preview functions, we just didn't provide a version.

...lugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/DocsV3Support.java

alex-spies · 2025-06-23T15:49:37Z

.../esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/DocsV3SupportTests.java

-                    version = "9.0.0",
-                    description = "The ability to count the number of emojis in a string has been discontinued since 9.0.0"
-                ) }
+            appliesTo = { @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.GA, version = "9.1.0") }


++ to what Craig said, it'd be great to make this example more complex again.

Definitely, work for follow up

alex-spies · 2025-06-23T15:52:46Z

docs/reference/query-languages/esql/_snippets/functions/layout/match.md

-:::
+```{applies_to}
+stack: preview 9.0.0
+serverless: preview


++ to Craig - it'd be nicer if the Serverless: GA wasn't changed in this PR

Also applies to the other full text search functions.

alex-spies · 2025-06-23T15:56:16Z

docs/reference/query-languages/esql/_snippets/functions/layout/term.md

-:::
-
+```{applies_to}
+stack: preview 9.0.0


To connect to my comment on the functions java code, here, it's confusing that it says 9.0.0 rather than just "preview"; even though the 9.0.0 is not going to be rendered, I as a human am confused.

alex-spies · 2025-06-23T15:58:08Z

docs/reference/query-languages/esql/kibana/definition/functions/match.json

@@ -2,7 +2,7 @@
  "comment" : "This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.",
  "type" : "scalar",
  "name" : "match",
-  "description" : "Use `MATCH` to perform a match query on the specified field.\nUsing `MATCH` is equivalent to using the `match` query in the Elasticsearch Query DSL.\n\nMatch can be used on fields from the text family like text and semantic_text,\nas well as other field types like keyword, boolean, dates, and numeric types.\nWhen Match is used on a semantic_text field, it will perform a semantic query on the field.\n\nMatch can use function named parameters to specify additional options for the match query.\nAll match query parameters are supported.\n\nFor a simplified syntax, you can use the match operator `:` operator instead of `MATCH`.\n\n`MATCH` returns true if the provided query matches the row.",


Observation: we're slimming down the docs exported to Kibana here, it seems.

Also happened in match_phrase.json.

I think whoever did those docs squished everything into Description which is incorrect for the normal docs, but smells like a hack to get everything into Kibana docs without updating the docs gen code to pull in DetailedDescription

will investigate in follow up

- added `stack: ga 9.1.0` blocks to match_phrase.md and qstr.md examples - simplified term.md version from `preview 9.0.0` to just `preview` - added `applies_to = "stack: ga 9.1.0"` to matchphrase and querystring java annotations - removed version `9.0.0` from term function annotation - deleted unused `makeCallout()` and `appendLifeCycleAndVersion()` methods

Pulled from elastic#128576 so it's easier to review.

…o` metadata (elastic#128576) - Add PREVIEW annotations to all preview functions - Update docs generation logic to use annotations instead of preview boolean * Changed stack: ga 9.1 to stack: coming in multiple places - Match.java: Updated the options parameter description - MultiMatch.java: Updated the options parameter description - ToLower.java: Reformatted parameter description and updated version annotation - ToUpper.java: Removed the appliesTo annotation entirely and reformatted parameter description - updated applies_to annotations to specify both preview 9.0.0 and ga 9.1.0 lifecycle stages - added version-specific documentation examples with applies_to markers for ga 9.1.0 features - enhanced to_lower and to_upper functions to indicate support for multi-valued expressions in ga 9.1.0 - added version guards around function parameters and descriptions using applies_to syntax - updated function parameter descriptions to indicate ga 9.1.0 availability for named parameters - use detailedDescription + and fix match_phrase applies_to syntax - strip inline applies_to from kibana docs - update roundto, matchphrase lifecycles - fix match named params info - various spatial functions are preview - unsigned long is preview - update qstr - Update TO_LOWER/TO_UPPER parameter descriptions for clarity - hide spatial functions per elastic#129839 - added `stack: ga 9.1.0` blocks to match_phrase.md and qstr.md examples - simplified term.md version from `preview 9.0.0` to just `preview` - added `applies_to = "stack: ga 9.1.0"` to matchphrase and querystring java annotations - removed version `9.0.0` from term function annotation - deleted unused `makeCallout()` and `appendLifeCycleAndVersion()` methods Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Liam Thompson <leemthompo@gmail.com> Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: Nik Everett <nik9000@gmail.com>

…#130613) * [9.1] Fixes to null-predicates and lookup-join docs (#130440 #130410) (#130537) * Support types table in lookup join docs (#130410) * Support types table in lookup join docs * Don't show a results column in the join types * Make LOOKUP JOIN types table more compact * Update docs/reference/query-languages/esql/esql-lookup-join.md Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: Alexander Spies <alexander.spies@elastic.co> * Fix the ESQL docs for the null predicates (#130440) Fix null predicates docs * This also includes moving the note above the examples in the Kibana inline docs. * Add missing knn docs * Refine newlines in kibana docs --------- Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: Alexander Spies <alexander.spies@elastic.co> * Fix Null predicates after cherry-pick * [DOCS] Update ES|QL generated docs to consistently use the `applies_to` metadata (#128576) - Add PREVIEW annotations to all preview functions - Update docs generation logic to use annotations instead of preview boolean * Changed stack: ga 9.1 to stack: coming in multiple places - Match.java: Updated the options parameter description - MultiMatch.java: Updated the options parameter description - ToLower.java: Reformatted parameter description and updated version annotation - ToUpper.java: Removed the appliesTo annotation entirely and reformatted parameter description - updated applies_to annotations to specify both preview 9.0.0 and ga 9.1.0 lifecycle stages - added version-specific documentation examples with applies_to markers for ga 9.1.0 features - enhanced to_lower and to_upper functions to indicate support for multi-valued expressions in ga 9.1.0 - added version guards around function parameters and descriptions using applies_to syntax - updated function parameter descriptions to indicate ga 9.1.0 availability for named parameters - use detailedDescription + and fix match_phrase applies_to syntax - strip inline applies_to from kibana docs - update roundto, matchphrase lifecycles - fix match named params info - various spatial functions are preview - unsigned long is preview - update qstr - Update TO_LOWER/TO_UPPER parameter descriptions for clarity - hide spatial functions per #129839 - added `stack: ga 9.1.0` blocks to match_phrase.md and qstr.md examples - simplified term.md version from `preview 9.0.0` to just `preview` - added `applies_to = "stack: ga 9.1.0"` to matchphrase and querystring java annotations - removed version `9.0.0` from term function annotation - deleted unused `makeCallout()` and `appendLifeCycleAndVersion()` methods Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Liam Thompson <leemthompo@gmail.com> Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: Nik Everett <nik9000@gmail.com> * First round cleanup of docs after cherry-picking Includes partial support for the grammer fix to the header comment. * Second round, just replacing `no` with `not` * Fix lookupjointypes test after backport --------- Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: Alexander Spies <alexander.spies@elastic.co> Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Liam Thompson <leemthompo@gmail.com> Co-authored-by: Nik Everett <nik9000@gmail.com>

WIP try make more use of real applies_to

97da1fb

craigtaverner added >docs General docs changes Team:Docs Meta label for docs team Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL labels May 28, 2025

github-actions bot deployed to docs-preview May 28, 2025 13:27 View deployment

[CI] Auto commit changes from spotless

86ff269

github-actions bot deployed to docs-preview May 28, 2025 13:35 View deployment

leemthompo added 3 commits June 3, 2025 19:56

Add @FunctionAppliesTo annotations to functions for new docs system

b30b7bd

- Add PREVIEW annotations to all preview functions - Update docs generation logic to use annotations instead of preview boolean - Keep existing preview boolean for Kibana

Tweak multimatch applies_to

4180611

Manually update list

ca242bd

github-actions bot deployed to docs-preview June 3, 2025 17:59 View deployment

[CI] Auto commit changes from spotless

d836aa8

github-actions bot deployed to docs-preview June 3, 2025 18:09 View deployment

leemthompo added 4 commits June 4, 2025 10:25

Merge branch 'esql_docs_real_applies_to' of https://github.com/craigt…

28ac656

…averner/elasticsearch into esql_docs_real_applies_to

fix conflict markers

55c8642

fix tests

4691d2b

github-actions bot deployed to docs-preview June 4, 2025 09:12 View deployment

[CI] Auto commit changes from spotless

7b7640c

github-actions bot deployed to docs-preview June 4, 2025 09:21 View deployment

remove inline applies_tos for time being

9483d72

github-actions bot deployed to docs-preview June 4, 2025 09:32 View deployment

[CI] Auto commit changes from spotless

be9b19d

github-actions bot deployed to docs-preview June 4, 2025 09:45 View deployment

leemthompo and others added 5 commits June 4, 2025 12:00

mismatch between local generation and CI generation?

183b131

Revert "mismatch between local generation and CI generation?"

5e0dec1

This reverts commit 183b131.

Add generated files, cleanup errors

5cc598f

Merge branch 'main' into esql_docs_real_applies_to

e23a7fc

Update lifecycles, cleanup

6142e78

leemthompo added 5 commits June 21, 2025 09:29

Merge branch 'main' into esql_docs_real_applies_to

37305da

spatial functions are preview

40b0fd1

unsigned long is preview

b400aef

revert GAing full-text functions, to come in followup

ef62b03

update qstr

c52896a

leemthompo and others added 3 commits June 21, 2025 10:46

multi-match docs arent published, reduce noise by reverting changes

7401620

Merge branch 'main' into esql_docs_real_applies_to

208cfac

Merge remote-tracking branch 'craigtaverner/esql_docs_real_applies_to…

c92fad9

…' into esql_docs_real_applies_to

leemthompo requested a review from alex-spies June 23, 2025 06:14

leemthompo and others added 4 commits June 23, 2025 10:27

update manual list of search functions

3a8d81d

fix typo

55ce8a8

Merge branch 'main' into esql_docs_real_applies_to

d144f83

The ST_DISTANCE function is not tech-preview

8347cd5

craigtaverner commented Jun 23, 2025

View reviewed changes

leemthompo and others added 4 commits June 23, 2025 15:38

StX and StY are not preview

09fb70e

hide spatial functions per elastic#129839

712c67d

Merge branch 'main' into esql_docs_real_applies_to

46bb433

alex-spies approved these changes Jun 23, 2025

View reviewed changes

leemthompo and others added 2 commits June 23, 2025 18:54

Merge branch 'main' into esql_docs_real_applies_to

810066a

leemthompo merged commit e6347b8 into elastic:main Jun 23, 2025
32 checks passed

kderusso pushed a commit to kderusso/elasticsearch that referenced this pull request Jun 23, 2025

ESQL: Fix misspelling in generated docs (elastic#129789)

4add2c3

Pulled from elastic#128576 so it's easier to review.

julian-elastic pushed a commit to julian-elastic/elasticsearch that referenced this pull request Jun 24, 2025

ESQL: Fix misspelling in generated docs (elastic#129789)

f018441

Pulled from elastic#128576 so it's easier to review.

mridula-s109 pushed a commit to mridula-s109/elasticsearch that referenced this pull request Jun 25, 2025

ESQL: Fix misspelling in generated docs (elastic#129789)

4d7e678

Pulled from elastic#128576 so it's easier to review.

[DOCS] Update ES|QL generated docs to consistently use the applies_to metadata #128576

[DOCS] Update ES|QL generated docs to consistently use the applies_to metadata #128576

Uh oh!

Conversation

craigtaverner commented May 28, 2025 • edited by leemthompo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key changes

1. Documentation generation code

2. Function status corrections

3. Generated documentation updates

Result

👀 URL preview

Uh oh!

leemthompo commented Jun 21, 2025

Uh oh!

nik9000 commented Jun 21, 2025

Uh oh!

craigtaverner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leemthompo Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

[DOCS] Update ES|QL generated docs to consistently use the `applies_to` metadata #128576

[DOCS] Update ES|QL generated docs to consistently use the `applies_to` metadata #128576

craigtaverner commented May 28, 2025 •

edited by leemthompo

Loading

leemthompo Jun 23, 2025 •

edited

Loading