feat: Generate expr method signatures, docs

[dangotbanned]

Closes #3563

Description

This PR builds on the work of #3466, and provides accurate signatures for alt.expr methods.

Benefits

Originally listed in #3563 (comment)

• Raising a TypeError at the time of expr definition
  • Rather than silently passing until validated within a ChartType
  • Importantly, this is within python and not javascript
• Providing a meaningful traceback, identifying both the expr and the missing argument
• Opens the door for defining default argument values
  • E.g. in waterfall_chart, there are repeat uses of "" for the elseValue
  • If this is a common case, we could use that as a default and it would be clear by the signature alone
• The current docstrings - which refer to parameters by name - would be easier to understand
  • Right now, you'd need to refer to Vega Expressions to determine the order they must appear in
• The change would not add any new requirements to currently valid specs
  • Therefore, suitable as a MINOR version feature

Screenshots

Implementation

In #3563 I proposed authoring these changes manually.
However, after revisiting a conversation I had with @jonmmease I thought it would be worth at least trying to generate the definitions.

I found that https://vega.github.io/vega/docs/expressions/ seemed to be the only place they are documented in a single place.
From my understanding, this document is manually maintained but updates occur infrequently.

There was enough consistency in the markup used to work out how various parameters aligned with python syntax https://docs.python.org/3/tutorial/controlflow.html#special-parameters.

I also used this opportunity to clean up the docs themselves, and get them closer to the rest of altair.
There is definitely more that could be explored on that front, and some of the changes here could be utilised to improve the existing generated docs.

Related

• feat: Reimplement expr as a class that is understood by IDEs #3466

Tasks

• Parse https://vega.github.io/vega/docs/expressions/
• Generate py-equivalent signatures
• Generate docs
  • Find target content
  • Clean up result
    • feat: Split doc summary, wrap body
    • Replace refs to non-altair functions
      • self reference
      • feat: Replace all Vega function refs in code blocks (See comment)
    • Replace italics with backticks for own parameters
    • fix mistune escaping
    • fix: Replace vega docs relative links
    • feat: Convert inline links to references
• Rewrite test_expr.py to account for number of arguments related
  • See thread
• build: Generate new alt.expr.__init__.py
• Split out .rst stuff commit
• Refactor vega_expr.py
  • Comments following feat: Generate expr method signatures, docs #3600 (review)
• Fix (now) missing definitions comment
  • These either
    • have their first parameter defined as regular text
    • accept no parameters

Deferred

Trying my best to keep the scope of the PR focused.
Working on this sparked a lot of ideas, so I've collected those here to possibly explore in the future:

• Inline code -> block code (comment)
• Also parsing properties from expressions.md
  • Wouldn't be that much work to add, but not a great deal of value
  • We're missing MAX_VALUE, MIN_VALUE
• tools.schemapi.vega_expr.py -> tools.vega_expr.py
  • The module is unrelated to the vega-lite schema
  • I think the review will be simpler if it stays here for now
• Soft-deprecating camelCase expr methods, replace w/ snake_case
  • Pretty sure adding an _ExprMeta.__getattr__ would make this possible
  • The current names would remain accessible (but undocumented)

(Early) Preview

Prior to 5e75051 (#3600) there wasn't any generated source on this branch.
Keeping these screenshots for a quick referenceuntil I've resolved some test issues.

Screenshots

[dangotbanned]

Is that something that is expected? Or was it naive to expect that something as such could work?

@mattijn I managed to get all of these working correctly:

# ruff: noqa: B018
import altair as alt

# import altair.expr as ae
from altair import expr
from altair import expr as ae
from altair.expr import expr as expr_cls

alt.expr.scale
expr.scale
ae.scale
expr_cls.scale

Edit

AFAIK this behavior isn't something I'd expect to have changed vs main - but would be interesting if so

[mattijn]

Thanks! It works as expected, but that also means it surface the following issue:
This works

import polars as pl
import altair as alt
from altair import expr as ae

df = pl.DataFrame(
    {
        "x": [0.32, 0.86, 0.10, 0.30, 0.47, 0.0, 1.0, 0.91, 0.88, 0.12],
        "color": list("ACABBCABBA"),
    }
)

param_array = alt.param(name="my_array", value=["B", "A", "C"])
alt.Chart(df).mark_tick(thickness=10).encode(
    x=alt.X("x"), 
    color=alt.Color("color").scale(domain=param_array)
).add_params(param_array)

But this will not:

param_array = alt.param(name="my_array", value=["B", "A", "C"])
param_sort = alt.param(name="my_sorted_array", expr=ae.sort(param_array))
alt.Chart(df).mark_tick(thickness=10).encode(
    x=alt.X("x"), 
    color=alt.Color("color").scale(domain=param_sort)
).add_params(param_array, param_sort)

Javascript Error: Unrecognized function: sort
This usually means there's a typo in your chart specification. See the javascript console for the full traceback.

And while ae.sort(param_array) is now recognised as valid, since it is now auto-generated from the docs (great!), it is not yet released as part of the current accepted Vega version within Altair therefore raising an JavaScript error.

The compiled vega-lite specification looks correctly though:

{
  "config": {"view": {"continuousWidth": 300, "continuousHeight": 300}},
  "data": {"name": "data-1d2849971aacb1c06bb7718af005021c"},
  "mark": {"type": "tick", "thickness": 10},
  "encoding": {
    "color": {
      "field": "color",
      "scale": {"domain": {"expr": "my_sorted_array"}},
      "type": "nominal"
    },
    "x": {"field": "x", "type": "quantitative"}
  },
  "params": [
    {"name": "my_array", "value": ["B", "A", "C"]},
    {"name": "my_sorted_array", "expr": "sort(my_array)"}
  ],
  "$schema": "https://vega.github.io/schema/vega-lite/v5.20.1.json",
  "datasets": {
    "data-1d2849971aacb1c06bb7718af005021c": [
      {"x": 0.32, "color": "A"},
      {"x": 0.86, "color": "C"},
      {"x": 0.1, "color": "A"},
      {"x": 0.3, "color": "B"},
      {"x": 0.47, "color": "B"},
      {"x": 0, "color": "C"},
      {"x": 1, "color": "A"},
      {"x": 0.91, "color": "B"},
      {"x": 0.88, "color": "B"},
      {"x": 0.12, "color": "A"}
    ]
  }
}

[dangotbanned]

But this will not:

...

And while ae.sort(param_array) is now recognised as valid, since it is now auto-generated from the docs (great!), it is not yet released as part of the current accepted Vega version within Altair therefore raising an JavaScript error.

The compiled vega-lite specification looks correctly though:

{
  "config": {"view": {"continuousWidth": 300, "continuousHeight": 300}},
  "data": {"name": "data-1d2849971aacb1c06bb7718af005021c"},
  "mark": {"type": "tick", "thickness": 10},
  "encoding": {
    "color": {
      "field": "color",
      "scale": {"domain": {"expr": "my_sorted_array"}},
      "type": "nominal"
    },
    "x": {"field": "x", "type": "quantitative"}
  },
  "params": [
    {"name": "my_array", "value": ["B", "A", "C"]},
    {"name": "my_sorted_array", "expr": "sort(my_array)"}
  ],
  "$schema": "https://vega.github.io/schema/vega-lite/v5.20.1.json",
  "datasets": {
    "data-1d2849971aacb1c06bb7718af005021c": [
      {"x": 0.32, "color": "A"},
      {"x": 0.86, "color": "C"},
      {"x": 0.1, "color": "A"},
      {"x": 0.3, "color": "B"},
      {"x": 0.47, "color": "B"},
      {"x": 0, "color": "C"},
      {"x": 1, "color": "A"},
      {"x": 0.91, "color": "B"},
      {"x": 0.88, "color": "B"},
      {"x": 0.12, "color": "A"}
    ]
  }
}

Well spotted @mattijn thank you!

I'll follow this up in #3600 (comment)

[dangotbanned]

Note

@mattijn I'm moving into draft to utilize #3633 and simplify #3600 (comment)

[mattijn]

Thanks for this PR. Looks good now!

[dangotbanned]

Thanks for this PR. Looks good now!

Thanks for your help with this PR @mattijn

One last thing I'd like to check before merging, from the Deferred section of the description

• tools.schemapi.vega_expr.py -> tools.vega_expr.py
  • The module is unrelated to the vega-lite schema
  • I think the review will be simpler if it stays here for now

I can either:

1. Leave the path unchanged
2. Move vega_expr.py in a follow-up PR
3. Move vega_expr.py now and then merge

[mattijn]

Option 3 is OK 👍