Parse begin/end in ref as Expr

[mlechu]

TODO:

• The same parsing change needs to land in JuliaSyntax first due to changes in julia-syntax.scm and views.jl (Parse arr[end] differently from arr[var"end"] JuliaSyntax.jl#537)

---

Fixes #57269.

Before:

julia> dump(Meta.parse("a[end - var\"end\"]"))
Expr
  head: Symbol ref
  args: Array{Any}((2,))
    1: Symbol a
    2: Expr
      head: Symbol call
      args: Array{Any}((3,))
        1: Symbol -
        2: Symbol end
        3: Symbol end

After:

julia> dump(Meta.parse("a[end - var\"end\"]"))
Expr
  head: Symbol ref
  args: Array{Any}((2,))
    1: Symbol a
    2: Expr
      head: Symbol call
      args: Array{Any}((3,))
        1: Symbol -
        2: Expr
          head: Symbol end
          args: Array{Any}((0,))
        3: Symbol end

[Keno]

Needs a pkgeval to evaluate how widespread the macro usage of this is. We don't technically consider the AST representation fully stable, but it's quite stable in practice. If this is too widespread, my suggestion would be to provide guidance to downstream packages to accept both forms, merge this off-by-default and then re-consider turning it on next release (possibly together with the JuliaLowering changes, which might entail some disruption in macros anyway).

[vtjnash]

I feel like this is too breaking to be worthwhile. Also note that this does not fix #57269, since the context-sensitivity of the expression is unchanged, but rather may make fixing that slightly harder since the syntax form for this becomes slightly more difficult to write literally.

[mlechu]

@nanosoldier runtests()

[nanosoldier]

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

Report summary

❗ Packages that crashed

5 packages crashed only on the current version.

• The process was aborted: 1 packages
• An internal error was encountered: 2 packages
• GC corruption was detected: 1 packages
• A segmentation fault happened: 1 packages

97 packages crashed on the previous version too.

✖ Packages that failed

517 packages failed only on the current version.

• Package has syntax issues: 466 packages
• Package fails to precompile: 3 packages
• Illegal method overwrites during precompilation: 7 packages
• Package has test failures: 4 packages
• Package tests unexpectedly errored: 1 packages
• Networking-related issues were detected: 16 packages
• Tests became inactive: 5 packages
• Test duration exceeded the time limit: 15 packages

1934 packages failed on the previous version too.

✔ Packages that passed tests

12 packages passed tests only on the current version.

• Other: 12 packages

4427 packages passed tests on the previous version too.

~ Packages that at least loaded

2 packages successfully loaded only on the current version.

• Other: 2 packages

2382 packages successfully loaded on the previous version too.

➖ Packages that were skipped altogether

3 packages were skipped only on the current version.

• Package could not be installed: 3 packages

918 packages were skipped on the previous version too.

[topolarity]

Looks like there's an issue with LHS type-asserts:

julia> function foo()
           v[end]::Int = 1
       end
ERROR: syntax: invalid syntax (end)

[mlechu]

LHS type asserts: in the special case (= (|::| x T) rhs), we call (remove-argument-side-effects x) which knew 'end has no side effects, but extracted '(end) into an ssavalue, so we got something silly like

((ref v #0=(ssavalue 0))
 (= #0# (end)))

Printing: I didn't catch it my first time looking through show.jl, but it looks like someone has already done the work to track whether we're in an indexing expression (though it might need some tweaking). If we decide not to go through with this PR, I can still print both a[end] and a[var"end"] as a[end] with a very small change, just one that would be incompatible with this PR.

[mlechu]

The bulk of the the pkgeval failures seem to come from macros in Setfield.jl and Accessors.jl. JuliaFormatter was also failing to precompile due to the LHS issue, but this should be fixed now (retrying it and depending packages).

@nanosoldier runtests(["JuliaFormatter", "GenieAuthorisation", "ConstraintsTranslator", "StippleTabs", "Sparlectra", "LanguageServer", "GenieFramework", "GraphsInterfaceChecker", "Hestia", "StippleCharts", "CompileBot", "GenieDeployDocker", "DefaultKeywordArguments", "QuantumControlBase", "LocalSearchSolvers", "ProToPortal", "CZML", "GenieAutoReload", "GenieSessionFileSession", "DocumenterCitations", "GeniePlugins", "GeniePackageManager", "ABCredit", "StippleMathjs", "MPIMagneticFields", "StippleTable", "Stipple", "ConsensusBasedXPlots", "Term", "StippleUI", "StippleMakie", "StippleDownloads", "StippleKeplerGL", "CBLS", "GridGraphs", "DocumenterInterLinks", "PkgDev", "SwagUI", "DocInventories", "Krotov", "DiscoDiff", "StippleMarkdown", "Meander", "HTMLBuilder", "Genie", "StippleTypedArrays", "DocumenterInventoryWritingBackport", "GraphIO", "VTKDataTypes", "ConsensusBasedX", "DocumentFormat", "GenieSession", "GenieDeploy", "BenchmarkTools", "SauterSchwabQuadrature", "Constraints", "ConstraintExplorer", "CompositionalNetworks", "TestTools", "Spehulak", "GenieCacheFileCache", "GenieAuthentication", "StippleLatex", "NearestCorrelationMatrix", "GenieCache", "GenieDeployHeroku", "JupyterFormatter", "DifferentiationInterface", "GenieDevTools", "AStarSearch", "StipplePlotly", "WebSession", "ObjectivePaths"])

Other "real" failures that I could find: TidierData, Unrolled, PEG, GraphPPL

[aplavin]

Just as another datapoint – I specifically asked about the parsed expr structure stability recently, in #53531. My resulting understanding was that parsing is expected to stay stable.
Of course, it may still happen if this is the only way to fix a major bug/inconsistency; just wanted to (hopefully...) confirm that expr parsing stability and breaking changes there are treated the same way as stability in other parts of Julia.

[Keno]

The AST is not technically part of the stable API surface as defined, but it's been pretty stable in practice. However, there have technically been minor breaking changes most releases. There'll likely be somewhat more disruption in 1.13 because of the contemplated switch to JuliaLowering. That said, there's of course a compatibility layer there. Additionally, we're looking at more gentle mechanisms for language evolution (including syntax) in general. As for this change, the general consensus is in favor, but there's a question of timing giving the reliance considerations. That's why my suggestion was to update the packages now to accept both versions. That way, this can either be folded into the general Julia 1.13 AST/lowering related changes, be a test case for whatever evolution mechanism we come up with, or if those both fail, we can probably merge this eventually in the 1.13 or 1.14 time frame once the ecosystem has adapted.

[aplavin]

However, there have technically been minor breaking changes most releases.

"Minor" breaking changes are common in Julia even aside from parsing, so seems consistent :)

Thanks for the context! So you recommend merging the linked Accessors.jl PR, as this parsing change is likely to appear anyways in some Julia version?

[Keno]

Thanks for the context! So you recommend merging the linked Accessors.jl PR, as this parsing change is likely to appear anyways in some Julia version?

Yes

[nanosoldier]

The package evaluation job you requested has completed - no new issues were detected.
The full report is available.

Report summary

✖ Packages that failed

18 packages failed on the previous version too.

✔ Packages that passed tests

42 packages passed tests on the previous version too.

~ Packages that at least loaded

12 packages successfully loaded on the previous version too.

➖ Packages that were skipped altogether

1 packages were skipped on the previous version too.

[mlechu]

Accessors and Setfield changes have been released (many thanks to @jw3126).

Expected to pass now: ~400-500 dependencies of the above

Expected to still fail: JuliaSyntax, PEG and ~10 dependent packages, TidierData, Unrolled, GraphPPL.

I know you're tired but @nanosoldier runtests()

[nanosoldier]

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

Report summary

❗ Packages that crashed

1 packages crashed only on the current version.

• GC corruption was detected: 1 packages

15 packages crashed on the previous version too.

✖ Packages that failed

62 packages failed only on the current version.

• Package has syntax issues: 27 packages
• Package fails to precompile: 4 packages
• Illegal method overwrites during precompilation: 1 packages
• Package has test failures: 7 packages
• Package tests unexpectedly errored: 2 packages
• Networking-related issues were detected: 2 packages
• Tests became inactive: 1 packages
• Test duration exceeded the time limit: 18 packages

1067 packages failed on the previous version too.

✔ Packages that passed tests

26 packages passed tests only on the current version.

• Other: 26 packages

5321 packages passed tests on the previous version too.

~ Packages that at least loaded

7 packages successfully loaded only on the current version.

• Other: 7 packages

2948 packages successfully loaded on the previous version too.

➖ Packages that were skipped altogether

899 packages were skipped on the previous version too.

[topolarity]

What's the status of the [compat]-related issues here @mlechu ?

[mlechu]

The following packages tried to use Roots v1.4.1 (latest is 2.2.6), which uses Setfield v0.8.2 (latest is 1.1.2):

EquationsOfStateOfSolids
Qsosed
ShiftedProximalOperators
StateSpaceRoutines
SmoothedSpectralAbscissa
MoistAir
HePPCAT
MarketRisk
QuasinormalModes
TwinCopulas
Symata
ProxTV
ProjectionPursuit
REoptLite

Of the available options (loosening the roots compat bound in all of these, loosening the setfield compat bound in old roots, or updating old setfield) I think updating setfield would be the most sensible.

If this sounds reasonable to you @jw3126, I've added the new parser compatibility to v0.8.2 of my fork of setfield here and I would just need you to make a backports-0.8 branch or similar so I can make a PR, so you can release it as 0.8.3

[jw3126]

@mlechu done: https://github.com/jw3126/Setfield.jl/tree/backports-0.8

[mlechu]

@nanosoldier runtests()

[nanosoldier]

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

Report summary

❗ Packages that crashed

1 packages crashed only on the current version.

• An internal error was encountered: 1 packages

4 packages crashed on the previous version too.

✖ Packages that failed

45 packages failed only on the current version.

• Networking-related issues were detected: 23 packages
• Tests became inactive: 1 packages
• Test duration exceeded the time limit: 21 packages

1438 packages failed on the previous version too.

✔ Packages that passed tests

11 packages passed tests only on the current version.

• Other: 11 packages

5258 packages passed tests on the previous version too.

~ Packages that at least loaded

8 packages successfully loaded only on the current version.

• Other: 8 packages

2841 packages successfully loaded on the previous version too.

➖ Packages that were skipped altogether

2 packages were skipped only on the current version.

• Package could not be installed: 2 packages

915 packages were skipped on the previous version too.

[nanosoldier]

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

Report summary

❗ Packages that crashed

5 packages crashed on the previous version too.

✖ Packages that failed

33 packages failed only on the current version.

• Illegal method overwrites during precompilation: 1 packages
• Networking-related issues were detected: 21 packages
• Test duration exceeded the time limit: 10 packages
• Test log exceeded the size limit: 1 packages

1360 packages failed on the previous version too.

✔ Packages that passed tests

12 packages passed tests only on the current version.

• Other: 12 packages

5313 packages passed tests on the previous version too.

~ Packages that at least loaded

6 packages successfully loaded only on the current version.

• Other: 6 packages

2879 packages successfully loaded on the previous version too.

➖ Packages that were skipped altogether

915 packages were skipped on the previous version too.