Meta-issue for improving the REPL experience

[tecosaur]

I've ended up on a path to improve the REPL experience in a few ways, and thought I'd actually write up the overall plan.

• Wait for JuliaSyntax to be merged Enable JuliaSyntax.jl as the default Julia parser #46372
• Get StyledStrings capabilities into Base AnnotatedStrings, and a string styling stdlib #49586
  • Prerequisite: Terminfo parser #50797
  • Write docs, tests, ...
• Start applying StyledStrings across the Julia codebase
  • Integrate with Logging: Allow AnnotatedStrings in log messages #51802
  • Implement a syntax highlighter: Introduce new syntax highlighting stdlib #51810
  • Revamp the display of Markdown.MD Styled markdown, with a few tweaks #51928
• Free the REPL stdlib (see Upgradable stdlibs #50697)
• Revamp TerminalMenus Improving TerminalMenus #48381
• Util for highlighting Julia code: Introduce new syntax highlighting stdlib #51810
  • Can be applied to: Markdown code blocks, history selection, REPL itself
• Improved history selection (also from my Replete.jl project) #51874
  • Reimplement with some nicer abstractions, get vertical completion/selection for free along the way #51875
• Investigate in-REPL syntax highlighting
• Investigate nicer REPL completion, and see if it can be done based on the JuliaSyntax parser

Here's the vision for the REPL I currently have in mind:

GraphViz code

digraph {
    graph [bgcolor="transparent"];
    node  [shape="underline" penwidth="2" style="rounded,filled" fillcolor="#efefef" color="#c9c9c9" fontcolor="#000000" fontname="Alegreya Sans"];
    edge  [color="#aaaaaa" penwidth="1.2" fontname="Alegreya Sans"]
    rankdir = "LR"

    // Merged
    node [color="#a371f7"];
    "#46372 (JuliaSyntax)"
    "#50797 (Terminfo parser)"
    "#49583 (AnnotatedStrings + StyledStrings)"
    "StyledStrings#6 (Legacy support)"
    "StyledStrings#3 (Precompile fix)"
    "#51867 (move banner function)"
    "ColorTypes.jl#293 (convert to SimpleColor)"
    "#51806 (Improved annotation *-concat)"
    "#51810 (JuliaSyntaxHighlighting)"
    "#51807 (AnnotatedIOBuffer)"
    "StyledStrings#12 (printstyled specialisation)"
    "#51869 (load StyledStrings in REPL)"
    "#X Merged"
    // Pending
    node [color="#3fb950"];
    "#51802 (AnnotatedStrings in log messages)"
    "#51816 (Themable stacktraces)"
    "StyledStrings#36 (Themable stacktraces)"
    "#51829 (StyledStrings for Logging)"
    "#51914 (Improved join method for AnnotatedStrings)"
    "#51928 (StyledStrings for Markdown)"
    "#51811 (Resizing tweaked banner)"
    "#X Open"
    // Draft
    node [color="#6e7680"];
    "#51887 (StyledStrings for REPL Prompts)"
    "#X Draft"
    // Conceptual
    node [fillcolor="#fafafa" color="#dddddd" fontcolor="#444444"];
    "#? Conceptual"

    subgraph cluster_0 {
        style=rounded
        fillcolor="#ffffff"
        color="#222222"
        rankdir="LR"
        "#? Conceptual" -> "#X Draft" -> "#X Open" -> "#X Merged"
    }

    "#50797 (Terminfo parser)" -> "#49583 (AnnotatedStrings + StyledStrings)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51802 (AnnotatedStrings in log messages)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51806 (Improved annotation *-concat)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51807 (AnnotatedIOBuffer)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51810 (JuliaSyntaxHighlighting)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "ColorTypes.jl#293 (convert to SimpleColor)"
    "#46372 (JuliaSyntax)" -> "#51810 (JuliaSyntaxHighlighting)"
    "#46372 (JuliaSyntax)" -> "#??? (Prettier JuliaSyntax errors)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#??? (Prettier JuliaSyntax errors)"
    "#51810 (JuliaSyntaxHighlighting)" -> "#??? (Prettier JuliaSyntax errors)"
    "#51810 (JuliaSyntaxHighlighting)" -> "#51928 (StyledStrings for Markdown)"
    "#51807 (AnnotatedIOBuffer)" -> "#51914 (Improved join method for AnnotatedStrings)"
    "#51807 (AnnotatedIOBuffer)" -> "#51928 (StyledStrings for Markdown)"
    "#51807 (AnnotatedIOBuffer)" -> "StyledStrings#12 (printstyled specialisation)"
    "#51807 (AnnotatedIOBuffer)" -> "#??? (show/showerror with StyledStrings)"
    "#??? (string similarity matcher in Base)" -> "#??? (More helpful error messages)"
    "#??? (show/showerror with StyledStrings)" -> "#??? (More helpful error messages)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51869 (load StyledStrings in REPL)"
    "#51869 (load StyledStrings in REPL)" -> "#51811 (Resizing tweaked banner)"
    "#51867 (move banner function)" -> "#51811 (Resizing tweaked banner)"
    "#51811 (Resizing tweaked banner)" -> "#??? (Spiffed up banner)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51816 (Themable stacktraces)"
    "#51869 (load StyledStrings in REPL)" -> "#51816 (Themable stacktraces)"
    "StyledStrings#36 (Themable stacktraces)" -> "#51816 (Themable stacktraces)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51829 (StyledStrings for Logging)"
    "#51869 (load StyledStrings in REPL)" -> "#51887 (StyledStrings for REPL Prompts)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51887 (StyledStrings for REPL Prompts)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#??? (Deeper StyledStrings ↔ REPL integration)"
    "#51869 (load StyledStrings in REPL)" -> "#??? (Deeper StyledStrings ↔ REPL integration)"
    "#??? (Deeper StyledStrings ↔ REPL integration)" -> "#??? (REPL syntax highlighting)"
    "#51810 (JuliaSyntaxHighlighting)" -> "#??? (REPL syntax highlighting)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "StyledStrings#3 (Precompile fix)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "StyledStrings#6 (Legacy support)"
    "StyledStrings#6 (Legacy support)" -> "#51869 (load StyledStrings in REPL)"
    "StyledStrings#6 (Legacy support)" -> "#51829 (StyledStrings for Logging)"
    "StyledStrings#3 (Precompile fix)" -> "#51829 (StyledStrings for Logging)"
    "#51869 (load StyledStrings in REPL)" -> "#51829 (StyledStrings for Logging)"
    "#51869 (load StyledStrings in REPL)" -> "#51928 (StyledStrings for Markdown)"
    "#51869 (load StyledStrings in REPL)" -> "#??? (StyledStrings for Pkg)"
    "#??? (Basic terminal pager)" -> "#??? (Pageable help)"
    "#??? (Resolution of [@ref] links)" -> "#??? (Interactively navigable docs)"
    "#??? (Pageable help)" -> "#??? (Interactively navigable docs)"
    "#51928 (StyledStrings for Markdown)" -> "#??? (Pageable help)"
    "#??? (Deeper StyledStrings ↔ REPL integration)" -> "#??? (Basic terminal pager)"
    "#??? (Deeper StyledStrings ↔ REPL integration)" -> "#??? (REPL menu revamp)"
    "#??? (REPL menu revamp)" -> "#??? (Multi-modal menus)"
    "#??? (REPL menu revamp)" -> "#??? (Interactive history search)"
    "#51810 (JuliaSyntaxHighlighting)" -> "#??? (Interactive history search)"
}

[c42f]

I am so onboard with this plan! Not confident how much I can promise to work on the code directly, but I'm happy to support via JuliaSyntax adjustments and advice as/if necessary.

Because REPL is a stdlib, I think you can feel free to use the the JuliaSyntax API (directly, not only via Meta.parse()) and I hope that should make REPL syntax highlighting fairly easy once we've got Styled strings merged. A nice side effect of this should somehow be syntax highlighting of broken syntax within the REPL as people type. So people can fix that syntax as they type rather than trying to eval it first. OhMyREPL already has some of this via the JuliaSyntax parser, but it's still based on tokens rather than the parse tree so some of that info goes missing along the way.

Maybe we can add another item to the list:

• Reimplement REPL completion based on the JuliaSyntax parser.

Because last time I looked at the REPL.jl completion code it's ... a bit of a scary disaster area of regexes and heuristics IIUC

[c42f]

Another item to add to the list:

• Make the REPL an external (upgradable) stdlib. I suspect this would greatly improve our ability to make progress. See Upgradable stdlibs #50697

[tecosaur]

I'm wondering, would it be worth making a channel on Zulip/etc. for chatting about this? I think it could be worth making a space for discussion around this, and I'm not sure that just this issue suffices.

[caleb-allen]

@tecosaur I think that's a good idea. I was surprised that a #repl channel doesn't already exist in the Slack

[tecosaur]

Personally, I'm much more a fan of the Zulip, so if other people are fine with that (yes?), I'll put in a request for a #repl channel there.

[tecosaur]

I've now made a Zulip stream: https://julialang.zulipchat.com/#narrow/stream/404905-repl 🙂

[tecosaur]

FYI, there is now also a Slack channel (that currently basically says "go to Zulip"), and the Slack/Zulip channels should be listed as part of the new "REPL working group" soon (with me as the contact).

Meetings etc. TBD, but for now I thought it would be a nice start to better collect people interested in driving this aspect of the Julia experience forward 🙂.

[tecosaur]

I've added a plot to the end of the initial message that people subscribed to this issue may find of interest, I've also plopped it in a zulip thread.

[tecosaur]

From slack, Ruby's interactive REPL has had a fun update that we might want to look at in this effort: https://railsatscale.com/2023-12-19-irb-for-ruby-3-3/