The Fanciest REPL History in the Land #59819

tecosaur · 2025-10-12T18:58:03Z

The Fanciest REPL History in the Land ✨

Do you dread typing out code for the second time? Are you a particular enjoyed of REPL history?

Well, I know I am, and for years I have yearned for something better than the current readline-style completion, better than OhMyREPL.jl's fzf-driven completion, better than any REPL history I've seen before!

It's not quite finished baking, but we're onto the final stretch 😀

repl_history_demo.webm

Thanks to @kdheepak, @jakobnissen, and @digital-carver for helping me design the UI and UX over on Zulip (#repl > Revamped REPL history).

Features

Zippy searching
- Event-driven asynchronous filtering UI
- Incremental, resumable searching with dynamic batch sizes
- Log-structured search checkpoints
Multi-selection
Faster histfile parsing (~2x)
Multiple search modes
A friendly help page
Syntax highlighting
Save multiple items to a file or your clipboard

TODO

Introduce annotation-preserving replace method
Thoroughly test the new replace method
Ask somebody more compiler-y about the performance pitfalls of the replace method (see: the REVIEW: ... code comments in annotated_io.jl)
Implement flashy REPL history
Restore up/down arrow history rotation in the REPL (collateral damage of over-zealous deleting)
Create a new test set for the new history
Make sure that everything that should be precompiled is

This PR is on top of #59778, because I think I can safely assume that will be merged first.

DilumAluthge · 2025-10-12T21:20:03Z

Anyone else getting "video playback aborted due to a network error"?

KristofferC · 2025-10-13T10:53:06Z

If possible, it would be nice to cut this up into some orthogonal pieces. For example, the AnnotatedString perf improvements could be a separate PR that (with benchmarks) could be merged and would make the diff here smaller and easier to review.

stdlib/REPL/src/History/search.jl

kdheepak · 2025-10-13T15:09:26Z

stdlib/REPL/src/History/search.jl

+        filename = try
+            readline(term.in_stream)
+        catch err
+            if err isa InterruptException
+                ""
+            else
+                rethrow()
+            end
+        end
+        isempty(filename) && (println(out, S"\e[F\e[2K{light,grey:{bold:history>} {red:×} History selection aborted}\n"); return)
+        write(filename, "# Julia REPL history excerpt\n\n", content)


Maybe it should default write to a location? The prompt could default to ~/.julia/history.jl (or a more appropriate location).

Ideally, the prompt for the filename would also allow scrolling up and down with the up arrow and down arrow, typing some characters of the file and hitting up arrow to narrow the search, have a ghost text of the last file name or the default file name so right arrow completes it etc. But I can imagine this being awkward the way things are set up in the REPL?

Ideally, the prompt for the filename would also allow scrolling up and down with the up arrow and down arrow, typing some characters of the file and hitting up arrow to narrow the search, have a ghost text of the last file name or the default file name so right arrow completes it etc. But I can imagine this being awkward the way things are set up in the REPL?

I've had exactly the same thought. I just can't be bothered to make it happen 😅

tecosaur · 2025-10-13T16:06:22Z

If possible, it would be nice to cut this up into some orthogonal pieces.

I'm happy to split this up, but there are really just three parts to it:

A refactor to _replace_final (needed to reuse some code for annotation-preserving replacement)
Introducing support for annotation-preserving replacement (needed for changing multi-line annotated strings into single-line with linebreak symbols)
Replacing the old history system with the fancy new one

I could split 1-2 off into a separate PR to review, if that sounds like a nice idea?

the AnnotatedString perf improvements

There's only suspicious performance here, not perf improvements, unfortunately 😞. I say "suspicious" because if I remove the keyword argument from replace skip reconstructing a named tuple with a range shifed the total time of a small replace op drops from ~4400ns to ~200ns. Given that an equivalent annotation-less replace takes ~65ns a time of ~200ns would be decent I think, but ~4400ns is strange — I can't figure it out though. If I can interest anyone else in investigating, that would be great.

For now, I've left these REVIEW code comments:

julia/base/strings/annotated_io.jl

Lines 202 to 226 in cc7ed22

    
           # REVIEW: For some reason, construction of `newannot` 
        
           # can be a significant contributor to the overall runtime 
        
           # of this function. For instance, executing: 
        
           # 
        
           #     replace(AnnotatedIOBuffer(), S"apple", 
        
           #             'e' => S"{red:x}", 'p' => S"{green:y}") 
        
           # 
        
           # results in 3 calls to `_insert_annotations!`. It takes 
        
           # ~570ns in total, compared to ~200ns if we push `annot` 
        
           # instead of `newannot`. Commenting out the `_insert_annotations!` 
        
           # line reduces the runtime to ~170ns, from which we can infer 
        
           # that constructing `newannot` is somehow responsible for 
        
           # a ~30ns -> ~400ns (~13x) increase in runtime!! 
        
           # This also comes with a marginal increase in allocations 
        
           # (compared to the commented out version) of 2 -> 14 (250b -> 720b). 
        
           # 
        
           # This seems quite strange, but I haven't dug into the generated 
        
           # LLVM or ASM code. If anybody reading this is interested in checking 
        
           # this out, that would be brilliant 🙏. 
        
           # 
        
           # What I have done is found that "direct tuple reconstruction" 
        
           # (as below) is several times faster than using `setindex`. 
        
           newannot = (region = start+offset:stop+offset, 
        
                       label = annot.label, 
        
                       value = annot.value)

julia/base/strings/annotated_io.jl

Lines 236 to 240 in cc7ed22

    
           # REVIEW: For some reason the `Core.kwcall` indirection seems to cause a 
        
           # substantial slowdown here. If we remove `; count` from the signature 
        
           # and run the sample code above in `_insert_annotations!`, the runtime 
        
           # drops from ~4400ns to ~580ns (~7x faster). I cannot guess why this is. 
        
           function replace(out::AnnotatedIOBuffer, str::AnnotatedString, pat_f::Pair...; count = typemax(Int))

tecosaur · 2025-10-13T18:50:24Z

I've just added back ~400 loc of the original code for using up/down arrow keys to go through recent history without ^R.

If you start a new REPL and immediately press up arrow, it does nothing the first time. Not quite sure why... (any ideas?)

I would like to add session-awareness. Maybe I'll try to drop that in this PR too?

tecosaur · 2025-10-14T13:43:20Z

Rebased to the latest REPL Syntax Highlighting HEAD.

tecosaur · 2025-10-14T13:45:35Z

Fixed some typos.

tecosaur · 2025-10-15T16:23:17Z

The precompilation situation is looking pretty decent, this is all I'm seeing with --trace-compile:

#=   19.4 ms =# precompile(Tuple{Type{Base.IOContext{IO_t} where IO_t<:IO}, Base.GenericIOBuffer{Memory{UInt8}}, Base.TTY})
#=   12.9 ms =# precompile(Tuple{typeof(Base.get), Base.Dict{Tuple{Symbol, Any}, Int64}, Tuple{Symbol, Symbol}, Int64})
#=   19.2 ms =# precompile(Tuple{typeof(Base.setindex!), Base.Dict{Tuple{Symbol, Any}, Int64}, Int64, Tuple{Symbol, Symbol}})
#=    2.5 ms =# precompile(Tuple{typeof(Base.print), Base.IOContext{Base.GenericIOBuffer{Memory{UInt8}}}, Base.AnnotatedString{String}})
#=    1.6 ms =# precompile(Tuple{typeof(Base._str_sizehint), UInt64})
#=    1.9 ms =# precompile(Tuple{typeof(Base.print), Base.GenericIOBuffer{Memory{UInt8}}, UInt64})
#=    2.5 ms =# precompile(Tuple{typeof(Base.AnnotatedDisplay.ansi_write), typeof(Base.write), Base.GenericIOBuffer{Memory{UInt8}}, Base.AnnotatedString{String}})
#=   14.6 ms =# precompile(Tuple{typeof(Base.getindex), Base.JuliaSyntax.GreenNode{Base.JuliaSyntax.SyntaxHead}, Int64})
#=    7.8 ms =# precompile(Tuple{typeof(Base.Terminals.cmove_up), Base.Terminals.TerminalBuffer})

tecosaur · 2025-10-15T16:28:36Z

Rebased now that #59778 has been merged.

tecosaur · 2025-10-15T16:50:25Z

If you start a new REPL and immediately press up arrow, it does nothing the first time. Not quite sure why... (any ideas?)

Fixed.

tecosaur · 2025-10-16T18:22:49Z

Thanks to Miguel, Camillo, and Sundar for taking this for a test-drive and providing feedback 🙂

Changes:

Clarified that ; uses and logic in quick help
Strip meaningless whitespace around filter segments
Changed "negative" → "negated" search term
Arrow keys (and others) no longer accidentally abort save
Prevent runaway redisplays
More minimal display updates
Make mmap work on Windows

tecosaur · 2025-10-17T18:21:10Z

Rebase (get the REPL precompilation improvements)

tecosaur · 2025-10-17T18:22:01Z

Fix oops with the save keymap
Pop gathered candidates to the active list, when appropriate

Extract the replacement loop body into `_replace_once` to ease future annotation tracking during string replacement operations. The new function returns match information (pattern index, match range, bytes written) that will be needed to properly adjust annotation positions when replacements occur.

tecosaur · 2025-10-18T05:34:12Z

Add vim up/down/pageup/pagedown keybindings
Add ^Y for copy to clipboard
Fixed a minor bug
Made tests pass
Create async history reading task during REPL init

Implement `replace` function for `AnnotatedString` that properly handles annotation regions during pattern replacement operations. The function tracks which bytes are replaced versus preserved, maintaining annotations only on original content and adding new annotations from replacement text. - Supports AnnotatedChar, AnnotatedString, and SubString replacements - Drops, shifts, and splits existing annotations appropriately - Refactored `_insert_annotations!` to work with annotation vectors directly - Adjacent replacements with identical annotations are merged into single regions - Lots of tests (thanks Claude!) Performance is strangely poor. For the test case mentioned in the REVIEW comment within `_insert_annotations!` we should be able to perform the replacement in ~200ns (compared to ~70ns for the equivalent unannotated case). However, for two reasons that are beyond me instead it takes ~4400ns. See the REVIEW comments for more details, help would be much appreciated.

tecosaur · 2025-10-18T17:07:01Z

Consolidate/organise Kristoffer's tweaks+fixes (rebase non-mac changes into the big new feature commit)
Fix annot replacement tests on 32-bit platforms

tecosaur · 2025-10-18T17:07:46Z

Make the colour of the history prompt match the prior mode, not the currently selected history item

Since the dawn of the Julia REPL, history completion has been limited to a readline-style interface. OhMyREPL improved the experience with fzf, but those who yearned for a delightful history completion experience (me) were left underwhelmed. With this overhaul, I now find myself spending more time looking through my history because it's just *so nice* to do so. The new history system is organised as a standalone module in stdlib/REPL/src/History with a clear separation of concerns: 1. History file management 2. Event-driven prompt/UI updating 3. Incremental filtering 4. UI display 5. Search coordination (prompt + display + filter) I've attempted to pull out all the (reasonable) stops to make history searching as fluid and snappy as possible. By memory mapping the history file in the initial read, and optimising the parser, we can read ~2 million history items per second. Result filtering is incremental and resumable, performed in dynamically sized batches to ensure responsiveness. Rapid user inputs are debouced. We store a log-structured record of previous search result, and compare search strictness to resume from prior partial results instead of filtering the history from scratch every time. Syncronisation between the interface and filtering is enabled via a Channel-based event loop. Enjoy! (I know I am)

Includes these changes: f5d171019 * Drop the search keymap, removed in Julia 1.13 (#???) 93a960c76 * origin/master Adapt delayed delete mechanism (JuliaLang#4392) 4184c9ba0 * Add ASCII requirement for package naming (JuliaLang#4404)

tecosaur · 2025-10-18T18:15:46Z

Better handle large numbers of gathered candidates

tecosaur · 2025-10-18T19:12:56Z

Unit tests are still needed, but I think this PR is well worth trying/reviewing now.

kdheepak · 2025-10-19T00:33:38Z

stdlib/REPL/src/History/display.jl

+    print(buf, "\eP=1s\e\\") # Start sync update
+    currentrow = 0
+    if newstate.query == FILTER_HELP_QUERY
+        print(buf, "\e[1G\e[J\n", FILTER_HELPSTRING)


When entering into help mode, currently the input prompt exists but is not displayed. I think it's fine to show the prompt as is:

Users will know to backspace to get out of help then.

Related: what are your thoughts on pulling these control sequences to inline functions:

function redisplay_all(io::IO, oldstate::SelectorState, newstate::SelectorState, pstate::REPL.LineEdit.PromptState; buf::IOContext{IOBuffer} = IOContext(IOBuffer(), io)) # Calculate dimensions oldrows = componentrows(oldstate) newrows = componentrows(newstate) # Redisplay components _begin_synchronized_update!(buf) currentrow = 0 if newstate.query == FILTER_HELP_QUERY _move_cursor_to_first_column!(buf) _erase_to_end_of_line!(buf) print(buf, "\n") print(buf, FILTER_HELPSTRING) ...

kdheepak reviewed Oct 13, 2025

View reviewed changes

stdlib/REPL/src/History/search.jl Outdated Show resolved Hide resolved

kdheepak reviewed Oct 13, 2025

View reviewed changes

tecosaur force-pushed the fancy-repl-history branch from cc7ed22 to 75fd56d Compare October 13, 2025 18:48

tecosaur force-pushed the fancy-repl-history branch from 75fd56d to 221b4e5 Compare October 14, 2025 13:42

tecosaur force-pushed the fancy-repl-history branch from 221b4e5 to ee23322 Compare October 14, 2025 13:45

tecosaur mentioned this pull request Oct 15, 2025

Make banner size depend on terminal size #51811

Open

KristofferC mentioned this pull request Oct 15, 2025

add support for automatic bracket completion in REPL input #59847

Merged

tecosaur force-pushed the fancy-repl-history branch from ee23322 to 3a6d8d7 Compare October 15, 2025 16:28

tecosaur force-pushed the fancy-repl-history branch from 3a6d8d7 to 4bac24f Compare October 15, 2025 16:50

tecosaur force-pushed the fancy-repl-history branch from 4bac24f to c7a66cb Compare October 16, 2025 18:22

tecosaur force-pushed the fancy-repl-history branch 2 times, most recently from 7126dd7 to 0da2a86 Compare October 17, 2025 18:20

tecosaur force-pushed the fancy-repl-history branch from 0da2a86 to 2fd2cea Compare October 17, 2025 18:21

tecosaur mentioned this pull request Oct 18, 2025

Drop the REPL search keymap JuliaLang/Pkg.jl#4425

Open

tecosaur force-pushed the fancy-repl-history branch from 97fd821 to 323f5df Compare October 18, 2025 05:20

KristofferC mentioned this pull request Oct 18, 2025

Try a blocking write before doing async write call #59795

Open

tecosaur force-pushed the fancy-repl-history branch from ff425b1 to fc8f7fb Compare October 18, 2025 17:05

tecosaur force-pushed the fancy-repl-history branch from fc8f7fb to 548f27d Compare October 18, 2025 17:07

tecosaur and others added 3 commits October 19, 2025 02:15

Bump Pkg for REPL changes

1c95717

Includes these changes: f5d171019 * Drop the search keymap, removed in Julia 1.13 (#???) 93a960c76 * origin/master Adapt delayed delete mechanism (JuliaLang#4392) 4184c9ba0 * Add ASCII requirement for package naming (JuliaLang#4404)

Fix Mac precompile hang

74ba1c6

tecosaur force-pushed the fancy-repl-history branch from 548f27d to 74ba1c6 Compare October 18, 2025 18:15

tecosaur marked this pull request as ready for review October 18, 2025 19:12

kdheepak reviewed Oct 19, 2025

View reviewed changes

Uh oh!

The Fanciest REPL History in the Land #59819

Are you sure you want to change the base?

The Fanciest REPL History in the Land #59819

Conversation

tecosaur commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!