Atypical "Erase In Display" behaviour

[pfitzseb]

Most terminals seem to implement ED (specifically \x1b[2J) by not actually deleting anything from the display buffer, but rather by scrolling. Conversely, xterm.js does delete the display buffer's contents, which is a) inconsistent and b) annoying because you lose output. I don't think this behaviour qualifies as a bug because the spec doesn't say about what should happen with the scrollback buffer, but would still be nice if this were changed.

Downstream issue in JunoLab/atom-julia-client#521.

Details

• Browser and browser version: Electron 2.0.8
• OS version: Manjaro
• xterm.js version: master (but also at least 3.6 and 3.7)

Steps to reproduce

Try

for i in {1..20}; do
  echo "$i"
done
echo '\e[1;1H\e[2J'

in xterm.js and another terminal emulator, and scroll up afterwards.

Expected behaviour is what e.g. Konsole does (middle window), and not how e.g. VSCode (left) or Juno's terminal (right, with xterm.js master) handles this.

[Tyriar]

Sounds good, Terminal.app and iterm2 also show this behavior.

[egmontkob]

VTE and Konsole also behave as suggested here, and have bugreports to revert to xterm's behavior of deleting those lines :D

The main problem is: if someone's intent is really to clear the entire scrollback (which the clear command supposedly does), it no longer works completely, leaves a screenful of content in the scrollback because clear emits the sequences in the wrong order – well, at least wrong order for these kinds of terminal emulators.

It's either a downside here or a downside there, you pick :)

[Tyriar]

@egmontkob I thought clear typically sent a different sequence to clear the scrollback as well?

[Tyriar]

Some good info on duplicate #1842

[seeruk]

Really keen to see this one resolved, it's the only thing holding me back from using terminal emulators like Hyper. I use Ctrl+L a lot to get rid of noise on my screen, but often want to be able to go back and see what was output. That includes things like testing. Personally, I don't see any reason not to work like iTerm2, VTE terminals, Terminal.app, etc. - there could be a separate way to clear the whole scrollback, or just push what's on screen up.

[egmontkob]

FYI: see also the latest comment in the aforelinked VTE thread – luckily the entire situation improved a lot by ncurses changing the order of the escape sequences it emits. If you decide to scroll out the lines, the clear -x command or bash's Ctrl+L clear command will not leave yet another screenful of data in the scrollback. (You see, I'm also getting totally confused in this story. :))

[seeruk]

There is some good discussion in that other issue; are there plans to change this behaviour so that the screen is pushed up instead of erased?

[pjlsergeant]

Reading the history here, it seems a decision is needed before any more code is needed, and that is: what's the behaviour that this project wants to support?

The behaviour of clear is probably a red-herring, as its behaviour appears to be different on different systems: this Linux example (by default) clears scrollback too, where the OS X version does not (and doesn't claim to either). The bugs mentioned in #1727 (comment) appear to describe an issue with adding empty lines to the scrollback.

@Tyriar Please could you make a decision either way? :-) The bug reports that keep getting duped suggest for many users the desired behaviour here is 2J shouldn't destroy any scrollback (and should neither add extra newlines to scrollback), but currently does, and that 3J should act the same way that 2J is currently handled?

[jerch]

Referring to the behavior of clear is indeed misleading because of the "E3" capability on some platforms.

Things get more clear, when looking at the sequences behind - ED2 and ED3:

• ED2 (CSI 2 J) - Erase all in activate display (cursor accessible region). There are two behavioral pattern - the "literal erase" one as seen in older devices and xterm doing a real erase (no scrollback additions), and a scroll-out one until the active region holds no data anymore. xterm.js implements the first one, which is more suitable in conjunction with ED3.
• ED3 (CSI 3 J) - Clear scrollback. Thats an xterm addition to the sequence realms. It is somewhat a surprising sequence, since it can directly manipulate scrollback, while kinda all other terminal content sequences are bound to the active cursor region. It is the "E3" capability denoted by some clear implementation. Support of the sequence varies across emulators, therefore clear tries to figured out by a terminfo lookup (thus TERM var dependent, quite often the source of issues here). xterm.js implements this feature.

@pjlsergeant Not sure what behavior you are looking for, but imho the pattern is obvious - we dont add things to scrollback on ED2 doing a real erase, and scrollback can be cleared with ED3. Ofc we could think about doing the scroll-out on ED2, but as mentioned by @egmontkob above this gets nasty in conjunction with ED3 support (I think we scrolled in the past and reverted back to erase at some point). Imho things work as intended - what do you think needs to be decided here?

[pjlsergeant]

@jerch If you think things work as intended, then presumably you are also suggesting this bug gets closed as WONTFIX?

EDIT: Currently xterm.js works differently from KDE's Konsole, iterm2, Terminal.app, and so on, and the fact that this bug is still open suggests a decision hasn't been made as to whether that's desired behaviour or not.

[jerch]

Just looked at the ncurses changelog - actually the bug @egmontkob described above got fixed in clear. Furthermore for older versions still doing it in the wrong order it can be fixed easily with an alias as clear && printf '\x1b[3J', or manually with printf '\x1b[2J\x1b[3J\x1b[H'.

So I am open for the discussion, whether ED2 should erase or move things into the scrollbuffer. Personally I think scrolling is more useful for apps/users. Well such a mode needs a bit thinking about the details, how to scroll - Scroll up to active cursor position, erase below? Or move all lines up, or only up to the first line without any data? Well that sort of things...

[pjlsergeant]

I think scrolling is more useful for apps/users

Yes, and also seems to be common behaviour.

Well such a mode needs a bit thinking about the details

Sure, although again, I think all mentions of clear are a bit of a goose-chase, as the specific issue here is the handling of 2J, regardless of whatever clear is printing on a given day/system/etc. The behaviour of the following one-liner is easier to reason about:

perl -e 'for ( 1 .. 100 ) { print "$_\n" }; print "\e[H\e[2J"'

I was half-thinking of attempting a patch for this to make it work like iTerm2 does, but it doesn't seem like that a decision has been made about that being the desired behaviour, yet!

[jerch]

Reading through the bugtracker comments in vte and konsole raises alot of questions regarding scrolling out, and make me think whether this is a good idea at all:

ED2 was never meant to do that, its purpose is to erase content on the active viewport in the sense of "clean that area", nothing more. In fact apps might use it do full screen refreshs to start over drawing. I use it myself that way in several fullscreen scripts. Now if such an app does not switch to alternate buffer, it will dump every screen refresh into the scrollbuffer. The semantics are broken further by the fact, that ED1 does not add anything to the scrollbuffer. But why? By the spec description ED0 + ED1 should be equivalent to ED2, but it isnt anymore for scroll-out terminals. So semantically adding changes to the scrollbuffer for ED was a mistake, no clue who broke with the spec initially. Furthermore with SU (scroll up, CSI <rows> S) we already have an appropriate sequence for it. No need to water other sequences.

Now regarding whether this is common or not. I see that all OSX terminals I know do that, even Terminal.app. Under POSIX terminal emulators in general it is not that common - xterm (the father of all xterm-alikes and thus most common graphical TEs) doesnt do it, neither do the kernel consoles (tested linux and FreeBSD). The latter are more on the legacy side of things, but considering the semantic issues above, I am not so sure anymore if coupling scrollbuffer changes to ED2 was a great idea.

TL;DR
After reading through the bugtracker and concerns listed there, I lean towards not implementing scrollout for ED2. It breaks horribly the sequence semantics and the basic unix principle "do one thing and do it right". If ppl want scroll out, use the appropriate sequence for it (SU). All imho and up for discussion.

[pjlsergeant]

For many people, xterm.js will be the only terminal they use that doesn't do it "properly". Could this behaviour be behind a flag / OS check?

[jerch]

@pjlsergeant Yes, I am fine with doing it behind a custom setting, so users/integrators can decide, if they want it or not (maybe something like scrollOnDisplayErase = true | false). About the inner mechanics I'd prolly go with full page dumps with trimming empty lines from the bottom (iTerm2 seems to do it that way). Idk if any emulator respects scroll region settings / origin mode here, needs some testing (my guess is that none limits the scrolling by those settings).

Feel free to do a PR, imho we can discuss the inner mechanics over there.

[sridhargopinath]

Hi, is there any progress on this issue? If not, Is there any workaround that I could use in the integrated terminal to not have the scroll back disappear? I continue to hit this issue on a daily basis where I lose output and just want to find a fix for it.