DECSDM is a poorly-understood hellscape

[j4james]

I haven't tested with a specific version of notcurses, but I've seen in the code that you set private mode 80 when you initialize sixel:

notcurses/src/lib/sixel.c

Line 871 in 3dbb3c1

return tty_emit("\e[?80;8452h", fd);

There is some confusion as to whether DECSDM is intended to enable sixel scrolling or disable it, and around half the terminals I've tested have implemented it one way, and half the other. On terminals that interpret mode 80 as a request to to disable sixel scrolling, your sixel images are going to be rendered in the wrong place.

For the record, I think you are using it is the wrong way, but considering how terminals are currently split on this issue, there really isn't a right way. The best course of action is to leave that mode alone. Every terminal I've tested does at least have sixel scrolling enabled by default, so unless you actually need to disable scrolling, there should be no reason to change the mode.

[j4james]

I forgot to add, these are some terminals that I've tested that disable scrolling when mode 80 is set:

• MLterm (3.8.4)
• RLogin (2.25.3)
• Reflection Desktop (16.2.0)
• Rxvt (the rxvt-unicode-sixel fork)

[dankamongmen]

thanks for the detailed report, @j4james ! i'll look into this a bit more this evening after work.

[dankamongmen]

cref #1533

[dankamongmen]

MLterm 3.9.0 (current in at least Debian Unstable) does properly place the sixel when DECSDM is used, in my experiments. any reason you're using 3.8.4?

[j4james]

This is the test I just ran. With mode 80 set, the line is rendered in the top left corner of the screen.

printf "\e[?80h\ePq\x21100~\e\\"

With mode 80 reset, it renders at the current cursor position.

printf "\e[?80l\ePq\x21100~\e\\"

Are you seeing something different in version 3.9.0? The reason I'm using 3.8.4 is because that is the latest version available on my distro, but I wouldn't have thought the version would make any difference. The latest source certainly looks like mode 80 still disables scrolling. See here:

https://github.com/arakiken/mlterm/blob/f863ca55b9d92a0b7a36746a7d7893fdf44234a5/vtemu/vt_parser.c#L3897-L3900

[dankamongmen]

i confirm the results you reported, and furthermore that 8452 doesn't affect them. hrmmm.

at the same time, i know we're emitting DECSDM, and yet ncplayer -k and notcurses-demo are printing bitmaps at the right places. hrmm.

i will look into this. tally-ho!

[dankamongmen]

hrmmm!

if i do printf "\e[?80h\ePq\x21100~\e\\", the behavior is persistent on ncplayer and notcurses-demo.

which suggests to me that maybe i'm not actually emitting the escape?

[dankamongmen]

strace shows 8452 only once:

write(1, "\33[39;49m", 8)               = 8
write(1, "\33[m\33(B", 6)               = 6
write(3, "\33[?8452l", 8)               = 8
write(3, "\33[?25h", 6)      

we would expect to see "\e[?80;8452h". i don't think we're emitting the escape!

[dankamongmen]

the recent revamp of terminal detection using queries omitted the call to pixel_init() when set, so yeah, i was not emitting this escape. so my knowledge of MLterm was incorrect, and i am in your debt, @j4james , for bringing this to my attention.

furthermore, i had thought MLterm was just kinda broken regarding sixel, lol, which i suppose it is -- but not in the way i was thinking (rather, that it interprets DECSDM differently from all other terminals i've tested with).

[dankamongmen]

so! the upshot of this seems to be that we indeed do not want to set DECSDM, assuming we begin executing with cursor-based-positioning enabled. unfortunately, we don't know that to be the case. but we don't want to just emit it unconditionally, because some terminals flip their interpretation. but in that case, i still want to force cursor-based-positioning, i just want to enable it a different way.

i do unambiguously identify MLterm via a query, so i can work around this on MLterm by inverting the call. it doesn't help for those other three terminals, but i've also never heard of two of them, and the other is a fork, so ... i'm inclined honestly to send PRs off to MLterm and this rxvt fork and try to bring them into conformance with everyone else. i dislike the idea of not forcing the mode. that's not our problem. our problem is yet another breakdown in interpretation among terminal authors.

luckily, this is a problem adderall and, like, gregariousness can hopefully fix.

[dankamongmen]

confirmed the MLterm reverse behavior you claim in 3.9.0.

[j4james]

i'm inclined honestly to send PRs off to MLterm and this rxvt fork and try to bring them into conformance with everyone else

Well you can try that approach, but I'm pretty certain that it's XTerm, Alacritty, et al that are wrong in this case, and MLTerm that is correct.

I suspect the misunderstanding came about because the VT330/340 manual documented DECSDM the way XTerm has currently implemented it, but I believe that was a mistake. The various VT382 manuals have it the other way around. See here, and here.

The author of Tera Term, who has a collection of DEC terminals, actually tested this on a VT330 and VT382 and reported the results here: https://twitter.com/ttdoda/status/479053314412126208

According to Google translate:

The actual machine and manual of VT382 and the actual machine of VT330 are DECSDM reset and sixel scrolling mode is enabled.

Which I understand to mean that sixel scrolling mode is enabled when DECSDM is reset.

[dankamongmen]

arakiken/mlterm#23

[dankamongmen]

Well you can try that approach, but I'm pretty certain that it's XTerm, Alacritty, et al that are wrong in this case, and MLTerm that is correct.

i completely agree with your analysis here. i think we have come to a prescriptivist vs descriptivist problem =].

[j4james]

I should also mention that the author of lsix has recently purchased a VT340 terminal for the sole purpose of verifying how this particular mode is meant to work. It should be arriving in two weeks, so I think it's worth at least waiting for a definitive answer from that before potentially pushing terminals to implement this incorrectly.

[dankamongmen]

I should also mention that the author of lsix has recently purchased a VT340 terminal for the sole purpose of verifying how this particular mode is meant to work. It should be arriving in two weeks, so I think it's worth at least waiting for a definitive answer from that before potentially pushing terminals to implement this incorrectly.

i honestly disagree with you there, though i can understand--from the depths of my engineer heart, i promise you i understand--your reasoning.

nonetheless, the purchase of this VT340 means 1 more person who might use my software is using it on a vt340. if the last owner was also using it, net zero change. in the meantime, many people every day are picking up terminals which are implementing this plausibly-incorrect behavior. if the VT340 puts on a fruit hat and dances ala Carmen Miranda in response to DECSDM, i still would want all the terminals i drive to do the same thing. as the number of terminals implementing it in the (same!) plausibly-incorrect fashion is very great, and i do not wish to try and change e.g. Thomas Dickey's mind regarding xterm behavior. surely you can understand.

furthermore, if everyone was getting it wrong the same way, i'd suspect future authors to make the same error, no? especially as they're probably ripping off XTerm code, and ... see previous para.

my objective is not VT340 conformance. it is putting my graphics where i want them.

i hope this did not come off as bitchy or diminishing; i respect your stance here, and your depth of knowledge.

[dankamongmen]

i'm glad to see you've commented on the mlterm bug; i referenced your arguments here (which are all very cogent and informative), but i'm sure you can make your case better in person.

[j4james]

I completely understand where you are coming from, but you also need to realise that there are two different groups in the terminal community, and they don't always share the same priorities.

There are those that just treat the terminal as a frontend for the *nix command line, and don't particularly care about VT standards and how the original terminals behaved. They are quite happy to go along with whatever "everyone else" is doing, regardless of whether it is correct.

But there are also some (much rarer I'll admit) that are actually used as real terminal emulators, and that need to be able to connect to legacy systems in commercial environments and accurately emulate the original devices they were designed to work with. These terminal emulators are never going to deliberately introduce bugs just to match a trend in the open source community.

The problem for you, is that XTerm is more likely to fall in the latter camp, and if Thomas is presented with evidence that his current implementation is incorrect, there is a reasonably good chance he will fix it (this I know from past experience). And if everyone else is just doing exactly what XTerm does, then they'll all follow along like lemmings.

Which again is why I'm suggesting you wait a couple weeks to see how that turns out.

[dankamongmen]

@j4james yep, i think we're in violent empathy and dedicated to incompatible goals. if mlterm rejects my patch, i will work around its behavior. ideally, i can then identify a true vt340, and handle it the exact same way.

i have enjoyed this discussion. thank you for being rational.

[dankamongmen]

@j4james yep, i think we're in violent empathy and dedicated to incompatible goals. if mlterm rejects my patch, i will work around its behavior. ideally, i can then identify a true vt340, and handle it the exact same way.

i have enjoyed this discussion. thank you for being rational.

in fact, this sounds like the thing to do regardless of what mlterm does. the world of emulators interpreting it this way is (a) irreducible to 0 due to hardware and (b) much smaller. so keep the default following the common reality, and work around it for the known offenders. if the vt340 is uniquely identifiable via query, that pretty much solves things for everyone.

[dankamongmen]

@j4james , i've explicitly voiced my support for your recommendation over on the MLterm PR.

with that done: given your initial report, how do you feel now about my plan to always set the mode, and invert it for terminals known to handle it in reverse? ideally this would be expressed as a terminfo extension capability, but this is not the best of all possible worlds, so who knows.

[dankamongmen]

on another topic: are you in contact with the author of lsix, then? or just reading their plans? i'd like to talk to them about using notcurses in lsix (see ncls as included with notcurses).

[dankamongmen]

i'd also be interested in what you're working on that sees you testing such a broad set of terminals!

[dankamongmen]

well, for better or worse, the mlterm PR got merged. for those following along at home,

• @j4james was correct -- everyone has inverted the sense of this report
• @arakiken claims that Grandmaster Dickey has been presented with this information, with no XTerm change resulting
  • admittedly, the evidence for this seemed to be several tweets in japanese

but i think the doom-and-gloom is unwarranted, @j4james. now that i'm doing query-based detection, i ought be capable of detecting a true VT340. alternatively, i could only write the inverted form for those i identify as supporting the inverted form. ideally we'd get a terminfo extension for this. in any case, i now have the fun task of distinguishing between two MLterm behaviors, though that's still better than two behaviors for every other terminal i test with (which would have been a problem even if everyone changed to one interpretation). le sigh.

i now can write both forms, and it's just a matter of classification. closing this bug up. sorry things didn't work out the way you wished, @j4james ; i don't at all feel that i "won" either =[.

[j4james]

That's rather depressing. I could understand this from everyone else, but for XTerm this choice is just bizarre. You've got to specifically request VT340 emulation to even get sixel support in XTerm, but then he's deliberately made it not match the VT340 in this regard. Admittedly XTerm's sixel emulation is too broken to be of much use as a VT340 emulator anyway, but that could at least have been improved over time.

Anyway, thanks for letting me know.

[dankamongmen]

That's rather depressing. I could understand this from everyone else, but for XTerm this choice is just bizarre. You've got to specifically request VT340 emulation to even get sixel support in XTerm, but then he's deliberately made it not match the VT340 in this regard. Admittedly XTerm's sixel emulation is too broken to be of much use as a VT340 emulator anyway, but that could at least have been improved over time.
Anyway, thanks for letting me know.

like i said, i'm not sure the claim seen in that PR about getting it to Dickey was the last word in xterm. if you want to present the information to Dickey in an authoritative and clear manner, along the channels he wants reports on, and see if a different result can be had, i'll cheerfully change all this back.

[dankamongmen]

with the proof acquired by @hackerb9 of the lsix project that @j4james is/was correct, we're beginning to see some terminals shift to the authentic vt340 behavior. i suppose this is a good thing overall. i will be maintaining mappings of termXversion to DECSDM interpretation in that most accursed of functions, apply_term_heuristics(). @j4james , continue your quest, and you can rely on me to pick up the pieces.

[dankamongmen]

and i guess it falls upon me to go prepare a PR to revert the change in MLterm, since i requested it, huh? doing otherwise would be an act without honor, sigh.

[mintty]

Just to notify, mintty/mintty#1127 (comment) mentions version checks; you can already add them for mintty which will follow the change from 3.5.2.

[dankamongmen]

Just to notify, mintty/mintty#1127 (comment) mentions version checks; you can already add them for mintty which will follow the change from 3.5.2.

implemented, thanks!

[mintty]

Thanks, however, I did not quickly find how ti->termversion is determined in notcurses, but the mintty response to the Secondary Device Attributes request CSI > c is CSI > 77;30502;0c, not a string with "3.5.2".

[dankamongmen]

Thanks, however, I did not quickly find how ti->termversion is determined in notcurses, but the mintty response to the Secondary Device Attributes request CSI > c is CSI > 77;30502;0c, not a string with "3.5.2".

but it is responding to XTVERSION with "mintty VERSION", and that's what i'm keying off of.

[dankamongmen]

[mintty]

Ah, I had forgotten about that sequence and that I implemented it :) Thanks for the reminder.

[dankamongmen]

no problem. hey, while i have you here....i assume you're fairly knowledgeable regarding the whole MSYS2 system, since mintty is the default terminal there? i'm having a problem where my windows binaries work fine in Windows Terrminal, but in mintty on windows they can't enter cbreak mode, and thus i can't get character-at-a-time input. it almost seems like there's a line discipline above the i/o handles i receive. from what i've been able to find online, it seems that perhaps mintty in this context is not running directly attached to a console, but somehow connected via pipe to some intermediary? i'm unsure. what it comes down to is: what incantation is necessary to eliminate line buffering of input on a msys2 mintty? @mintty thanks for any help you can provide! #2116 has full background info.

[j4james]

@dankamongmen In case you weren't aware, both Alacritty and Xtermjs have committed fixes for this, and Contour is planning to fix it in an upcoming bugfix release. I don't think there are any terminals left that still have it wrong (discounting those with planned fixes that haven't yet been released).

Edit: I see now you've already raised issue #2204 for this.

[mintty]

The version compare function fails on multi-digit version numbers, e.g.
compare_versions ("3.5.2", "3.5.3") is -1
compare_versions ("3.5.2", "13.5.2") is 1

[dankamongmen]

Sacre Bleu! I'll get that fixed up when I'm in front of my computer, dumb, thanks @mintty

[dankamongmen]

fix is in for the issue @mintty identified; thanks a lot!