Support for downloadable "soft fonts" (DRCS)

[j4james]

Description of the new feature/enhancement

Starting with the VT220 terminal, it was possible for apps to define their own "soft fonts", also known as dynamically replaceable character sets (DRCS). You would download the font to the terminal with a DECDLD escape sequence, and assign it a character set ID that could then be designated via the usual SCS escape sequences.

Some example use cases:

Custom fonts

More examples at https://vt100.net/dec/vt320/fonts

Simple monochromatic images

CMatrix with Japanese characters

Source: https://github.com/jhamby/cmatrix

Game sprites

Proposed technical implementation details (optional)

The screenshots above were taken from a POC I've been working on for conhost. In the current implemenation, when you've designated a DRCS character set, those characters get mapped to values in the Unicode PUA area. Then when the renderer encounters values in that range, it uses a BitBlt to render the glyph in place of the usual PolyTextOut calls.

The quality isn't fantastic - I'm just using StretchBlt to resize the provided glyphs to match the current font size. And the performance can seem a bit sluggish on apps like CMatrix when writing a lot of content to the screen. Nevertheless, I think it works reasonably well, and I'm sure someone smarter than me will have suggestions for how it could be improved.

I'm still a long way from producing a PR, but I wanted to raise the issue now to see how much interest there was in the idea before spending too much time on it. Initially it would be conhost-only - I'm not sure about the conpty feasibility without #1173.

[DHowett]

This is extremely cool. I'd love to see it, and I'm sure @miniksa would agree. It does open up a minor can of worms though...

• If we have support for custom bitmap fonts, folks may revive their (sometimes quote angry) requests for supporting old FON files or whatever weird old bitmap format they wanted
• I'd love to keep renderer feature divergence between conhost and terminal to a minimum since eventually we're gonna use the DX renderer in conhost; does that make this significantly more difficult as to not be feasible?
• ConPTY feasibility, as you mentioned -- though I think that it might work if it's treated as a graphic rendition for a line/segment of a line and we have a way to pass off DECDLD¹

None of these concerns is significant enough to stop me from wanting it. Michael may feel differently?

¹ we may experience a 1-frame or (-1)-frame "tear" if the font changes and the sequence is passed through directly (using the quasi-broken conpty "pass through unknown sequences" feature, and without the renderer's consent/knowledge), right? but that's pretty minor.

[j4james]

• I'd love to keep renderer feature divergence between conhost and terminal to a minimum since eventually we're gonna use the DX renderer in conhost; does that make this significantly more difficult as to not be feasible?

I'd very much like that too. And in theory it shouldn't be a problem to add support in the DX renderer, although it might be a bit tricky to test without the conpty side of things. I could probably hack something though.

If you're also thinking about the double-width/height escape sequences, that's more of a political problem. I did initially implement a basic DX renderer for that, but it became clear that it would never work correctly with the way we're handling RTL in Windows Terminal, and I didn't want to start another argument on that subject.

• ConPTY feasibility, as you mentioned -- though I think that it might work if it's treated as a graphic rendition for a line/segment of a line and we have a way to pass off DECDLD¹

I wouldn't say it definitely can't be done, but it's more complicated than you might think, because fonts can be updated after they've been rendered. To correctly handle that behaviour over conpty would probably require tracking character set data associated with each character. Maybe not impossible, but it seems pointlessly complicated for something that should be trivial with a pass-through version of conpty.

[j4james]

It just occurred to me that it might be possible to get this working over conpty without that much extra effort if we limited our support to a single font buffer (which was standard for most of the earlier terminals). That way we probably only have to remember the last charset ID, and it should be safe to use that ID for forwarding all soft characters, regardless of the charset they were originally generated with.

I may be missing something, though. I've been through this several times before where I thought I had a plan that would work, then later realized there was a complication I had overlooked. Either way, I'd still prefer to leave this for a follow-up PR. Just getting it working in conhost is going to be complicated enough as it is.

[miniksa]

• ConPTY feasibility, as you mentioned -- though I think that it might work if it's treated as a graphic rendition for a line/segment of a line and we have a way to pass off DECDLD¹

Oh no. What if the conhost can handle all of these fonts and it just emits any of the complexity out the PTY side as sixels and we make Terminal understand sixels? :P

This is extremely cool. I'd love to see it, and I'm sure @miniksa would agree. It does open up a minor can of worms though...

None of these concerns is significant enough to stop me from wanting it. Michael may feel differently?

I do agree though that this is exciting and I'd love to see it. I'm not concerned about any of the issues up front to wholesale stop us from trying to do it.

The quality isn't fantastic - I'm just using StretchBlt to resize the provided glyphs to match the current font size. And the performance can seem a bit sluggish on apps like CMatrix when writing a lot of content to the screen. Nevertheless, I think it works reasonably well, and I'm sure someone smarter than me will have suggestions for how it could be improved.

If you're having trouble with the performance in GDI... I can look at it under the performance analyzer once there's a prototype and see if there's anything to be done. But generally speaking, BitBlt and StretchBlt ought to be hardware accelerated and pretty quick per https://docs.microsoft.com/en-us/windows-hardware/drivers/display/specifying-gdi-hardware-accelerated-rendering-operations. It might be faster to pre-stretch them all to the right size on another in-memory canvas and blt them over from there instead of making it do the stretch calculation every time.

I'd very much like that too. And in theory it shouldn't be a problem to add support in the DX renderer, although it might be a bit tricky to test without the conpty side of things. I could probably hack something though.

If you're also thinking about the double-width/height escape sequences, that's more of a political problem. I did initially implement a basic DX renderer for that, but it became clear that it would never work correctly with the way we're handling RTL in Windows Terminal, and I didn't want to start another argument on that subject.

And as for the DX side...I'm sure we can figure something out. I almost have the vaguest idea of writing all the glyphs into an in-memory font that exposes the interfaces and applying that... or a sprite map sort of thing with blits... hmmmm.

I'm not that worried about the RTL either. If we have to refine RTL further, then we do.

The screenshots above were taken from a POC I've been working on for conhost. In the current implemenation, when you've designated a DRCS character set, those characters get mapped to values in the Unicode PUA area.

I'm a bit worried about using the PUA and having someone have another legitimate use of it that gets in the way. But maybe it's OK to say you can't do both at the same time? I think the only way of avoiding it otherwise would be to attach a specific font override to pieces of the text buffer. Or some sort of flag. A bit ick, but possible.

• If we have support for custom bitmap fonts, folks may revive their (sometimes quote angry) requests for supporting old FON files or whatever weird old bitmap format they wanted

NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN. Not excited about that potential. I want FON files to go die. But it might make it easier to do if there's some sort of bltting thing already established for this.

[j4james]

It might be faster to pre-stretch them all to the right size on another in-memory canvas and blt them over from there instead of making it do the stretch calculation every time.

Yeah, I am doing that now, and I think it was an improvement, but it still feels sluggish. My current plan is to see if I can get it working with a run-time raster font, and see if that's any faster. That way I think I can render a whole line of characters with a single GDI call instead of having to do a separate BitBlit for each character - I'm hoping that might make a bit of a difference.

I'm a bit worried about using the PUA and having someone have another legitimate use of it that gets in the way.

This concerned me too. But if we only support a single font buffer, then it's only 96 characters, so I figured we should be able to find a safe gap somewhere that we can reserve. That said, I don't think adding a flag in the TextAttribute class would be such a bad idea either. It might actually make the renderer a little simpler too.

[miniksa]

It might be faster to pre-stretch them all to the right size on another in-memory canvas and blt them over from there instead of making it do the stretch calculation every time.

Yeah, I am doing that now, and I think it was an improvement, but it still feels sluggish. My current plan is to see if I can get it working with a run-time raster font, and see if that's any faster. That way I think I can render a whole line of characters with a single GDI call instead of having to do a separate BitBlit for each character - I'm hoping that might make a bit of a difference.

Yeah that's sort of the idea I was going for with what I was describing with DirectX... maybe making a fake in-memory font for it and letting those optimizations happen in the text drawing code.

The problem with BitBlt is honestly probably that you're doing a syscall for every single operation. I don't think there's a way around that beyond the font trickery or... composing your own complete bitmap in a chunk of memory in usermode somehow and transferring that whole thing to GDI for rendering at the end. That's really why I was using PolyTextOut in the GDI renderer... calling TextOut in a loop was DEATH in just the syscalls when I did a perf trace on it.

This is where things like DirectWrite will have the blting advantage by composing up a ton of instructions in a queue in user mode and then dispatching them in a chunk. Also it'll run some of it on the GPU probably versus GDI doing virtually all of the work on the CPU. Though I still bet if you can fit it into a font-shaped thing, it'd be faster for both DX and GDI. Or there is technically a DX/DWrite/GDI compatibility shim thing to allow interoperability.... but that may be too crazy.