Cyrilyc font 6x8, and special character support for scrolling text

[bsveselov-rgb]

Good day, gentlemen! I'm trying to edit a 6x8 font file to add Cyrillic letters in scrolling text mode. I followed the instructions on the official website, but the project won't compile in VS Code, and I'm getting a ton of errors when selecting the board. I have zero programming experience. Could anyone help me with this? Thanks in advance.

[softhack007]

@bsveselov-rgb will your cyrilic letters replace the normal "roman" ASCII 32 thru 127 ?

currently WLED scrolling text does not support unicode, or any other extended character set. For example, not even the German "Umlaut" letters (äöüÄÖÜß) are supported. Same for other writing system like Greek, Arabic, Hebrew, Hindi, Korean, Chinese, Japanese, Cyrilic, Emoji, etc.

This was a design decision, to cope with limited Flash memory.

[bsveselov-rgb]

Yes. I need a version with only the Cyrillic font. After consulting DeepSeek and reading, I found a font with 6x8 Cyrillic characters. Is it possible to compile the firmware with just one font version, for example?

[softhack007]

you would need to replace the content of wled00/src/font/console_font_6x8.h

WLED/wled00/src/font/console_font_6x8.h

Lines 788 to 798 in aaad450

	* code=65, hex=0x41, ascii="A"
	*/
	0x38, /* 001110 */
	0x44, /* 010001 */
	0x44, /* 010001 */
	0x44, /* 010001 */
	0x7C, /* 011111 */
	0x44, /* 010001 */
	0x44, /* 010001 */
	0x00, /* 000000 */

by exchanging all "roman" letters like 'A' with a Cyrillic letter (bitmap encoded in bytes), one by one.

I don't know how a cycilic keyboard works - does it produce ASCII codes below 126?
It might still not lead to the expected result when you use the keyboard to type messages - maybe you'll have to "encode" each cyrilic character of the text manually, letter-by-letter.

[bsveselov-rgb]

Yes, I understood that and found the necessary code for the characters. Here's an example:

// А (0xC0)
    0x30, /* 001100 */
    0x48, /* 010010 */
    0x48, /* 010010 */
    0x78, /* 011110 */
    0x48, /* 010010 */
    0x48, /* 010010 */
    0x00, /* 000000 */
    0x00, /* 000000 */
    
    // Б (0xC1)
    0x78, /* 011110 */
    0x40, /* 010000 */
    0x70, /* 011100 */
    0x48, /* 010010 */
    0x48, /* 010010 */
    0x70, /* 011100 */
    0x00, /* 000000 */
    0x00, /* 000000 */
    
    // В (0xC2)
    0x70, /* 011100 */
    0x48, /* 010010 */
    0x70, /* 011100 */
    0x48, /* 010010 */
    0x48, /* 010010 */
    0x70, /* 011100 */
    0x00, /* 000000 */
    0x00, /* 000000 */
    
    // Г (0xC3)
    0x78, /* 011110 */
    0x40, /* 010000 */
    0x40, /* 010000 */
    0x40, /* 010000 */
    0x40, /* 010000 */
    0x40, /* 010000 */
    0x00, /* 000000 */
    0x00, /* 000000 */
    
    // Д (0xC4)
    0x30, /* 001100 */
    0x50, /* 010100 */
    0x50, /* 010100 */
    0x50, /* 010100 */
    0x50, /* 010100 */
    0x78, /* 011110 */
    0x88, /* 100010 */
    0x00, /* 000000 */
    
    // Е (0xC5)
    0x78, /* 011110 */
    0x40, /* 010000 */
    0x70, /* 011100 */
    0x40, /* 010000 */
    0x40, /* 010000 */
    0x78, /* 011110 */
    0x00, /* 000000 */
    0x00, /* 000000 */

The problem is, I can't compile the bin file for the firmware. Can I ask you for help by posting or sending the code for the Cyrillic characters?

[softhack007]

Can I ask you for help by posting or sending the code for the Cyrillic characters?

sorry, not myself - I'm too busy with other stuff atm :-(

But you can ask on discord, where we have a very active user community and lots of people who are ready to help others with their projects

Join the Discord server to discuss everything about WLED!

[softhack007]

Yes, I understood that and found the necessary code for the characters. Here's an example:

// А (0xC0)

PS: maybe you still did not do it right 🤔
as I said, you cannot "add" the cyrilic letters at their ASCII position (here: 0xC0) because WLED scrolling text only accepts letters up to ASCII 126 = 0x7E . It means that you must replace roman letters, for example exchange the bytes for "B" (0x42) with the bytes to draw "Б".

[softhack007]

Just for the curious - i've made a rough overview of what would be needed to support the full CP437, and later even add fonts for other codepages

• (experimental) full Codepage 437 support in scrolling text MoonModules/WLED-MM#281

There are of course a lot of "known unknowns" - for example I'm not sure if the esp32 framework already provides something to convert UTF-8 strings to CP437 strings. Finding Font files for other codepages might be a big challenge, and it is not clear if extra font files would ever be integrated into the main WLED codebase. Full codepage support would always be optional (custom build_flags), because localized fonts will increase firmware size considerably.

Btw, 8266 will probably never have this feature, due to 8266 firmware size limits. But I could image a solution for esp32 boards with at least 8MB flash.

[blazoncek]

@softhack007 I've already recommended that someone implements loading font files (binary or JSON or whatever) from file system. That would allow anyone to upload custom font without the need for custom compiling.
However, that feature might be only feasible on ESP32 (and variants) as ESP8266 lacks free RAM for font data.

I do not think implementing full Unicode support is feasible but a font file might allow this (at the cost of accessing flash during effect drawing).

[DedeHai]

I've already recommended that someone implements loading font files (binary or JSON or whatever) from file system. That would allow anyone to upload custom font without the need for custom compiling.

I just had the same thought and ran a quick test to see if it can be read directly from the file, well it can't. Opening a file and reading a few bytes then closing takes 25ms... BUT if the file is kept open, the access is very fast, 100us or so. littleFS allows for multiple files to be open, I checket up to 4 without issues, if this also works on ESP8266 then there is no need to load it to RAM.

edit: accessing 4 files in parallel also works on ESP8266, it is a bit slower though, reading 10 bytes per pass takes 270us, still more than fast enough so this appraoch would be feasible.

[softhack007]

@DedeHai so for the full picture, we'd need a string conversion from UTF-8 (or unicode?) to codepage xyz encoding, to have an index for the font tables ... any ideas?

edit: for boards with PSRAM, we could even load the font files into PSRAM for caching.

[DedeHai]

I already have it sketched out in my head, something like this:

• pack the chars as bits, padded to full bytes for easier access, so a 7x9 char would be 8 bytes (63bits plus one padding bit) instead of the currently use 9 bytes.
• to keep it simple: the first three bytes in the font file describe the font format, first byte is the bits-per-char, second is the start offset, third is the end of the table.

need of course some pythong script to generate these binary blobs from the current format, or if there are other commonly available pixel font formats, also support converting those.
ASCII tables are always 256 bytes in size, 128-255 are extended chars so here is for greek for example: https://www.ascii-codes.com/cp869.html

so a "greek only font" without fancy stuff would start somewhere around char 164 and end at char 238 but nothing sais it cannot be a full 0-255 ASCII table supporting all chars, including numbers.

the naming could be to replace the current 5 fonts as overrides, font1.fnt to font5.fnt. it would in theory also allow for fonts with much larger chars for HUB75, the limit would onyl be the access time to read the chars but it is quite fast, less than 10us per byte.

edit: actually the limit would be 256bits per char unless we make that header 4 bytes, which would be reasonable I guess.

edit2: now that I think about it, it really makes little sense to not always start a font at char 0, since it's stored in FS we can spare a few hundred extra bytes to keep the fonts simple and more universal for users. If a font does not contain the extended chars > 127, the file read at that position will just fail, so should be easy to catch.

edit3: forgot we need width and height of a char, so header would be 2 bytes for width and height. To support all languages, we'd need UTF-8 suport which I dont think is feasible with reasonable effort.

[softhack007]

@DedeHai so we would still need a way to convert unicode (UTF-8) - which is used by the WebUI when you name your segment in greek, or even you name it "schöne Grüße aus Grønland 👋 ", into the "codepage" that's used by your font file. Of course some unicode chars will not be supported by the font. However some basic unicode parser / translator would be necessary to pick the right character to display.

[softhack007]

To support all languages, we'd need UTF-8 suport which I dont think is feasible with reasonable effort.

A possible solution for the "codepage mapping" might be to translate UTF-8 into UTF-16 (aka UCS-2), which covers almost any letter supported by UTF-8 but always needs two bytes per letter - if we forget about the 4-byte variable codes for exotic stuff.
Codes < 127 are always ASCII codes, so no mapping needed for roman letters, numbers etc.

1. Font file: for all "characters" > 126, add two bytes which indicate the UTF-16 code for the letter or symbol (might be replaced with a generic UTF16-to-codepage_xyz translation list for all user fonts)

2. WLED translates the segment name from UTF-8 into UTF-16 (straight forward, no LUT involved)

3. if UTF16_index < 127 --> display font[UTF16_index]

4. if UTF16_index >= 127 -> scan the font until we find font[x].utf16 == UTF16_index --> display if found, show " " otherwise

The whole process could be accelerated if we can build a translation table in RAM.

Some magic python script could generate the font file, and generate UTF-16 indices when needed.