[core] IME Support

[mia-bentzen]

In order to be able to type text in japanese (and i assume chinese too, though i'm not familiar with it) you pretty much have to use to OS's IME (Input MEthod) to get input working right.

To give a small example of how it works: when you type が you don't actually type が directly, rather you type ga and it then automatically replaces that with が. It also comes up with suggestions based on a dictionary, and pressing space changes kana (basically letters) into kanji (basically complicated letters) so for instance you could have おんな, press space, and then it would change it into 女.

In order for the OS to know where and when to draw the IME suggestions box, the application has to tell it. The way SDL2 accomplishes this is with three methods: SDL_StartTextInput, SDL_StopTextInput, and SDL_SetTextInputRect.

The reason this would be nice to have is that without it japanese users simply can't type japanese text in any raylib application, and i'm not aware of any alternative or workaround.

[orcmid]

That's very interesting. I was thinking that we need a way to do that and there are also considerations with respect to accessibility. This is also valuable for additional languages/keyboards.

The TLDR: This software is extensive and maybe larger than raylib :). Figuring out a mapping and then being able to deal with SDL2 updates is going to be challenging. I recommend that this issue be moved to Discussion since adapting SDL for inter-working with raylib is going to require extensive effort and substantial analysis and testing.

=== Observations

The SDL2 Tutorial is tied to mingw-w64 and it involves libraries as well as a DLL that applications are expected to bundle. It is useful that the code downloads are signed. There is no public development server (GitHub or elsewhere).

It may take some work to adapt that to the raylib way of doing things, whether as an external or extra. I assume that raylib should be moved to handle Unicode (probably UTF-8) in these cases.

I gave the source-code download a quick look. The code is under a BSD-flavor license. Considering that this is about a Windows-centric facility (I thought) the plethora of Unix-isms is startling (*.sh, *.m4, etc.) in areas involving use of Visual C.. Apparently there are comparable integrations for other platforms. There are Visual Studio solutions and projects as part of the code. Some may be dated (one is for VS2012). The dates on files in the Zip appear to be from late July although it is not clear what that means. There is no .git and change-control history.

[mia-bentzen]

First off, I might be misunderstanding what you're saying, so absolutely correct me if i'm wrong!

You seem to be suggesting that we use SDL2 to solve this problem, while that might be possible my intention was more just to show what the API might look like (i'm not really experienced enough with this kind of thing to be much help with the low level details anyway).

Another thing is that IMEs are not a windows thing, they're on mac and linux as well. Otherwise you wouldn't be able to type in japanese on either of those.

As for whether or not it should be moved to discussion i don't really know how any of that works (this is the first issue i've ever opened). So if i need to do something then i might need instructions.

[orcmid]

Thank you @mia-bentzen, I understand your concern more now. が　I managed to use IME right here, after using Windows 10 to install Japanese as an additional language. I did not know how to do the rest of your example, of course, although bringing up special soft keyboards might help..

I am uncertain that my browser has anything to do with this with regard to enabling IME. I am ignorant of how the interaction is handled.

Two things:

1. The integration model for enabling IME (if that is needed), or allowing it to be shown, needs to be understood better. I hope you can help us understand that.
2. To change this issue to a discussion, there should be an option on the right sidebar here to do that. I don't see it, probably because I am not the author of the issue. Ideally, it will take the entire thread along with it.

[mia-bentzen]

So i suppose there's two parts to understanding how the IME works. There's the way the user interacts with it, which i explained a bit, and the more technical details (i can only really give educated guesses here)

For completeness i'll try to give as complete of an explanation of both as i can.

So first the user has to select a text box of some kind, which then prompts the IME to become enabled. You can then activate it by switching to hiragana or katakana. Once that happens if you start typing it will start giving auto complete suggestions (this is definitely handled by the OS, as it's very consistent in it's behavior and you can customize the dictionary in the language settings somewhere).

The user can type the romaji (english letters representing the japanese character) of any kana. So for instance my earlier example of ga being turned into が. At this point the kana that was typed is in a sort of limbo state where it gets displayed by the application, but it hasn't been sent as input as the OS is in full control (or at least mostly) of what is happening to it.

Once you hit enter you commit those characters and it gets sent to the application as text that has been inputted, and the application is free to handle it however it wants.

this process can be viewed very intuitively in sublime text because for some reaseon it actually shows the 'limbo' text in a separate box.

Another thing that you can do is press space (only during the limbo state) to 'collapse' the kana into kanji. This is very important as kanji is extremely common in japanese.

Based on all of this, i think that during the limbo state the application only gets told what the current 'limbo' text is, and then once you commit all that gets sent as some kind of text input event (i'm not familiar with how this is handled in the low level OS APIs, so it might either be multiple events for each character or one event with the entire string). And the application needs to tell the OS when a textbox (could be anything really, but the point is that you type text there and it has a location) is selected that the IME needs to start and where it needs to start. (To clarify, when i say start here i don't mean opening the suggestions, that is done by the user typing. Rather i mean that we tell the OS that IME input is now possible).

Apologies for the wall of text, i figured it was better to be too detailed than not detailed enough however. If anything was unclear just tell me and i'll try and reexplain that part better!

(Also, i don't seem to have the option to change the issue to a discussion either, i don't know if i'm looking at the wrong place or doing something wrong or something else)

EDIT: I forgot to mention, i think you can also customize it to some extent (at least on windows). I however have no idea how it works or how extensive it is. Also personally i'd say it would be fine to enforce the default appearance, most applications don't seem to make use of it anyway.

[orcmid]

Aside: When you view this issue, the right sidebar should have something like this at the end of the material there:

As the originator of the issue, I think it should appear for you. I am uncertain whether it brings all of your nice comments along. I would hope so.

[mia-bentzen]

For whatever reason none of those buttons are there for me.

[raysan5]

@mia-bentzen Thanks for the detailed report, this issue could require some investigation.

@orcmid I think only the owner of the repo can convert an Issue to a Discussion. For me it's ok to keep it as an issue.

[orcmid]

I have been sleuthing in SDL. Here's what I have so far:

Aside: To conduct this analysis I had to install grepWin. Any good grep can be used to search an entire project. It is a quick way to narrow in. Sometimes there are too many hits though.

The functions SDL_StartTextInput, SDL_StopTextInput, and SDL_SetTextInputRec and others have their signatures specified in the file SDL_keyboard.h around line 231 and down through to the end.

The functions SDL_StartTextInput, SDL_StopTextInput, SDL_SetTextInputRec, and soft-keyboard displaying are implemented in SDL_video.c at lines 4034-4106.

These are wrappers that rely on a global variable, _this, defined at SDL_video.c line 128.

	static SDL_VideoDevice *_this = NULL;

The struct SDL_VideoDevice pointed to is a structure of pointers to functions defined in SDL_sysvideo.h. An SDL_VideoDevice is a very large vTable of sorts, a kind of vector into functions implemented elsewhere. The functions pointed to in an initialized SDL_VideoDevice struct must all take a _THIS parameter, where

		 #define _THIS  SDL_VideoDevice *_this	

is the parameter type used in the prototypes of various functions. This is demonstrated in the code for SDL_StartTextInput which uses the
structure member void (*StartTextInput)(_THIS_) to access a platform-specific implementation:

	_this->StartTextInput(_this);

The point is that the functions pointed to in the SDL_VideoDevice are initialized with platform-specific solutions, depending on what platform (e.g., Linux, Windows, MacOS) SDL has been compiled for. The use of providing the vTable as a parameter makes all of the specific SDL_VideoDevice struct available to each of the functions linked from it and the table could even be altered by them. This is a kind of roll-your own virtual class (C++) expressed in C language. I will have more to say about that.

There are 7 relevant text-input and possible-screen-keyboard functions that have pointers stashed in an SDL_VideoDevice. The slots have the following names and prototypes for the functions pointed to.

 /* Text input */

void (*StartTextInput) (_THIS);

void (*StopTextInput) (_THIS);

void** (*SetTextInputRect) (_THIS, SDL_Rect *rect);


/* Screen keyboard */

SDL_bool (*HasScreenKeyboardSupport) (_THIS);

void (*ShowScreenKeyboard) (_THIS, SDL_Window *window);

void (\*HideScreenKeyboard) (_THIS, SDL_Window *window);

SDL_bool (*IsScreenKeyboardShown) (_THIS, SDL_Window *window);

There are checks on whether a pointer is present, as in

if (! _this->HasScreenKeyboard ) ... ;

And if it is available, the function is used via the _this pointer as in

if (! _this->HasScreenKeyboard )
		 return SDL_false; 
	else return this->HasScreenKeyboard(_this);

We have seen how the keyboard functions are implemented by short procedures that call for platform-specific implementations via the global _this pointer to an initialized struct SDL_VideoDevice.

Now we need to understand where such initializations are made and what are the functions pointed to after initialization.

---

Note 1: This is a bit like hand-rolled COM interfacing. It is weirder than that in the extreme use of indirection, even though _this is likely held in registers by compiler optimization. There is reference counting in SDL2, and in-process COM would provide a general solution for that as well. One would not encapsulate all of this behind a single COM interface such as the mammoth SDL_VideoDevice struct. I speculate that better functional encapsulation is achievable. Likewise if this was done with C++ abstract classes.

Note 2: raylib also relies on a global data structure, CORE, which means essentially that an application can have only one (main) window. At least CORE is directly reached rather than via a global pointer. Current reliance on a DLL (or other shared library) for SDL2 appears to be a problem if the DLL ends up being shared between two running programs at the same time. It can work. It seems unnecessarily fragile. There is a different philosophy with raylib, where the application and raylib modules are compiled together and all loaded into the created executable (e.g., an .exe)

[jmorel33]

Fantastic dive and information https://github.com/orcmid not everyone would be willing to dig into. Thanks for getting your hands dirty on this.

[orcmid]

Continuing ...

One place where initialization of a SDL_VideoDevice struct occurs is in the function

	static SDL_VideoDevice *
		WIN_CreateDevice(int devindex)

start ing at line 97 of SDL_windowsvideo.c. Here a struct VideoDevice is allocated and then populated. The function will set WIN_StartTextInput, WIN_StopTextInput, and Win_SetTextInputRect in the corresponding structure slots StartTextInput, StopTextInput, and SetTextInputRect. There are no settings made for soft keyboards. The devindex parameters is not used.

The Win_CreatDevice function is not called directly - it is private to the SDL_windowsvideo.c. However, its address is included in the data declaration.

	VideoBootStrap WINDOWS_bootstrap = {

		"windows", "SDL Windows video driver", WIN_CreateDevice };

The typedef for type struct VideoBootStrap is defined in sysvideo.h and the WINDOWS_bootstrap is made external there.

In SDL_video.c the &WINDOWS_bootstrap address, if used, will be made an entry in an array of bootstrap pointers.

Assume that Win_CreateDevice is called. Then the interesting functions are WIN_StartTextInput, WIN_StopTextInput, and WIN_SetTextInputRect. These are defined in windowskeyboard.c They are relatively long, including the other functions called, and I have no idea where this fits in how input is processed in raylib. The length seems a function of being compiled to include IME support (the default).

This pattern applies for each of the video systems supported in SDL2. I have traced out the Windows case out of consistency and familiarity with how IME's are handled. I expect the general structure is the same for other platforms, with the implementation details specific to those systems. In the case of Linux, we are dependent on the GUI system, so there might need to be variations for different desktop Linux GUI systems.

I think the hard part remains. It is beyond my capability and capacity. Like all
of us, I am interested in raylib precisely because I don't have to wade in the paddies of Windows-level APIs and functionality.

[orcmid]

==Afterthoughts
I did all the digging to figure out how StartTextInput, StopTextInput, and SetTextInputRect get wired up with platform-specific implementations, and a bit how the result is held global (_this).

What I did not address is how, using SDL2, this is all tied into the event loop and the arrival of keypresses/character codes in the interactive operation of an SDL2-based application.

The equivalent operations in raylib need to be understood. The handling of keyboard/mouse/controller events and their filtering/routing are relevant to how and when IME-processing is spliced into the arrival of character codes in the graphical application. That is, how the app shifts from keyboard/mouse actions (aim, shoot, reload, run, walk, crouch, etc.) and acceptance of text (e.g., talk, respond to a prompt).

For me, the raylib handling of all this is yet to be learned and understood. There's probably even more that I don't know I don't know ;). And to think that I got here because I was frustrated with "C++ Fundamentals for Game Development" not using Native Windows tools while intrigued by it using raylib as a lightweight introduction into video apps. I now return to my regularly-scheduled programming.

[raysan5]

@orcmid raylib is similar to SDL2 in the way Events are received but SDL2 requires manual polling and processing of events while raylib does it automatically on EndDrawing() by default (lately the option for user-managed event polling was added).

I've been insvestigating a bit and it seems SDL_StartTextInput() calls ShowScreenKeyboard() but I couldn't find that function code...

[orcmid]

... it seems SDL_StartTextInput() calls ShowScreenKeyboard() but I couldn't find that function code...

It will only do that if _this->ShowScreenKeyboard is not NULL. I did not see any code that sets up the Screen Keyboard functions with non-NULL entries as part of the WIN_CreateDevice setup of a SDL_VideoDevice struct. Maybe it is done for other platforms.

[mia-bentzen]

It will only do that if _this->ShowScreenKeyboard is not NULL. I did not see any code that sets up the Screen Keyboard functions with non-NULL entries as part of the WIN_CreateDevice setup of a SDL_VideoDevice struct. Maybe it is done for other platforms.

I think this might be because SDL2 doesn't fully implement the IME on windows. I don't know if it's still a thing (the posts were from 2020 for reference) but when searching for the SDL IME API i found quite a few people having problems with it not showing the suggestions box at all, and some say that it's not actually implemented. (It seems to work consistently on mac and linux though, so it should still be a good reference for those platforms)

Also, i managed to find the windows IME API. https://docs.microsoft.com/en-us/windows/win32/intl/input-method-manager
This might be the 'old' API however (the new one seems to be called Text Services Framework or TSF). I'm not sure how i'm important it is the use the new one, but regardless here is the docs for that as well: https://docs.microsoft.com/en-us/windows/win32/tsf/text-services-framework

[orcmid]

It will only do that if _this->ShowScreenKeyboard is not NULL. ...
I think this might be because SDL2 doesn't fully implement the IME on windows.... (It seems to work consistently on mac and linux though, so it should still be a good reference for those platforms)

I did find where it wires up IME in Windows, it is just not in a ShowScreenKeyboard function. That appears to get deep into the Windows display loop, so I don't have a good way to dig farther on the SDL2 side. I am also in no position to deal with Mac and Linux cases. :(