Japanese input in simutrans 120.0.1 under Linux

[Dwachs]

I will add a message EVENT_STRING, which takes an malloced utf8 string as ev_ptr parameter, see r7455

This is not complete: there is nothing committed that would generate such an event.

[Ters]

Maybe prissi thought kagari could do the rest or something.

[kagari]

I will add a message EVENT_STRING, which takes an malloced utf8 string as ev_ptr parameter, see r7455

Thank you for taking this!

Why can't it be an array in the event? Too big to have on the stack?

It's better to make no limit against the length that users can type at once. Let's not follow SDL2 in this regard. For some reason, they truncate the input to 32 bytes despite the fact that East Asian characters take at least three bytes in UTF-8.

I've attached a new patch that takes advantage of EVENT_STRING. Please note that it only deals with the SDL_TEXTINPUT bug that is preventing to input any CJK characters on SDL2 backend. Even with the patch, it's still nearly unusable for typical CJK players because Simutrans doesn't show the typed characters on the screen until they commit it. To fix it, it will need to make gui_textinput_t aware of the concept of the composition text. GDI Simutrans have had no issue by letting IMs show it in their own windows.

[prissi]

The not showing of characters is rather an SDL2 problem I would say ...

Now the IME just needs to be aware of the test field with input under GDI.

[kagari]

Going back to the original topic; As their implementation for Windows/IMM and X11/IBus is a disappointment, it probably isn't possible to get East Asian input right using SDL2 at the moment. I will try to report bugs I found but it would take at least months to get it fixed. Writing another backend in matured toolkits (such as GTK+, Qt) is obviously overkill, either...

[TurfIt]

Committing this as r7469 has completely broke GDI for MinGW. EDIT: just missing #include <imm.h>  fixed r7471



Also, the SDL2 change has completely broke text input there. SDL_TEXTINPUT is used for all text input (hotkeys...) not just gui_textinput which is the only widgit using EVENT_STRING. The only way to get sane character input in SDL2 is through the SDL_TEXTINPUT event, hence that's what was done. The event.key.keysym.unicode member from the SDL_KEYDOWN that worked just fine with SDL1 was removed for SDL2. Single characters received from SDL_TEXTINPUT *must* map to EVENT_KEYBOARD, not EVENT_STRING.

[kagari]

Oops, I'll fix it soon.

[kagari]

Patch to restore hotkeys. Sorry.

[kagari]

prissi, I feel that some changes in r7474 was rather odd, specifically the part that sending SIM_KEYBOARD out of composition result. It assigned hotkeys to one-length string sent from IME, but in order to send it to the application, people would actually have to press two keys, a character key plus enter key, or even more to pick it from candidates. So they wouldn't bother to use IME just for hotkeys... But It's probably harmless and I think fine to leave it as is.

[prissi]

OK, I added also support for input at cursor position (if a textinput is active).

[kagari]

I was thinking I might have to add another event in order to implement preedit. It looks much more simpler. 

[prissi]

I wonder, can this added to SDL2 too ...

[kagari]

I'm getting close to it.
I will make a patch after merging the branch with r7477, and some more testing. Perhaps next weekend.

[kagari]

Here is a patch to add advanced support for East Asian text input, based on r7477 and subsequent changesets.

• As seen in my previous screenshot, it draws the composition text in between the normal text, with underline. The target clause of conversion is marked by color inversion.
• It adds a new simsys function 'dr_notify_input_pos', which tells IME where its own windows (e.g. suggests, candidates) should be opened. Implemented in GDI and SDL2. (Is it acceptable?)
• SDL_TEXTEDITING is used only on Mac for now. Its field values (start, length) oddly varies by platforms, and doesn't give meaningful data except for Mac.

IMM code in GDI is briefly tested on these Windows IMEs:

• MS-IME 2007 (Japanese, Korean)
• Google Japanese Input
• MS New Phonetic IME 10.0 (Traditional Chinese - Pinyin)

It has some complicated changes... Thank you again for taking the time.

[prissi]

THank you for the patch. Incorporated in r7512; I wonder what games on Linux with SDL2 do?

[kagari]

We will have to wait for the SDL team to take a look at IME bugs I have reported before enable it on Linux. Looks like they usually take a few months to respond...