Japanese input in simutrans 120.0.1 under Linux

[danivenk]

Dear Co-Simutransplayers,

When I try to rename my stations in Japanese it won't do it.
I'm using simutrans 120.0.1, lang= jp
For Japanese input I use Anthy.
My operating system is Linux Ubuntu LTS 14.04
I really need those japanese names. So please help me out.

danivenk

[Dwachs]

Seems like this bug is already fixed in the nightly versions:

http://forum.simutrans.com/index.php?topic=14088.0

[TurfIt]

Not likely given that fix was to the GDI backend which is not applicable to Linux...
I really don't know how non-latin language input works, so can't help. My only suggestion is to try SDL2 if SDL doesn't work, and v.v.

[danivenk]

Seems like this bug is already fixed in the nightly versions:

http://forum.simutrans.com/index.php?topic=14088.0

You know this is about WIndows, so I don't know if it'll work under Linux.

[Ters]

Just to clear it up: What Dwachs links to has nothing to do with anything but Windows GDI builds of Simutrans. It was also fixed before 120.0.1.

danivenk, do you know if your Simutrans uses SDL1 or SDL2? And is the problem just with Japanese, or does it affect other non-English letters like Ü and IJ as well?

[danivenk]

danivenk, do you know if your Simutrans uses SDL1 or SDL2? And is the problem just with Japanese, or does it affect other non-English letters like Ü and IJ as well?

I don't know if it uses SDL1 or 2, do you know how I can see that?
I just tried the Ü and Ÿ and they both don't show up.

[Ters]

I don't know if it uses SDL1 or 2, do you know how I can see that?

That depends on how you got Simutrans. Maybe it says so on the download page. Maybe you got Simutrans from someplace we know. Some more technical tools, like ldd, might reveal something. Just don't ask me how to get that working.

I just tried the Ü and Ÿ and they both don't show up.

That means it's a generic Unicode problem, and not just something with these IME things I understand is used for East Asian scripts. It should also mean more people are affected. Unfortunately, I left my Linux box two hours before you posted this bug report, and won't be back at it for two weeks. There should be other Linux users around here.

[DrSuperGood]

Could it be missing typefaces/fonts? Unless Simutrans explicitly bundles all used typeface/font (which it would appear not looking at there being code to locate OS typeface/fonts) then it may run into problems with Unicode support at times. Not all installations have for all typeface/font all Unicode extensions. It could be that on that particular installation the particular typeface/font used by Simutrans is missing the Unicode Japanese character set extension.

[Ters]

Could it be missing typefaces/fonts? Unless Simutrans explicitly bundles all used typeface/font (which it would appear not looking at there being code to locate OS typeface/fonts) then it may run into problems with Unicode support at times. Not all installations have for all typeface/font all Unicode extensions. It could be that on that particular installation the particular typeface/font used by Simutrans is missing the Unicode Japanese character set extension.

Simutrans has always bundled it's fonts. They are in a special format not used by much else anyway. At least characters like ü should work out of the box.

[DrSuperGood]

Does it only affect station renaming? All renaming? What about multiplayer chat? It is important to limit where the characters could be getting lost.

[Ters]

Does it only affect station renaming? All renaming? What about multiplayer chat? It is important to limit where the characters could be getting lost.

I just assumes it affects all text. Otherwise, it would affect all platforms, not just Linux. But if this is some special Ubuntu version of Simutrans, they can have modified anything.

[danivenk]

I just assumes it affects all text. Otherwise, it would affect all platforms, not just Linux. But if this is some special Ubuntu version of Simutrans, they can have modified anything.

I've got the simutrans version for ubuntu that's on the simutrans website. And it affects indeed all renaming.

[DrSuperGood]

Could it be some issue with character input? If it is platform dependant (not a problem on Windows/mac builds) then that would point to one of the platform specific libraries responsible for inputing characters. Unless it provides the represented character in a wider character format (eg Unicode) then it would have problems inputting such characters. There is also an issue of what wider character format is used since there are multiple >8bit character encoding schemes of which Unicode is only one and they have varying degrees of compatibility with each other. It would be a possible cause if the input character is in an incorrect encode scheme (although usually that would result in the missing character box to be displayed).

[Ters]

I've got the simutrans version for ubuntu that's on the simutrans website. And it affects indeed all renaming.

I can't see any Ubuntu specific download on the Simutrans webpage. Are you referring to the generic Linux version? If so, was it the 32-bit or 64-bit version (zip file will contain the number 86 and 64 respectively)?

Could it be some issue with character input?

That's what I'm suspecting. There are three suspects: A bug in Simutrans itself (should affect many users), a bug/error in the SDL used by danivenk (probably the standard Ubuntu package, and if so, there should still be several affected users), or something misconfigued on danivenk's system (would most likely affect other programs as well).

[danivenk]

I can't see any Ubuntu specific download on the Simutrans webpage. Are you referring to the generic Linux version? If so, was it the 32-bit or 64-bit version (zip file will contain the number 86 and 64 respectively)?

I'm using the 32 bit version.

[htrkdk]

Hi,

I'm playing Simutrans r7373 on debian jessie (64 bit) and Ubuntu 14.04 (32 bit) in Japanese,
but I've never met such Japanese input problem... (Please see the attachment.)

My environment is:
Simutrans r7373 32bit
SDL 1.2
Japanese input method: fcitx-mozc

If there is still input trouble, please try following:
1. Open a text editor such as gedit.
2. Type  the Japanese words you want to put in Simutrans.
3. Copy the words and paste them in the text box to rename.

This method is used by many Japanese Simutrans players when sim-wingdi had the same problem.
I hope this reply will help you in any ways


ADD: Sorry, the version of SDL was incorrect, so I fixed it. (2 -> 1.2)

[Ters]

I have been able to test out the official 64-bit Simutrans 12.0.1 release on another Linux box I have, although it was not set up for desktop use. There were no problems typing the letters æ, ø and å. So either this problem with non-English letters is specific to the 32-bit release, or something is wrong with danivenk's system or Ubuntu in general (or at least that version of Ubuntu). I really thought there were some other Ubuntu users around here. Maybe they're just a bit busy this time of year, be it exams, end-of-year deadlines or just pre-Christmas stress.

If there is still input trouble, please try following:
1. Open a text editor such as gedit.
2. Type  the Japanese words you want to put in Simutrans.
3. Copy the words and paste them in the text box to rename.

This method is used by many Japanese Simutrans players when sim-wingdi had the same problem.

I believe Simutrans only integrates with the system clipboard on Windows. If so, and if pasting from outside Simutrans works elsewhere, those systems must emulate keyboard input in order to paste into Simutrans (and copying from Simutrans won't work). It could still be interesting to try, just in case there is emulated keyboard input and if that behaves any differently from the real thing.

[htrkdk]

I believe Simutrans only integrates with the system clipboard on Windows. If so, and if pasting from outside Simutrans works elsewhere, those systems must emulate keyboard input in order to paste into Simutrans (and copying from Simutrans won't work). It could still be interesting to try, just in case there is emulated keyboard input and if that behaves any differently from the real thing.

I'm very sorry that I didn't check if the method would work successfully...

So either this problem with non-English letters is specific to the 32-bit release, or something is wrong with danivenk's system or Ubuntu in general (or at least that version of Ubuntu). I really thought there were some other Ubuntu users around here. Maybe they're just a bit busy this time of year, be it exams, end-of-year deadlines or just pre-Christmas stress.

After I posted the first reply, I tried to reproduce danivenk's situation on my Ubuntu again, but I couldn't. It seems there are no problems with 32-bit version of Simutrans and Ubuntu 14.04... Still I have no problems to input Japanese.

[danivenk]

I yesterday tried to fix it, but before I even can start simutrans is gives this error:

Code Select Expand

FATAL ERROR: simmain() - No GUI themes found! Please re-install!
Aborting program execution ...

Sorry I know this is about something else but if this doesn't get fixed I won't be able to use simutrans. (by the way, I tried to re-install but failed still the same error.
:(

[kierongreen]

This indicates you haven't got all the files necessary to start Simutrans. Ensure that you download the complete package and a pakset.

[Ters]

Simutrans 120.0.1 was originally released with some missing files in the Linux release. It should be fixed now (what I downloaded works), but maybe some sourceforge mirrors are out-of-date.

[danivenk]

Simutrans 120.0.1 was originally released with some missing files in the Linux release. It should be fixed now (what I downloaded works), but maybe some sourceforge mirrors are out-of-date.

So how do I get every file I need. If through the simutrans website not everything available is?

[Ters]

So how do I get every file I need. If through the simutrans website not everything available is?

Everything should be available. What I got was complete. You must have been unlucky in the mirror sourceforge selected for you. Try to select a different mirror on the download page, that is after simutrans.com sends you to sourceforge.net.

[danivenk]

Everything should be available. What I got was complete. You must have been unlucky in the mirror sourceforge selected for you. Try to select a different mirror on the download page, that is after simutrans.com sends you to sourceforge.net.

Could you send that version of yours?

[Ters]

All I have is the 64-bit version, which isn't really recommended, and you use the 32-bit version and might not have a 64-bit system. (Maybe just the 32-bit version lacks some files, but then someone really needs to fix that. I thought that was done, and you're first attempt at downloading it seems to have gotten all files.)

[danivenk]

All I have is the 64-bit version, which isn't really recommended, and you use the 32-bit version and might not have a 64-bit system. (Maybe just the 32-bit version lacks some files, but then someone really needs to fix that. I thought that was done, and you're first attempt at downloading it seems to have gotten all files.)

I used 32 by default but I have a 64-bit system so it won't matter.

[danivenk]

SDL and GDI What is different? †
The main features of the GDI version (Windows native)

Windows only
I can Japanese input with an IME
Since the mixer is not, sound effects Buchigireru. I would be arbitrarily change the volume setting
Can the character of the paste from the clipboard [Ctrl + V]
The main features of the SDL version

Supports many platforms
It is not possible to Japanese input (Windows version is acceptable by the character of the paste)
Sound effect is reproduced properly
Character of the paste from the clipboard Windows version only
Or more of either to work smoothly, it seems also different depending on the environment. Japanese input is necessary if GDI version, it is SDL version if you want to enjoy the sound effects.
Save data but compatibility has been tentatively secured, is rarely better you play focus on one or the other because no longer be read in one.

I found this on the http://japanese.simutrans.com site under the FAQ, how unfortune :(
Is this for real or is there a good way of inputing Japanese (hiragana, katakana and kanji)?

[prissi]

When I tried the SDL version under LInux, I had trouble to install an IME, so I gave up (like eight years ago).

If there would ahve been an easy way to use the Xwindows clipboard under Linux, then one could at least copy & paste. You may try also to compile for SDL2 which may support the IDE better (although I doub it).

[kagari]

I've got the simutrans version for ubuntu that's on the simutrans website. And it affects indeed all renaming.

If you got a package labelled as "Linux SDL", it's depends to SDL1. I'd like to ask what Input Method framework are you using (such as IBus, fcitx), as Anthy you mentioned is a conversion engine, which communicates with applications through these frameworks. It's Ubuntu so most likely IBus, though.

Anyway, I recently explored with a hope to take advantage of latest IBus support in SDL2 development repository, and I found: SDL2 backend has a bug!

When you type random text contains multiple characters using Input Method Editors, only first character will show up in the input field, and rest of the text you wrote will be ignored.

What You Typed: いろは
On The Screen: い

I figured out the problem is down to the SDL_TEXTINPUT event handler.

simsys_s2.cpp:

Code Select Expand


case SDL_TEXTINPUT: {
size_t len = 0;
sys_event.type    = SIM_KEYBOARD;
sys_event.code    = utf8_to_utf16( (utf8*)event.text.text, &len );
sys_event.key_mod = ModifierKeys();
break;
}


In the text input process involving IMs, SDL_TEXTINPUT is possible to come with multiple characters, but current code breaks immediately after it issues a SIM_KEYBOARD event out of the first character of 'event.text.text', leaving trailing characters unprocessed.

Attached patch is an attempt to modify the SDL_TEXTINPUT handler to issue SIM_KEYBOARD events for every character it received by introducing static variables in the function. I spend some time with it and had seen no issues at least on my Linux desktop. This issue should also affect Windows and Mac, both SDL.

danivenk, could you try the patch with SDL2? IBus support code has not yet arrived to stable release, so in first you'll need to compile SDL source code _with_ IBus development packages (ibus-dev or ibus-devel or etc.) installed. And then apply patch to latest Simutrans and again compile it. It's a long road unfortunately.

Also, I saw "No GUI themes found!" a number of times when I tried to launch nightly. cd into working Simutrans directory, and launch your new simutrans with specifying -use_workdir option.

[Ters]

Attached patch is an attempt to modify the SDL_TEXTINPUT handler to issue SIM_KEYBOARD events for every character it received by introducing static variables in the function. I spend some time with it and had seen no issues at least on my Linux desktop. This issue should also affect Windows and Mac, both SDL.

There are some issues here. Since Simutrans processes one (non-mouse) event per frame, since character and keyboard input is handled the same, and since Simutrans binds actions to characters rather than keys, external input generating multiple characters can lead to strange behaviour. The first character may trigger something, while the other characters go someplace entirely different. It's possible that Simutrans' event model is unsuited for use with IMEs, but since prissi has some connection to Japan, it seems strange that he hasn't reacted.

I also don't like this static variable in function thing. If lots of characters are generated so that there always is something in extra_text_buffer, Simutrans won't process other events.

[prissi]

Prissi was on holiday with a broken laptop power supply ... And currently no working Linux on my computer (or rather not enough time to install it in a virtual machine). HOwever, since we have some paste support, maybe multiple character should be incorporated as a paste event? (It still needs some modification of the event system ... )

That seems like the clean solution.

[Ters]

Prissi was on holiday with a broken laptop power supply ... And currently no working Linux on my computer (or rather not enough time to install it in a virtual machine). HOwever, since we have some paste support, maybe multiple character should be incorporated as a paste event? (It still needs some modification of the event system ... )

That seems like the clean solution.

As long as the internal clipboard is used, which is the case on anything but Windows. Sending this input through the Windows clipboard can range from confusing to annoying (where did my important clipboard contents go?).

[kagari]

There are some issues here. Since Simutrans processes one (non-mouse) event per frame, since character and keyboard input is handled the same, and since Simutrans binds actions to characters rather than keys, external input generating multiple characters can lead to strange behaviour. The first character may trigger something, while the other characters go someplace entirely different. It's possible that Simutrans' event model is unsuited for use with IMEs, but since prissi has some connection to Japan, it seems strange that he hasn't reacted.

I also don't like this static variable in function thing. If lots of characters are generated so that there always is something in extra_text_buffer, Simutrans won't process other events.

These are actual problems present in GDI Simutrans due to it isn't really dealing with IMM32 or TSF, relying entirely on DefWindowProc to translate IME input messages into WM_CHARs. An old MSDN document details the behaviour:

IME-unaware applications basically ignore all IME-specific Windows messages. Most applications that target single-byte languages are IME-unaware. When the user presses Enter or clicks a character to place it in a document, the IME, by default, posts either a WM_IME_CHAR message containing an entire DBCS character or a WM_IME_COMPOSITION message containing an entire DBCS string. If the application ignores either message, it falls through to the application's DefWindowProc, which in turn notifies the Input Method Manager that the message has been ignored. The IME then resends the character or string byte by byte via multiple WM_CHAR messages.

So, the basic idea of the patch was to imitate the trick that somewhat magically enabling IME support on Windows Simutrans and to bring it into other platforms. It certainly isn't a nice solution, but after glancing through low-end part of the code I felt it's better not to dive into a major surgery in the first attempt. I'm willing to make another tries in more robust way, something like replacing SIM_KEYBOARD with a more general text input event. I'd like to hear more suggestions from developers.

EDIT: grammar

[prissi]

The IME handling of windows changed at least three times since WIndows 98, and each time documentation was sparse and often incorrect. Not IME messages are discouraged completely; hence ignoring seems the best way.

I will add a message EVENT_STRING, which takes an malloced utf8 string as ev_ptr parameter, see r7455

Still the string has to be in a static variable or we have a memeory leak.

[Ters]

Still the string has to be in a static variable or we have a memeory leak.

Why can't it be an array in the event? Too big to have on the stack?

[Dwachs]

I will add a message EVENT_STRING, which takes an malloced utf8 string as ev_ptr parameter, see r7455

This is not complete: there is nothing committed that would generate such an event.

[Ters]

Maybe prissi thought kagari could do the rest or something.

[kagari]

I will add a message EVENT_STRING, which takes an malloced utf8 string as ev_ptr parameter, see r7455

Thank you for taking this!

Why can't it be an array in the event? Too big to have on the stack?

It's better to make no limit against the length that users can type at once. Let's not follow SDL2 in this regard. For some reason, they truncate the input to 32 bytes despite the fact that East Asian characters take at least three bytes in UTF-8.

I've attached a new patch that takes advantage of EVENT_STRING. Please note that it only deals with the SDL_TEXTINPUT bug that is preventing to input any CJK characters on SDL2 backend. Even with the patch, it's still nearly unusable for typical CJK players because Simutrans doesn't show the typed characters on the screen until they commit it. To fix it, it will need to make gui_textinput_t aware of the concept of the composition text. GDI Simutrans have had no issue by letting IMs show it in their own windows.

[prissi]

The not showing of characters is rather an SDL2 problem I would say ...

Now the IME just needs to be aware of the test field with input under GDI.

[kagari]

Going back to the original topic; As their implementation for Windows/IMM and X11/IBus is a disappointment, it probably isn't possible to get East Asian input right using SDL2 at the moment. I will try to report bugs I found but it would take at least months to get it fixed. Writing another backend in matured toolkits (such as GTK+, Qt) is obviously overkill, either...

[TurfIt]

Committing this as r7469 has completely broke GDI for MinGW. EDIT: just missing #include <imm.h>  fixed r7471



Also, the SDL2 change has completely broke text input there. SDL_TEXTINPUT is used for all text input (hotkeys...) not just gui_textinput which is the only widgit using EVENT_STRING. The only way to get sane character input in SDL2 is through the SDL_TEXTINPUT event, hence that's what was done. The event.key.keysym.unicode member from the SDL_KEYDOWN that worked just fine with SDL1 was removed for SDL2. Single characters received from SDL_TEXTINPUT *must* map to EVENT_KEYBOARD, not EVENT_STRING.

[kagari]

Oops, I'll fix it soon.

[kagari]

Patch to restore hotkeys. Sorry.

[kagari]

prissi, I feel that some changes in r7474 was rather odd, specifically the part that sending SIM_KEYBOARD out of composition result. It assigned hotkeys to one-length string sent from IME, but in order to send it to the application, people would actually have to press two keys, a character key plus enter key, or even more to pick it from candidates. So they wouldn't bother to use IME just for hotkeys... But It's probably harmless and I think fine to leave it as is.

[prissi]

OK, I added also support for input at cursor position (if a textinput is active).

[kagari]

I was thinking I might have to add another event in order to implement preedit. It looks much more simpler. 

[prissi]

I wonder, can this added to SDL2 too ...

[kagari]

I'm getting close to it.
I will make a patch after merging the branch with r7477, and some more testing. Perhaps next weekend.

[kagari]

Here is a patch to add advanced support for East Asian text input, based on r7477 and subsequent changesets.

• As seen in my previous screenshot, it draws the composition text in between the normal text, with underline. The target clause of conversion is marked by color inversion.
• It adds a new simsys function 'dr_notify_input_pos', which tells IME where its own windows (e.g. suggests, candidates) should be opened. Implemented in GDI and SDL2. (Is it acceptable?)
• SDL_TEXTEDITING is used only on Mac for now. Its field values (start, length) oddly varies by platforms, and doesn't give meaningful data except for Mac.

IMM code in GDI is briefly tested on these Windows IMEs:

• MS-IME 2007 (Japanese, Korean)
• Google Japanese Input
• MS New Phonetic IME 10.0 (Traditional Chinese - Pinyin)

It has some complicated changes... Thank you again for taking the time.

[prissi]

THank you for the patch. Incorporated in r7512; I wonder what games on Linux with SDL2 do?

[kagari]

We will have to wait for the SDL team to take a look at IME bugs I have reported before enable it on Linux. Looks like they usually take a few months to respond...