[Project] Native Language Support — CJK, Cyrillic, RTL and complex scripts

[victor_18993]

I would like to start a broader Simutrans Standard project focused on native-language support.

The goal is not simply to check whether translations exist or whether Unicode characters can be displayed.

The objective is to verify that native users can actually use Simutrans naturally in their own language, including:

- native keyboard and IME input;
- composition and candidate selection where required;
- correct text rendering and font coverage;
- cursor movement, selection, Backspace and Delete;
- clipboard operations;
- station, line, convoy and other object names;
- Unicode save/load persistence;
- filenames where applicable;
- Squirrel/API string handling;
- multiplayer/chat paths where practical.

The recent Windows GDI IME investigation showed why this distinction matters.

In r12178, the Windows GDI backend was fixed because native IME handling had been suppressed. On current Windows this could not only hide the Japanese candidate window, but could also prevent Japanese text from being committed into Simutrans edit fields.

That made it clear that "the language is translated" and "the language is fully usable" are two different questions.

I therefore propose treating native-language support as a systematic Standard project.

Planned audit groups
--------------------

1. CJK

  Japanese
  Korean
  Simplified Chinese
  Traditional Chinese

  This audit is already in progress.

  It includes native IME composition, candidate windows, text commit, UTF-8 editing, fonts, persistence and backend comparison between Windows GDI and SDL2.

2. Cyrillic

  Russian
  Ukrainian
  Belarusian
  Bulgarian
  Serbian
  Macedonian
  and representative Central Asian Cyrillic languages

  This will also revisit historical reports concerning Russian/Cyrillic encoding, fonts and generated city names.

3. Hebrew / RTL

  The next stage will investigate right-to-left and bidirectional text behaviour, mixed RTL/LTR strings, cursor movement, selection and combining characters.

4. Complex scripts

  Arabic, Persian and related scripts would follow separately because they add contextual shaping and ligature requirements on top of RTL/BiDi.

Development approach
--------------------

This is intended to be an audit-and-fix project, not a large Unicode rewrite.

For each language family:

    audit
      ->
    reproducible compatibility matrix
      ->
    individual confirmed defects
      ->
    small independent patches
      ->
    regression tests where possible
      ->
    integration into Standard

A failure in Japanese input, Korean composition, a Cyrillic font, Hebrew BiDi handling, etc. should therefore become its own reviewable patch rather than being hidden inside one large internationalisation change.

The audits will also establish an acceptance contract for future platform backends. For example, an SDL3 backend should not be considered equivalent if it regresses native-language behaviour already working in GDI or SDL2.

Native-speaker feedback is particularly welcome.

If you currently use Simutrans in Japanese, Korean, Simplified or Traditional Chinese, Russian or another Cyrillic language, Hebrew, Arabic, Persian, or another non-Latin writing system, reports of input, editing, font, save/load or UI problems would be very useful.

The long-term objective is simple:

A native user should not feel that their language is only partially supported because Simutrans was originally developed around a different writing system.

[victor_18993]

I would also like to add a more personal note about why I wanted to start this project.

I am not speaking on behalf of the whole Simutrans development team here. I am speaking as a contributor and developer working on Simutrans Standard.

For me, this project is important because I do not think language support should stop at translation.

Many of the translations that exist today were created and maintained by members of those communities themselves. Players, developers, pakset authors, scenario creators and translators have invested time in making Simutrans accessible in their own languages.

I think we should respect that effort by making sure that those languages are not only visible in the interface, but genuinely comfortable to use.

If you are Japanese, I want you to be able to type, convert and edit Japanese naturally.

If you are Korean, I want Hangul input and editing to behave as you expect.

If you use Simplified or Traditional Chinese, I want native input methods, candidate selection and text handling to work properly.

If you use Russian, Ukrainian, Serbian, Kazakh or another Cyrillic language, I want your own alphabet to work correctly throughout the game.

And later I want us to look seriously at Hebrew, Arabic, Persian and other writing systems that require right-to-left or more complex text handling.

I know that Simutrans has had historical problems in some of these areas. Some languages have had partial support, encoding problems, font problems or input limitations.

That is precisely why I think this work is worth doing.

I would rather identify those weaknesses openly, test them properly and fix them step by step than simply assume that a language is fully supported because a translation exists.

For me, the objective is simple:

I want native users to feel comfortable in Simutrans.

I want you to be able to read, write, name your stations and lines, create scenarios, communicate and play without feeling that your language is an afterthought.

Simutrans is an international community.

It does not belong to one language.

And from my side, as a contributor to Simutrans Standard, I want to help make sure that the people who build, translate, create content for and play Simutrans can use it naturally in the language that feels like home to them.

So if you are a native speaker and something does not work correctly in your language, please tell us.

Those reports are valuable.

They help us understand what the real experience is, and they can directly guide the fixes that come next.

[prissi]

My very first work was Unicode support for Simutrans ... since then windows changed the ime several times and SDL1 at that time had no ime support at all.

The Japanese (and all CJK) input work very well for SDL2, actually, Japan is by far the biggest community. The Windows IME work differently on Japanese and non-Japanese and different windows versions. I would rather ditch GDI Support, which I only use for debugging.

However, the translation system we have is ill suited as it cannot handled Slavic plurals for instance (Different for 1, 2-4, and more) and advanced conjugations. The gettext lib came way too late for Simutrans ...

In any way, the main problem with languages is translators. In 26 years, we have just on Hebrew speaking person posting in the forum, and very few downloads from the right to left countries at all. So adding support would need also to attract translators.

[victor_18993]

Thank you, this historical context is very useful.

I agree that translators are essential, and my intention with the Native Language Support project is not to replace that work. It is to make sure that the technical side of Simutrans does not become an additional obstacle for native users, translators or content creators.

The first CJK audit is already giving us useful results.

The good news is that the underlying Unicode/UTF-8 data handling appears to be very solid: committed CJK text survives editing, clipboard operations, object names, filenames, save/load and Squirrel correctly.

The problems found so far are mostly in the user-facing input/rendering layer rather than in the stored data.

For example, we found that an active IME pre-edit is not drawn when an edit field is empty, even though Simutrans receives the composition correctly. This affects both GDI and SDL2 and can make it look as if nothing is being typed until the composition is committed.

On my current Windows test system, SDL2 conversion itself works correctly, but the native candidate window is not displayed, so I am keeping that as a separate finding for investigation rather than assuming the cause.

The font situation is also interesting: using one CJK-capable font solves CJK rendering but can remove coverage for other scripts. That suggests that proper font fallback may eventually be more appropriate than simply choosing another default font.

Your point about Slavic plurals and advanced conjugation is also very important. I had initially concentrated mainly on input, Unicode, fonts and persistence, but I will add translation semantics as a separate part of the project. A language can be technically writable and still not be properly expressible by the translation system.

Regarding GDI, I understand your point. I will not make any decision about removing it as part of this project; for the language audits I can treat SDL2 as the main reference and GDI as a compatibility/debug comparison where useful.

And I completely agree about the difficulty of supporting languages without native speakers and translators. That is one reason I wanted to make this work visible: if Japanese, Korean, Chinese, Russian, Hebrew or other native users see that their reports are actively useful, hopefully it becomes a little easier for them to participate.

I will continue keeping the individual fixes small and independent. The audit should tell us what is actually broken first; then each confirmed problem can be reviewed separately.

[victor_18993]

Note: The forum is currently not allowing me to attach the documentation files. The full multilingual documentation package is ready and will be attached here as soon as the forum attachment function is working again.

Native Language Support documentation — IME text input and r12182

The first documentation package produced by the Native Language Support project is now ready.

It documents the IME text-input behaviour corrected in Simutrans Standard r12182:

    r12182 — FIX: display IME composition in empty text fields

Before this change, Simutrans could receive a valid IME composition while an edit field was empty, but the pre-edit text was not drawn. This could make native input appear to be doing nothing until the composition was finally committed.

The fix changes only the display path. The committed UTF-8 data and normal text-editing behaviour remain unchanged.

The behaviour was verified with:

- Japanese
- Simplified Chinese
- Traditional Chinese
- Windows GDI
- Windows SDL2

Korean documentation is also included because Korean is now part of the Native Language Support work, but Korean runtime validation is still pending and is clearly marked as such in the document.

Developer documentation has also been added to the official Simutrans Doxygen sources:

    r12183 — DOC: document native IME text input support

The forum documentation is available in:

1. English
2. Español
3. Deutsch
4. 日本語
5. 한국어
6. 简体中文
7. 繁體中文

Each edition is provided both as an editable source document and as a PDF.

The English edition is the canonical technical source. Translated editions clearly state their translation and human-review status. Some translations are currently machine-assisted / machine-translated and are awaiting human language review.

Native speakers are very welcome to review the translations and suggest corrections. The technical content has been kept aligned with the English source so that language corrections can be made without changing the documented behaviour.

This documentation does not mean that CJK/native-language support is complete. Other input, font and IME issues identified by the audit remain separate and will continue to be addressed incrementally.

The objective is to improve native-language support in small, reviewable steps while making each completed improvement available to users as soon as it is ready.

[Isaac Eiland-Hall]

working on attachment problem, sorry!

edit: Fixed I think

[victor_18993]

working on attachment problem, sorry!

edit: Fixed I think

Thank you very much for working on it, Isaac! And no need to apologize — you're always very attentive to the forum and quick to help when something goes wrong. 🙂

Unfortunately, it still seems to be failing on my side. I'm getting the same error message saying that there was a problem during the upload and that I should contact the administrator.
Thanks again for taking a look at it!

[victor_18993]

Korean native IME input has now been verified on Simutrans Standard.

The good news is that no additional Korean-specific code change was required.

The fix already integrated in:

    r12182 — FIX: display IME composition in empty text fields

also works correctly with the Microsoft Korean IME.

The Korean tests covered both Windows GDI and SDL2, including real incremental Hangul composition and recomposition:

    ㅎ → 하 → 한 → 한국

and Backspace through an active composition.

The results were green for:

- native Hangul keyboard input
- composition/pre-edit in an empty text field
- incremental Hangul recomposition
- Backspace during composition
- committed UTF-8 integrity
- normal cursor and text editing
- clipboard round-trip
- in-game naming
- save/load persistence
- Windows GDI
- Windows SDL2

The committed text "한국" was also verified as the exact expected UTF-8 data.

This means that the same basic native-input baseline is now verified for all four East Asian languages currently covered by this first NLS pass:

- Japanese
- Korean
- Simplified Chinese
- Traditional Chinese

No Korean-specific patch was necessary: r12182 fixed the underlying IME composition display path generically.

The official English developer documentation has also been updated accordingly:

    r12184 — DOC: record Korean IME input validation

This does not mean that all CJK input work is finished. Other previously identified IME and font-related issues remain separate and will continue to be addressed incrementally.

With the basic East Asian native-input baseline now established, the Native Language Support work is moving on to the Cyrillic family.