News:

The Forum Rules and Guidelines
Our forum has Rules and Guidelines. Please, be kind and read them ;).

How to compile Simutrans with Msys2 (mingw64)

Started by prissi, September 01, 2020, 07:57:10 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

prissi

While there is an old documentation https://forum.simutrans.com/index.php/topic,16688.0.html I have tried to make things a little easier (especially since libminiupnpc is broken for years now).

1. step: Download msys2 from here: https://www.msys2.org/

2. step: Unpack from the attached file setup-ming.sh to your mingw home directory (typically C:\msys64\home\USERNAME).

3. step: Open either mingw32 (if you want to build 32 bit executables) or mingw64 (for 64 bit if you use really large maps) from the start menu and type in the terminal windows "./setup-mingw.sh"

If there are errors, type "pacman -S gcc" and confirm and then repeat the step above.

Simutrans environment is now almost complete. (It lacks the Installer, which your probably not need.)

To compile, just type "make" into the window. The executable in now at "C:\mingw64\home\simutrans\trunk\sim.exe".

Next time you open the window you type

cd simutrans
svn up
cd trunk
make

And you have again the latest version of Simutrans.

Matthew

#1
Prissi, thank you for this! I was able to compile Standard for the first time. I know this was because the instructions were so clear; I suspect it's also because there's a lot of cleverness and effort packed into that script!

Here are some other steps that I found necessary so that other beginners can find all the information that they need in one place:

4. Change to the directory containing the Simutrans code by entering cd simutrans/trunk

5. [As above] To compile, just type "make" into the window. The executable is now in C:\msys64\home\USERNAME\simutrans\trunk, called "sim.exe".

As I did not already have a Simutrans-Standard installation, I also had to do the following

6. Copy C:\msys64\home\USERNAME\simutrans\trunk\simutrans to a new folder that the game will use, such as C:\simutrans , and add sim.exe there

7. Copy a pakset folder into that game folder and note the location of the pakset's folder

8. Open C:\simutrans\config\simuconf.tab in a text editor (e.g. Notepad), search for "program stuff", and make sure that you have a pak_file_path = line pointing to your pakset, including the slash (/) at the end.

9. Run sim.exe.

Now the next challenge for me is to try to adapt this method to compiling Extended......
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

prissi

This installation of mingw64 should compile also extended. Download (clone) the git into a path below C:\mingw64\home\username The config.default you can probably copy from standard. And type make. It should compile and link.

Roboron

I also had to install gcc (not $MINGW_PACKAGE_PREFIX-gcc) otherwise miniupnp make failed:

make -f Makefile.mingw
cc -Wall -Os -DNDEBUG -D_WIN32_WINNT=0X501 -o wingenminiupnpcstrings.exe wingenminiupnpcstrings.c
make: cc: No such file or directory
make: *** [Makefile.mingw:80: wingenminiupnpcstrings.exe] Error 127


Doing a pacman -Syu at the start of the script is also a good idea.

prissi

If you do pacman -Syu then it often happens that pacman stops and say to close this window. It will confuse people.

On GCC I had it the other way round, installing gcc prevented building. Which target (32 or 64 bit)?

Roboron

Quote from: prissi on September 05, 2020, 01:10:17 PMIf you do pacman -Syu then it often happens that pacman stops and say to close this window. It will confuse people.

Does it? I have never seen such behaviour in pacman. However, maybe MINGW implementation of pacman is different from the original...

I was targeting 64 bit.

TurfIt

You need to restart the msys terminal when pacman or msys base files are updated, pacman tells you when to do so.
But, I don't get this script. Appears in violation of the msys instructions: MSYS2 introduction

Use msys2 shell for running pacman, makepkg, makepkg-mingw and for building POSIX-dependent software that you don't intend to distribute. Use mingw shells for building native Windows software and other tasks.

You don't/can't?/shouldn't? pacman and compile (expecting native Windows) from the same shell.

prissi

Acutally, you have to use pacman from mingw32 or ming64 or else no development libraries will be installed (at least when I tried this with this script two weeks ago). That is why you need the MING prefix.

If you use it from MSYS, you cannot guess the target too.

Also the string daos not say not to use msys2 only for pacmac. It reads like a revcommendation (and I used pacman from mingw32 commandline for years, so even if it not encouraged, it seems to work well.)

Matthew

Quote from: prissi on September 03, 2020, 12:22:01 PM
This installation of mingw64 should compile also extended. Download (clone) the git into a path below C:\mingw64\home\username The config.default you can probably copy from standard. And type make. It should compile and link.

Thank you for the suggestions. I have tried this with mixed results. I will describe it all here to get things clear in my head and for the benefit of others coming across the same problem(s).

Cloning the Extended repository worked fine. Using the mingw64 shell to compile using the Standard config.default failed because it has no COLOR_DEPTH setting. So I melded the GDI settings from that file into the Extended config.default that I successfully use on Linux (merged file attached below). This failed with the following error message:

===> RES simres.rc
/bin/sh: x86_64-w64-mingw32-windres: command not found
make: *** [common.mk:60: build/default/simres.o] Error 127
make: *** Waiting for unfinished jobs....


(I tried compiling with SDL2 instead of GDI, but unsurprisingly got exactly the same error).

According to discussions on Stack Overflow, make error 127 means that make could not find the path for a required file. I guess that x86_64-w64-mingw32-windres is the Resource Compiler needed to compile simres.rc, so make is looking for x86_64-w64-mingw32-windres but can't find it.

I found an MSYS2 bug report which says that the file that everyone expects to be called x86_64-w64-mingw32-windres is just called windres in MSYS2. That would fit with these lines in Simutrans-Extended's makefile:

ifneq ($(findstring $(OSTYPE), cygwin mingw32 mingw64),)
  SOURCES += simres.rc
  # See https://sourceforge.net/p/mingw-w64/discussion/723798/thread/bf2a464d/
  ifeq ($(OSTYPE), mingw32)
    WINDRES ?= windres -F pe-i386
  else
    ifeq ($(OSTYPE), mingw64)
      WINDRES ?= x86_64-w64-mingw32-windres
    endif
  endif
endif


I can't read makefile language, but it looks as though Extended is indeed configured with the 'wrong' file name. I had a look at the Standard makefile and it's got the 'right' name:

ifeq ($(OSTYPE),mingw)
  SOURCES += simres.rc
  WINDRES ?= windres
endif


But it seems that the Extended makefile also contains logic that the Standard one does not. And I am slightly concerned that the file name is x86_64-w64-mingw32-windres. Yes, the first part (x86_64) suggests that it's the right resource compiler for 64-bit builds, but I thought that mingw32 and mingw64 refer to different sets of packages?

Next time I have access to Windows, I will try to track down windres in MSYS2's filesystem.

If anyone who understands makefile language is able to advise how to edit the Extended makefile to point to windres in MSYS2 without breaking other platforms, then I would be grateful and will try to test it. I would rather not try to learn makefile on my own when I am have only just begun to learn C++  :-[

BTW if anyone is using CMake, a bug report suggests that it has (or perhaps had?) exactly the same problem.
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

TurfIt

#9
Quote from: prissi on September 06, 2020, 11:36:25 AM
Acutally, you have to use pacman from mingw32 or ming64 or else no development libraries will be installed (at least when I tried this with this script two weeks ago). That is why you need the MING prefix.
Not sure what you mean by "no development libraries will be installed"...?
"pacman -S zstd" from the Msys2.exe shell gets you zstd for msys2, "pacman -S mingw-w64-x86_64-zstd" get you zstd for mingw64, to be used in the mingw64.exe shell.


Quote from: Matthew on September 03, 2020, 06:23:57 AM
Now the next challenge for me is to try to adapt this method to compiling Extended......
Good luck. The Makefile is usually broken, and quite often the code itself infested with MSVC'isms, better than last I tried a few years ago, but still...

windres is a good example. Standard has it correct, there's no need to differentiate mingw32 from mingw64, the respective /mingw32/bin/windres.exe and /mingw64/bin/windres.exe have the correct defaults for each environment.  The full prefixed versions are for cross compiling, and ones who cross compile should set their environment correctly, not mess with the makefile breaking everybody else.  You can simply set WINDRES=windres in your config.default to undo - the cross compiler people should have done that themselves to set the cross compile versions leaving the makefile for native builds.

For config.default, it's made from config.template. Use Extendeds, not standards....

But even with that, current GCC will not link Extended:

===> LD  /r/build/Simutrans-Extended.exe
R:/build/bauer/brueckenbauer.o:brueckenbauer.cc:(.text+0x55ee): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `TLS init function for karte_t::marker_index'
R:/build/bauer/brueckenbauer.o:brueckenbauer.cc:(.text+0x560b): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `TLS init function for karte_t::marker_index'
...
R:/build/bauer/wegbauer.o:wegbauer.cc:(.text+0x7615): additional relocation overflows omitted from the output
collect2.exe: error: ld returned 1 exit status
make: *** [common.mk:27: /r/build/Simutrans-Extended.exe] Error 1

Changing memory model doesn't fix, and mucking with linker order is too much work right now... maybe in another few years...
Well it does build in 32bit, but that won't work for the BB server game which needs 8.5GB ram to init!

Ters

Quote from: Matthew on September 07, 2020, 01:57:02 PMAnd I am slightly concerned that the file name is x86_64-w64-mingw32-windres. Yes, the first part (x86_64) suggests that it's the right resource compiler for 64-bit builds, but I thought that mingw32 and mingw64 refer to different sets of packages?

Both mingw32 and mingw-w64 appead to build their executables from mainline GCC and binutils. There is no difference between them there, so the old name sticks. See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55886. The difference between the two projects is the header files and import libraries, or rather how they go about making them. mingw-w64's approach simply allowed them to get 64-bit support faster, which possibly was the driving goal, hence the name. It is however likely that the project also provided the necessary changes for GCC and binutils. I think MSYS2 only gets its headers and libraries from mingw-w64.

Matthew

I have now been able to compile Extended in MSYS2!  :D

The necessary change to the Extended makefile (to use the correct windres as discussed above) is in this branch.

The resulting .exe appears to be a 32-bit executable. It's called Simutrans-Extended.exe (as opposed to -64.exe) and it's ~9kB as opposed to the ~16kB of the 64-bit file.

But it loads and runs (for a few minutes at least) a Bridgewater-Brunel save in single-player! I don't know how this is possible, since the save expands to more than 6GB, and the game is using 7 to 8GB including sound and graphics, which should exceed the 32-bit memory limit, right?

The next challenge is connecting to the online, multi-player game. Extended uses the latest Git commit hash to ensure that server and client are compatible. This poses two issues.

Firstly, the hashes need to be the same, i.e., the same commit must be at the end of the commit history. But I'm not sure how to do this. If I incorporate new changes from upstream/master ( = James' master branch) using git merge, then the latest commit is the merge commit. If I incorporate new changes from upstream using git rebase, then my change to the makefile becomes the latest commit. So either way, I have the wrong latest commit.

Secondly, I wonder whether my MSYS2+GDI client actually is compatible with the GCC/Linux server, as I don't want to crash the server (or even worse, corrupt the savefile!). I do not expect the GDI part to be a problem, because the server has no graphics at all. And since my client is compiled using the GCC toolchain, it should be closer to the server than MSVC builds (which are known to be incompatible). But I do wonder whether the 32-bit vs 64-bit mystery will come back to bite me here.
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

prissi

The GUI (SDL2 or GDI) should not matter for the server.

If you can load a 6GB save game, then you have a 64bit built. Anything above 3 GB really does only work with 64 bit.

You can connect to any server, by explicitely loading "net:bridgewater-brunel.me.uk" This bypasses the checksum check. If it stays in sync, this I depends on that you have really the same version. Otherwise you can compile an explicitely set the revision/hash to the current one.

Roboron

Quote from: Matthew on October 01, 2020, 04:14:45 AMFirstly, the hashes need to be the same, i.e., the same commit must be at the end of the commit history. But I'm not sure how to do this.

I have the same problem with AUR package (since it pulls the code from github), and I have briefly thought previously about it. The solution is simple, but it has to be done on simutrans-extended side: make a nightly release of the source code at the very same time the binaries are updated. You could then download the same source code which has been used to compile the nightly binaries.

Matthew

I am have had another attempt at compiling Simutrans-Extended in MSYS2/mingw64 with make (without using CMake).

The program compiles and runs, but crashes. I would appreciate knowing whether my diagnosis of the problem is at least plausible.

Trace and logfile

DrMinGW gives the following trace from a level 1 debug build:


The final lines in simu.log seem irrelevant, except that they suggest the default savefile has been loaded and has begun to run.


Analysis of trace

So it appears that the crash occurs when the Path Explorer begins to create new threads. Simutrans' pthread_create_wrapper function is calling msvcrt.dll's callthreadstartex function. My understanding is that msvcrt.dll is the correct C++ library for mingw64 builds, except for threading, because it uses a different set of functions (Win32 threads) to Simutrans (which wants POSIX threads). So Extended is distributed with pthreadsGC2.dll and we want the Simutrans executable to call the functions there. But it is apparently still calling msvcrt.dll.

Attempted diagnosis of cause

Multithreading is enabled for this build, because configure.default includes this line: MULTI_THREAD = 1 # Enable multithreading

That line should tell make to use pthreads, because the Simutrans-Extended Makefile includes this section:

ifneq ($(MULTI_THREAD),)
  ifeq ($(shell expr $(MULTI_THREAD) \>= 1), 1)
    CFLAGS += -DMULTI_THREAD
    ifeq ($(OSTYPE),mingw32 mingw64)
#use lpthreadGC2d for debug alternatively
#       Disabled, as this does not work for cross-compiling
#      LDFLAGS += -lpthreadGC2
       LDFLAGS += -static -lpthread
    else
      ifneq ($(OSTYPE),haiku)
        LDFLAGS += -lpthread
      endif
    endif
  endif
endif


We have set the MULTI_THREAD and mingw64 flags, so I think the linker (LD) should know to link to pthread, not msvcrt. So my diagnosis of the problem is that the linker is not linking to pthreads in the executable. Is that at least plausible?
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

prissi

The thread model does not matter, simutrans could do with any which have the same API. Standard switched to the default one, which is pthread using the winodws native threads. (There is even a header file only to enable threads, which, at least at some point, also compiled simutrans correctly.)

The access violation surely should not depend on the library and rather hint at some fundamental problem. It could be that GC threads allows this even though it has not been done properly. The memory was not allowed to be written to, but the reason out of context is unclear.

A thread in windows gets 1 MB, which is a lot. However, if there are non static large datastructures (arrays for instance) created as local variables, one can run out of stack. But I am not sure what happened here.

EDIT: That variable is declared static thread_local together with a friend function. Maybe mingw in the newest version handles thread local differently and thus the friend void* path_explorer_threaded(void* args) is not allowed access because the std::thread and pthread are not the same. However, I have no experience with the thread_local keyword.