Author Topic: Cross-compiling: Linux to Windows  (Read 6506 times)

0 Members and 1 Guest are viewing this topic.

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #70 on: February 05, 2017, 11:20:53 PM »
Some research suggests that static linking pthread is difficult.

However, I am afraid that I cannot find pthreadGC2-w32.dll anywhere on my installation: it is not in the directory mentioned (my version is different, so my actual directory is /usr/lib/gcc/i686-w64-mingw32/5.3-win32 or /usr/lib/gcc/i686-w64-mingw32/5.3-posix: I am not sure which is correct, but, in any event,  pthreadGC2-w32.dll is in neither of them), and "locate pthreadGC2-w32.dll" produces no results.

Edit: On a slightly different point, gdb tells me that the crashes that seem to occur when loading games with the MinGW compiled versions appear to occur in the destructor for marker_t (although GDB still crashes when I try to get a full backtrace). It complains of a segmentation fault. The only code in the destructor to marker_t is delete [] bits - it is really not clear what could be causing this.

Edit 2: Running this with the Visual Studio debugger, I get lots of calls to the destructor to marker_t when "bits" is a null value, but this causes no error in that case.

Edit 3: Running Dr. Memory gives me some more details on this issue:

Code: [Select]

Error #1: UNADDRESSABLE ACCESS: reading 0x00000000-0x00000004 4 byte(s)
# 0 marker_t::~marker_t                        [dataobj/marker.cc:44]
# 1 (anonymous namespace)::run                 [bauer/../simworld.h:2652]
# 2 _pthread_cleanup_dest.part.6               [bauer/brueckenbauer.cc:1396]
# 3 pthread_exit                               [bauer/brueckenbauer.cc:1396]
# 4 pthread_create_wrapper                     [bauer/brueckenbauer.cc:1396]
# 5 msvcrt.dll!_endthreadex
# 6 msvcrt.dll!_endthreadex
# 7 KERNEL32.dll!BaseThreadInitThunk          +0x11     (0x7573336a <KERNEL32.dll+0x1336a>)
Note: @0:12:10.819 in thread 18036
Note: instruction: mov    (%eax) -> %eax

Edit 4: Neither modifying the code thus:

Code: [Select]
arker_t::~marker_t()
{
    if (bits)
    {
        delete[] bits;
    }
    else
    {
        dbg->warning("marker_t::~marker_t()", "Calling destructor on a marker with NULL bits");
    }
}

nor even commenting out "delete[] bits'" has any effect on these crashes, and GDB still persistently crashes when I try to run this (although it is noted that the crash occurs on a thread other than the main thread; I do not know how well that GDB copes with multi-threading).

The crash occurs only when loading a game after a game has already been loaded (including demo.sve), and is intermittent (i.e. sometimes does not occur).

What seems to happen is that marker_t::the_instance, which I have made thread local so as to work with multi-threading, is deleted when threads are destroyed (as the threads for, e.g. convoy route finding, etc. are on loading a new map), and it is at this point that ~marker_t::() is called and fails. I can only assume that this is a double free error given the symptoms described above, but it is not at all clear how or why this is happening or why this only happens in MinGW builds. I wonder whether there is some error in the implementation of thread_local in MinGW builds or a version of the pthreads dll file that is linked with it?
« Last Edit: February 06, 2017, 01:47:31 AM by jamespetts »
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4624
  • Total likes: 168
  • Helpful: 107
  • Languages: EN, NO
Re: Cross-compiling: Linux to Windows
« Reply #71 on: February 06, 2017, 06:44:47 AM »
Some research suggests that static linking pthread is difficult.

However, I am afraid that I cannot find pthreadGC2-w32.dll anywhere on my installation: it is not in the directory mentioned (my version is different, so my actual directory is /usr/lib/gcc/i686-w64-mingw32/5.3-win32 or /usr/lib/gcc/i686-w64-mingw32/5.3-posix: I am not sure which is correct, but, in any event,  pthreadGC2-w32.dll is in neither of them), and "locate pthreadGC2-w32.dll" produces no results.

Seems like you have both threading models installed. pthreads should come with the posix one, but I'm not sure what is actually used when both are present. Furthermore, the pthread that comes with my mingw64 calls its dll libwinpthread-1.dll. So maybe you have installed pthread separately as well.

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #72 on: February 06, 2017, 09:11:58 AM »
Pthread is of course installed independently from MinGW on the server in order to compile the Linux binaries, but presumably this would not put an extra subdirectory into the MinGW directory?

In any event, libwinpthread-1.dll is not in the /usr/lib/gcc/i686-w64-mingw32/5.3-win32 or the /usr/lib/gcc/i686-w64-mingw32/5.3-posix directories, either.

Edit: Returning to the crash problem, as this seems to be the issue of real importance (the earlier crashes with the cross-compiled version before I switched to static linking are very likely to be the same as the issue that I am having now with the Windows-built MinGW builds): altering the code in marker.cc to:

Code: [Select]
marker_t::~marker_t()
{
    if (this)
    {
        delete[] bits;
    }
}

now produces this, different crash:

Code: [Select]
Dr. Memory version 1.11.0 build 2 built on Aug 29 2016 02:42:07
Dr. Memory results for pid 16740: "simutrans-experimental.exe"
Application cmdline: "C:\msys32\home\James\simutrans\simutrans-experimental\build\default\simutrans-experimental.exe"
Recorded 115 suppression(s) from default C:\Program Files (x86)\Dr. Memory\bin\suppress-default.txt

Error #1: UNADDRESSABLE ACCESS: reading 0x000004bc-0x000004c0 4 byte(s)
# 0 slist_tpl<>::clear                                  [dataobj/../boden/../tpl/slist_tpl.h:359]
# 1 slist_tpl<>::~slist_tpl                             [dataobj/../boden/../tpl/slist_tpl.h:155]
# 2 hashtable_tpl<>::~hashtable_tpl                     [dataobj/../boden/../tpl/hashtable_tpl.h:17]
# 3 ptrhashtable_tpl<>::~ptrhashtable_tpl               [dataobj/../tpl/ptrhashtable_tpl.h:49]
# 4 marker_t::~marker_t                                 [dataobj/marker.cc:42]
# 5 (anonymous namespace)::run                          [bauer/../simworld.h:2652]
# 6 _pthread_cleanup_dest.part.6                        [bauer/brueckenbauer.cc:1396]
# 7 pthread_exit                                        [bauer/brueckenbauer.cc:1396]
# 8 pthread_create_wrapper                              [bauer/brueckenbauer.cc:1396]
# 9 msvcrt.dll!_endthreadex
#10 msvcrt.dll!_endthreadex
#11 KERNEL32.dll!BaseThreadInitThunk                   +0x11     (0x7573336a <KERNEL32.dll+0x1336a>)
Note: @0:01:07.681 in thread 4928
Note: instruction: mov    (%eax) -> %eax

Thus, the problem still appears to occur when marker_t::the_instance is deleted, but the crash bizarrely seems to occur in entirely unrelated parts of the code (a ptrhashtable, presumably ptrhashtable_tpl <const grund_t *, bool> more as declared in marker.h - I assume that the other collection classes are used as part of ptrhashtable_tpl).

Does anyone have any idea why this might be happening (and why only in MinGW builds)? This looks as though it might be some sort of double deletion of marker_t::the_instance (a thread_local variable), but it is how clear how this could be happening or why it does not occur in Visual Studio builds or Linux GCC builds.
 
Edit 2: It occurs to me that these errors are strongly indicative of an error with either the compiler or the pthread library: marker_t::the_instance is a static, thread_local variable, not created with the new keyword: surely, the C++ standard is violated if it is even possible to have a segmentation fault/access violation on the automatic deletion of such an object on the termination of a thread?

Edit 3: It looks as though somebody else is having a similar problem, but there is no solution in that Stack Overflow thread for it.

Edit 4: The output of g++ -v is

Code: [Select]
$ g++ -v
Using built-in specs.
COLLECT_GCC=C:\msys32\mingw32\bin\g++.exe
COLLECT_LTO_WRAPPER=C:/msys32/mingw32/bin/../lib/gcc/i686-w64-mingw32/6.3.0/lto-wrapper.exe
Target: i686-w64-mingw32
Configured with: ../gcc-6.3.0/configure --prefix=/mingw32 --with-local-prefix=/mingw32/local --build=i686-w64-mingw32 --host=i686-w64-mingw32 --target=i686-w64-mingw32 --with-native-system-header-dir=/mingw32/i686-w64-mingw32/include --libexecdir=/mingw32/lib --enable-bootstrap --with-arch=i686 --with-tune=generic --enable-languages=c,lto,c++,objc,obj-c++,fortran,ada --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw32 --with-mpfr=/mingw32 --with-mpc=/mingw32 --with-isl=/mingw32 --with-pkgversion='Rev1, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld --disable-sjlj-exceptions --with-dwarf2
Thread model: posix
gcc version 6.3.0 (Rev1, Built by MSYS2 project)
« Last Edit: February 06, 2017, 10:20:54 AM by jamespetts »
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 8718
  • Total likes: 303
  • Helpful: 228
  • Languages: De,EN,JP
Re: Cross-compiling: Linux to Windows
« Reply #73 on: February 06, 2017, 10:23:12 AM »
Something with an instance (which is a static) cannot really be thread local. But the marker on gihub is anyway not.

It may be that the marker is used by a thread while you delete it. (A variable will not be deleted at the end of a thread. Only if this is a local one.) Also route search does no go well wih thread, because then the order of vehicles arriving at a crossing/stop is random and no network games can stay in sync. (Only on single CPU systems the threads maybe  sequencial and thus in the same order.)

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #74 on: February 06, 2017, 10:31:20 AM »
Thank you for your thoughts on this. It is hard to see how the marker could be being used by the thread during deletion, since the destructor is called from the thread termination routine (# 6 _pthread_cleanup_dest.part.6 ), so the thread in question is already terminating at this point.

As to multi-threading the convoy route searching, this has been done in a particular way to prevent desyncs: the route search has been separated from a convoy's normal step and put into a special thread step. Each thread handles a fraction of the convoys in the game such that between them they all call each convoy's thread step. Only convoys in the ROUTING state have any action called on them in the thread step: in this state, they just find their route, but they then switch to a new ROUTE_JUST_FOUND state, and then perform all of the immediate post-routing actions on the next single-threaded step. This has been tested on my 4-core machine with a large and well developed saved game, and it stays in sync overnight. This system might well be helpful in Standard, too, especially in maps with lots of ships.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 8718
  • Total likes: 303
  • Helpful: 228
  • Languages: De,EN,JP
Re: Cross-compiling: Linux to Windows
« Reply #75 on: February 06, 2017, 12:28:15 PM »
Still, what prevent a route search finishing faster on one machine (i.e. found winthin the next step() ) while it takes two steps() on a slower machine, especially on small servers. That is why route searches are in the step on the standard in the first place. Those things will surface on slow versus fast machines, and probably requiring a lot of ships or other convois, so the routing task is fully occupied. Or like find_route, which must assume the reservations are the same on each machine. Here the sync_steps are important, if your really do this asynchronous.

In route 222 you call welt->unmarkiere_all, which does not look very thread local. SO I assume there is one routing task with an fifo, calculation routes sequetially?

Also, the routing in standard is safe to assume that the world is there static, i.e. unchanged in every moment. Otherwise the bridge (or tunnel) the search is currently on, can be deleted, or the sea has raised, leaving an invalid ground ptr in the NODE search array.

And finally, the initialising of the NODE arrays may take as much time as most route searches. I am not sure this is worth the hassle. Did you do profiling before? Here is an old one (my mingw is currently not compiling) http://forum.simutrans.com/index.php?topic=3166.msg33899#msg33899 where routing of convois did not even reached into the 0.5% range. And that was with lots of convois (about 500 on a 256x256 map)

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #76 on: February 06, 2017, 03:46:55 PM »
I think that you are looking at the code on the master branch, which is the old 11.32 release build from 2013, before multi-threading was implemented. The multi-threaded code is on the devel-new-2 branch, and there is no call to unmarkiere_alle(), whereas there is at line 222 of route.cc on the master branch.

The routing does not run across steps: in other words, calc_route() will not return until a complete route has been found. (I thought that this is how it worked in Standard, too). This means that it is not possible for one vehicle to find a route in a different step on one machine than on another. The thread_step() method is called just before the single threaded step() method and called on the same schedule as step() rather than sync_step(). The idea is that multiple convoys can use the route search simultaneously in parallel to find their routes, reducing the time that the rest of the program has to wait for the route searching of all the convoys that need it in a given step to complete. The route finding is not run concurrently with any function that could change the state of the world (e.g. player interaction). There are some instances of route searching still called from the main thread, such as those used with choose signals, and these run single threadedly.

As to performance, the NODE arrays are only initialised once per thread per loading of a game or creating a new map: a pool of dedicated threads for route searching is retained for the duration of the running of a single saved game/map and they work on an array of convoys indexed by a thread number given to them on thread creation so that they each work on only a subset of convoys but between them complete the route searching for all convoys that need it. The task of assigning convoys to different thread arrays is handled in a single-threaded part of the code (and is much quicker than the route searching: it does not show up as significant on profiling).

On the subject of profiling, performance profiling is not an easy way of finding the real-world performance improvement from multi-threading, as it can only report the percentage of CPU time used by any given method: this, of course, does not change if methods are spread across multiple threads (and therefore multiple CPUs) even if performance is improved by the sharing of the load. However, the idea of multi-threading the convoy route searching in the first place came from profiling showing that, in some cases at least, it takes a significant amount of CPU time (I suspect that ships and road vehicles take much  more CPU load in routing than rail and similar vehicles). I have run the Visual Studio profiler, which does indeed confirm that threads other than the main thread are now taking the bulk of the CPU load relating to convoy route searching. Also, I tested it with one particular map, which was most unresponsive without multi-threading of the convoy route searching, but which became noticeably more responsive when the multi-threading code for this was enabled.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4624
  • Total likes: 168
  • Helpful: 107
  • Languages: EN, NO
Re: Cross-compiling: Linux to Windows
« Reply #77 on: February 06, 2017, 06:00:36 PM »
Pthread is of course installed independently from MinGW on the server in order to compile the Linux binaries, but presumably this would not put an extra subdirectory into the MinGW directory?

In any event, libwinpthread-1.dll is not in the /usr/lib/gcc/i686-w64-mingw32/5.3-win32 or the /usr/lib/gcc/i686-w64-mingw32/5.3-posix directories, either.

Edit 4: The output of g++ -v is

Code: [Select]
$ g++ -v
Using built-in specs.
COLLECT_GCC=C:\msys32\mingw32\bin\g++.exe
COLLECT_LTO_WRAPPER=C:/msys32/mingw32/bin/../lib/gcc/i686-w64-mingw32/6.3.0/lto-wrapper.exe
Target: i686-w64-mingw32
Configured with: ../gcc-6.3.0/configure --prefix=/mingw32 --with-local-prefix=/mingw32/local --build=i686-w64-mingw32 --host=i686-w64-mingw32 --target=i686-w64-mingw32 --with-native-system-header-dir=/mingw32/i686-w64-mingw32/include --libexecdir=/mingw32/lib --enable-bootstrap --with-arch=i686 --with-tune=generic --enable-languages=c,lto,c++,objc,obj-c++,fortran,ada --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw32 --with-mpfr=/mingw32 --with-mpc=/mingw32 --with-isl=/mingw32 --with-pkgversion='Rev1, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld --disable-sjlj-exceptions --with-dwarf2
Thread model: posix
gcc version 6.3.0 (Rev1, Built by MSYS2 project)

Can you provide the same for the cross-compiler?

Code: [Select]
marker_t::~marker_t()
{
    if (this)
    {
        delete[] bits;
    }
}

That if is unnecessary. delete does such a check implicitly.

marker_t::the_instance is a static, thread_local variable, not created with the new keyword: surely, the C++ standard is violated if it is even possible to have a segmentation fault/access violation on the automatic deletion of such an object on the termination of a thread?

marker_t in standard Simutrans doesn't have any guards against copying it, so while the marker_t instance may be thread local, maybe the bits inside it has been inadvertently shared?

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #78 on: February 06, 2017, 09:47:09 PM »
Thank you for your reply: that is most helpful. The output of the command for the cross-compiler is:

Code: [Select]
# i686-w64-mingw32-g++ -v
Using built-in specs.
COLLECT_GCC=i686-w64-mingw32-g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/i686-w64-mingw32/5.3-win32/lto-wrapper
Target: i686-w64-mingw32
Configured with: ../../src/configure --build=x86_64-linux-gnu --prefix=/usr --includedir='/usr/include' --mandir='/usr/share/man' --infodir='/usr/share/info' --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libexecdir='/usr/lib/gcc-mingw-w64' --disable-maintainer-mode --disable-dependency-tracking --prefix=/usr --enable-shared --enable-static --disable-multilib --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --libdir=/usr/lib --enable-libstdcxx-time=yes --with-tune=generic --enable-version-specific-runtime-libs --enable-fully-dynamic-string --enable-libgomp --enable-languages=c,c++,fortran,objc,obj-c++ --enable-lto --with-plugin-ld --enable-threads=win32 --program-suffix=-win32 --program-prefix=i686-w64-mingw32- --target=i686-w64-mingw32 --with-as=/usr/bin/i686-w64-mingw32-as --with-ld=/usr/bin/i686-w64-mingw32-ld --disable-libatomic
Thread model: win32
gcc version 5.3.1 20160211 (GCC)

I see that the "thread model" is different: win32 for the cross-compiler compared to posix for the Windows version of MinGW with Msys.

As to marker_t's destructor, I did work out in the end that the if(this) achieved nothing that is not implicit in any event.

In relation to copying marker_t, I have not, I do not think, done anything in particular with marker_t in Experimental, other than make it thread local, that is not already done in Standard. I have not much altered vehicle routing - at least the fundamentals of it, although I have added checks for things such as bridges for which the convoy is too heavy and added an alternative to standard relaxation of the route for the private car destination checker in which it will continue to find all possible destinations from the starting point, but neither of those should affect marker_t - this would also not explain why it would work on a native Linux GCC and a Visual Studio build on Windows but not on a MinGW build, which is most odd.

I am wondering whether the following might help: assuming this to be a MinGW or pthreads error, to replace the thread_local declaration for marker_t with an array of marker_t objects indexed by a unique number assigned to each thread: although this number itself is thread local, it already exists for other purposes (indexing the array of convoys to process in any given step) and does not fail; in any event, this would be an inherent type and would not have a destructor. Does this seem sane?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 8718
  • Total likes: 303
  • Helpful: 228
  • Languages: De,EN,JP
Re: Cross-compiling: Linux to Windows
« Reply #79 on: February 07, 2017, 03:32:45 AM »
I think that you are looking at the code on the master branch, which is the old 11.32 release build from 2013, before multi-threading was implemented. The multi-threaded code is on the devel-new-2 branch, and there is no call to unmarkiere_alle(), whereas there is at line 222 of route.cc on the master branch.
That is confusing; you have tags for releases for that purpose. But thanks for the hint.

Quote
The routing does not run across steps: in other words, calc_route() will not return until a complete route has been found. (I thought that this is how it worked in Standard, too). This means that it is not possible for one vehicle to find a route in a different step on one machine than on another. The thread_step() method is called just before the single threaded step() method and called on the same schedule as step() rather than sync_step(). The idea is that multiple convoys can use the route search simultaneously in parallel to find their routes, reducing the time that the rest of the program has to wait for the route searching of all the convoys that need it in a given step to complete. The route finding is not run concurrently with any function that could change the state of the world (e.g. player interaction). There are some instances of route searching still called from the main thread, such as those used with choose signals, and these run single threadedly.

Then you will get almost nothing from threading. Instead the overhead from threading likely make any threaded  route search slower than single threaded searches (starting a new thread and scheduling etc. for typical routes of less than 100 tiles assumed.) If you are worried about ships, I would rather introduced caching of routes and only recalculate them when water was added/removed, and share routes across lines. So a route has to be recalculated only once per line (for ships and airplanes).

But then, I never found route search very high on the profiling list, so I never worked on that. Especially after Dwachs improved ship route finder.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4624
  • Total likes: 168
  • Helpful: 107
  • Languages: EN, NO
Re: Cross-compiling: Linux to Windows
« Reply #80 on: February 07, 2017, 07:05:05 AM »
I see that the "thread model" is different: win32 for the cross-compiler compared to posix for the Windows version of MinGW with Msys.

That is the same as I have. MSYS2 comes with a compiler suite that is based on and includes (some variant of) pthreads. This is to support the multithread features in C++11. However, the cross-compiler does not include pthreads, so the multithreading features in the C++ library are either missing or weakened.

The routing does not run across steps: in other words, calc_route() will not return until a complete route has been found. (I thought that this is how it worked in Standard, too). This means that it is not possible for one vehicle to find a route in a different step on one machine than on another.

I seem to remember noticing that route calculation will only run for a certain number of tiles per step. That means that calc_route() may return before a route has been found, but it will take the same number of steps on all machines, keeping them in sync. But it has been long since I've looked into routing, and even then it was only briefly. (My contributions have been in the code interacting with the OS.)

Then you will get almost nothing from threading. Instead the overhead from threading likely make any threaded  route search slower than single threaded searches (starting a new thread and scheduling etc. for typical routes of less than 100 tiles assumed.) If you are worried about ships, I would rather introduced caching of routes and only recalculate them when water was added/removed, and share routes across lines. So a route has to be recalculated only once per line (for ships and airplanes).

The typical solution is to have a pool of threads standing by, so that you don't have to start new threads all the time. Doesn't the multithreaded rendering do that? There is still scheduling and synchronization, though. And wasn't the whole point of Experimental to do much more advanced routing than Standard? Maybe that only was routing the cargo, not the vehicles.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 8718
  • Total likes: 303
  • Helpful: 228
  • Languages: De,EN,JP
Re: Cross-compiling: Linux to Windows
« Reply #81 on: February 07, 2017, 09:06:27 AM »
So far the only extra thing experimental does, is looking for additional restrictions (weight, third rail).

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #82 on: February 07, 2017, 10:51:56 AM »
I should note that Simutrans-Experimental does use a pool of threads for each threaded task, so there is no overhead associated with starting threads. As to the proportion of CPU time spent routing, in a 2047,1663 map about half of which is water and with numerous ships, taken from an old game played on the online server (the map being well developed), the convoy route finding took 5.85% of the CPU time (multi-threaded). This compares to 4.96% for the path explorer (goods/passenger routing), 8.25% for the display_region_thread and 15.42% for karte_t::step (single threaded), convoi_t::step (the single threaded part with the routing removed) taking 1.99%.

I imagine that in maps of 256x256 size, there is little to be gained from multi-therading convoy route-finding, but in very large maps such as this with many ships, it does make a real difference.

Do either of you have any idea how to obtain the correct .dll file to work with the cross-compiled build as now statically linked? The cross-compiled build could be run on Windows before it was statically linked, but it would crash on loading saved games - just as the Msys2 build now does in any event.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4624
  • Total likes: 168
  • Helpful: 107
  • Languages: EN, NO
Re: Cross-compiling: Linux to Windows
« Reply #83 on: February 07, 2017, 05:01:06 PM »
You got the link library (.a file) from somewhere. That same place must also provide the corresponding DLL.

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #84 on: February 07, 2017, 10:56:48 PM »
There is a libpthread.a in /usr/i686-w64-mingw32/lib. There is also /usr/share/mxe/mxe/usr/i686-w64-mingw32.static/lib/libpthread.a. There is /usr/lib32/libpthread.a
, which I assume is the native Linux version. Then there are these:

Code: [Select]
/usr/i686-w64-mingw32/lib/libpthread.a
/usr/i686-w64-mingw32/lib/libpthread.a.old
/usr/i686-w64-mingw32/lib/libpthread.dll.a
/usr/i686-w64-mingw32/lib/libwinpthread-1.dll
/usr/i686-w64-mingw32/lib/libwinpthread.a
/usr/i686-w64-mingw32/lib/libwinpthread.dll.a

None of these have the file name that the Windows executable seems to want. The last set seem the most promising, and are consistent with what the Windows version of MinGW (that supplied by Msys requests), but it is a file of a different name (as reported above) that the error message complains is missing when I link statically.

Edit: I am having some very bizarre and quite incomprehensible trouble: as noted in this post in this thread, I had been able to get cross-compiling to work (subject to the crashes on loading issue) by putting these .dll files in my simutrans directory until I made this change to the makefile to attempt static linking (which in the end did not statically link pthreads). Now, I have reverted that with this commit, but the same failure as I had after static linking persists for no apparent reason, despite me having run the make clean command before producing the build. I am extremely confused: I really do not understand what is happening or how to deal with it as there is no consistent behaviour at all.

Edit 2: I have managed to find what seems to be an official MinGW/pthread example of pthreadGC2-w32.dll. Putting this in the simutrans directory allows the cross-compiled executable to run. However, I now get a new sort of failure on loading saved games: freezes instead of crashes, and these freezes occur much earlier in the loading process than the crashes: immediately after selecting the saved game in many cases. The behaviour is reminiscent of a thread deadlock (there is one which very occasionally occurs in the Visual Studio build on loading saved games and which I have been unable to eradicate despite working at it for many hours), but it is hard to tell whether this is actually what is happening because I cannot debug these builds, so the failure could be something else entirely.

I have also tried unsuccessfully to link pthreads statically using the instruction here, and, oddly, the resulting binary now requires more .dlls relating to pthreads than it did before (all of libwinpthread-1.dll, pthreadGC2-w32.dll and libgcc_s_dw2-1.dll). Static linking of pthreads would at least eliminate the problems caused by not being sure of having the right version, but this seems to be fantastically difficult to achieve.

Edit 3: Running the cross-compiled version in the Windows MinGW debugger, it seems that, despite the difference in observed behaviour (freezing rather than crashing to desktop), the actual cause of the problem with this build is the same, namely a segmentation fault in the destructor of marker_t.

Edit 4: I think that I have managed to make a version that works. using the pthreadGC2-w32.dll from edit 2 above, reverting the static linking as suggested by, I think, Ters earlier in this thread, and by replacing the thread local static marker with a static array of markers indexed by thread local thread numbers, a version compiled on my Linux server can now run on my Windows machine, repeatedly load games and connect to the Linux server without crashing. However, I now seem to get a desync, albeit only after a period in excess of 10 minutes. It is not clear whether this is specific to the cross-compiled version or what is causing this, and I am checking with the Windows MinGW build is any better in this respect.

Edit 5: The nightly builds, together with the necessary .dll file, can now be found here.
« Last Edit: February 08, 2017, 05:11:55 PM by jamespetts »
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 8718
  • Total likes: 303
  • Helpful: 228
  • Languages: De,EN,JP
Re: Cross-compiling: Linux to Windows
« Reply #85 on: February 11, 2017, 11:54:07 AM »
I had an entire day "fun" with mingw64. You *must* use the posix threads, winpthreads will crash during reading. Aparently the mingw64 runtime messes up the winpthreads (despite other documentation).

And you must built bzlib and whatever else you need with the same compiler!

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #86 on: February 11, 2017, 12:00:29 PM »
Thank you for that. I am not quite sure how to choose posix threads over and above winpthreads, but the build that I have seems to work now, save for the desync, which further testing has revealed is in any event common to the pure Linux GCC build.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline HaydenRead

Re: Cross-compiling: Linux to Windows
« Reply #87 on: February 12, 2017, 09:09:31 AM »
Did the 'Simutrans Ex' 'devel-new-2' build using 'SDL2' for linux, and windows on the box which I use for compiling Steam builds, and the only change I had to make was to remove the '-static' flag from the Makefile mingw section, and add '#include <cstdio>' to sdl_sound.cc (it uses printf which is declared in cstdio - for some reason it compiles linux build fine with or without <ctsdio>, but not for windows using mingw on linux).

Also will take a look at Mac OSX build (my Mac OSX build is done on a server that charges by the hour, so I don't do Mac builds as often).

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4624
  • Total likes: 168
  • Helpful: 107
  • Languages: EN, NO
Re: Cross-compiling: Linux to Windows
« Reply #88 on: February 12, 2017, 10:15:38 AM »
(it uses printf which is declared in cstdio - for some reason it compiles linux build fine with or without <ctsdio>, but not for windows using mingw on linux).

It is not uncommon to find that when trying out a different compiler suite, even different versions of the same, one suddenly has to include an std... header file (for me, usually stdlib.h). In hindsight, it was always required according to the docs/spec, but these headers files a very often included by other header files, and you had somehow come to depend on it. I'm not sure what the standard says about its header files including each other. However, it might also be 3rd party headers that conditionally include headers depending on platform (or other stuff).

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #89 on: March 21, 2017, 01:10:37 AM »
Having got this to work, I am now encountering a fresh problem with makoebj (but, oddly, not the main Simutrans-Extended executable). I am now getting the following error when cross-compiling:

Code: [Select]
===> LD  ../build/mingw/makeobj-extended/makeobj-extended
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x32): undefined reference to `png_create_read_struct'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x45): undefined reference to `png_create_info_struct'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x7d): undefined reference to `png_init_io'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x8f): undefined reference to `png_read_info'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0xd5): undefined reference to `png_get_IHDR'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x10c): undefined reference to `png_set_strip_16'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x117): undefined reference to `png_set_packing'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x147): undefined reference to `png_read_update_info'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x159): undefined reference to `png_get_rowbytes'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x1a8): undefined reference to `png_read_image'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x235): undefined reference to `png_read_end'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x24f): undefined reference to `png_destroy_read_struct'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x262): undefined reference to `png_set_expand'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x26d): undefined reference to `png_set_strip_alpha'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x288): undefined reference to `png_set_filler'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x298): undefined reference to `png_set_swap_alpha'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x2a3): undefined reference to `png_set_invert_alpha'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x2e9): undefined reference to `png_destroy_read_struct'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x4f2): undefined reference to `png_create_write_struct'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x505): undefined reference to `png_create_info_struct'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x53d): undefined reference to `png_init_io'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x550): undefined reference to `png_set_compression_level'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x598): undefined reference to `png_set_IHDR'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x5aa): undefined reference to `png_write_info'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x5cf): undefined reference to `png_set_filler'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x5fc): undefined reference to `png_write_row'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x615): undefined reference to `png_write_end'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x627): undefined reference to `png_destroy_write_struct'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x64a): undefined reference to `png_destroy_write_struct'
../build/mingw/utils/dr_rdpng-makeobj-extended.o:dr_rdpng.cc:(.text+0x6ab): undefined reference to `png_destroy_write_struct'
collect2: error: ld returned 1 exit status
../uncommon.mk:22: recipe for target '../build/mingw/makeobj-extended/makeobj-extended' failed
make: *** [../build/mingw/makeobj-extended/makeobj-extended] Error 1

The normal Linux build works, as well as the Windows Visual Studio build. This had been working until recently, but I have of late been merging in some of the latest changes from Standard.

This looks like a linker error relating to the PNG libraries, but I do not really understand what has changed such that my previous setup now no longer works. If anyone could suggest where I might look to try to find the answer to this, I should be most grateful, as I find this all very opaque.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Vladki

Re: Cross-compiling: Linux to Windows
« Reply #90 on: April 12, 2017, 07:39:04 AM »
I'm no expert on cross compiling, but I suppose you had to prepare all libraries for cross compile somehow (libc, libbz,...). Did you do the same for libpng?




Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #91 on: April 12, 2017, 09:44:43 AM »
I did - and it initially worked, but do then stopped working after I merged some changes from Standard for no discernable reason.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 8718
  • Total likes: 303
  • Helpful: 228
  • Languages: De,EN,JP
Re: Cross-compiling: Linux to Windows
« Reply #92 on: April 12, 2017, 01:37:38 PM »
You likely have to define something to farce name mangling, because it does now regard the functions as extern.

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 15365
  • Total likes: 368
  • Helpful: 165
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Cross-compiling: Linux to Windows
« Reply #93 on: April 12, 2017, 09:33:47 PM »
I am afraid that I am not sure that I quite follow: what functions are now regarded as extern that were not before? Apologies if I am being a bit dim.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 8718
  • Total likes: 303
  • Helpful: 228
  • Languages: De,EN,JP
Re: Cross-compiling: Linux to Windows
« Reply #94 on: April 13, 2017, 07:05:16 AM »
functions are oten renamed "void doing(int,int)" from "_doing" for extern functuion usual in a library to "@Vdoing22"to check types for function comiled within the same program. So if the extern somehow is missing, then the names are decorated (mangled), which is a problem for libaries. However, if you are using the same compiler for both library and code, then mangling may be on.