compiling on osx

[jamespetts]

You could always merge it manually: there doesn't seem to be that much code to merge. I must confess, I have never quite got the hang of cherry picking.

[dannyman]

ojii: change freight_list_sorter.cc to include malloc/malloc.h

Will try fetch, cherry-pick, compile later, family obligations permitting . . .

-d

[ojii]

dannyman: tried that, now it fails with:

https://github.com/jamespetts/simutrans-experimental/pull/7

(again, add .diff or .patch if you want to pull the changes without git)

[MCollett]

ojii: change freight_list_sorter.cc to include malloc/malloc.h

I commented out the include altogether, and changed the call to malloc from
wlist = (ware_t*) malloc(warray_size);
to
wlist = MALLOCN(ware_t,warray_size);

Best wishes,
Matthew

[dannyman]

Thanks for the malloc, Matthew!

One thing I did try, which didn't seem to help, was to steal the pthread_barrier_t from http://blog.albertarmea.com/post/47089939939/using-pthread-barrier-on-mac-os-x and put it in threading.h, and then try and declare the functions from threading.cc ... but I'm getting error like ojii where the pthread_barrier_t and the related functions aren't found.  So ... ??!!

.. not tonight ..

-danny

[jamespetts]

Thank you everyone for working on this - it is much appreciated.

I commented out the include altogether, and changed the call to malloc from
wlist = (ware_t*) malloc(warray_size);
to
wlist = MALLOCN(ware_t,warray_size);

Best wishes,
Matthew

If there are going to be substantial changes to the code like this to allow compilation for one particular platform, they should be behind conditional compilation directives so as not to affect code that works on other platforms.

[Milko]

Hello James and ojii

Ojii, removing #include "malloc.h" (in freight_list_sorter.cc) broke mingw compilation (I see your branch)

To obtain a conditional compilation usable for MingW and Mac Os I suggest to add these lines:

#ifdef __MINGW32__
#include "malloc.h"
#endif

#ifdef __APPLE__
#include "malloc/malloc.h"
#endif

I have already done the test with Mingw and it's all ok, Ojii if you manage to fill in for Mac os you're done, James can use those lines directly in the branch 11.x

Giuseppe

[jamespetts]

Thank you - that is very helpful. Can somebody test on Mac OS X?

[ojii]

@Milko: Sorry for breaking MinGW. Pushed a fix to my branch.

Still doesn't compile though. pthread is not happy.

@James: maybe we should move this thread to the development subsection?

[dannyman]

Ooooooookay.

I have a build.  I think it is a clean, portable build -- see the diff and the pthread_barrier attachments ...

... but it segfaults.

Code Select Expand


0-10:55 dannhowa@dannhowa-mac simutrans$ ./sim
Use work dir /Users/dannhowa/Dropbox/simutrans/
Reading low level config data ...
Parsed simuconf.tab for directory layout; multiuser = 1
parse_simuconf() in program dir (config/simuconf.tab): Reading simuconf.tab successful!
Preparing display ...
SDL_driver=Quartz, hw_available=0, video_mem=0, blit_sw=0, bpp=32, bytes=4
Screen Flags: requested=10, actual=10
dr_os_open(SDL): SDL realized screen size width=704, height=560 (requested w=704, h=560)Loading font 'font/prop.fnt'
font/prop.fnt successfully loaded as old format prop font!
Segmentation fault: 11

Crying baby ... ciao .. :/

[jamespetts]

Can you get a backtrace?

[dannyman]

jamespetts,

Does the crash_report.txt attached to my previous post suffice?  (I'm new to Mac..)

It looks like, yup, it killed itself when trying to do pthread barriers at the first attempt to draw the screen.  :/

-danny

[jamespetts]

Hmm, I don't think that that's an actual backtrace. Sadly, I don't have enough knowledge of either Macs or pthreads to assist with the details of this...

[prissi]

The part

Code Select Expand


0x00000001023ed838 calc_base_pal_from_night_shift(int) + 216
0x00000001023f728d simgraph_init(short, short, int) + 541
0x000000010240d6c7 simu_main(int, char**) + 4455
0x000000010241c075 sysmain(int, char**) + 117

looks very well like a backtrace to me. Although a crash there seems strange as this code most likely is identical to the standard version.

[jamespetts]

Hmm - very odd indeed.

[jamespetts]

I have now released version 11.2 - would anybody be able to compile a Mac OS X version of this (even if only single threaded)? I should be most grateful.

[Milko]

Hello ojii

@Milko: Sorry for breaking MinGW. Pushed a fix to my branch.

No problem, I'm happy to see compatibility with many operating systems and many compilers  . thanks for your help.

Giuseppe

[MCollett]

I have now released version 11.2 - would anybody be able to compile a Mac OS X version of this (even if only single threaded)? I should be most grateful.

Done.

freight_list_sorter.cc still needs the malloc include to be ifdefed to MingW only.

Best wishes,
Matthew

[jamespetts]

Thank you very much. Does anyone have any idea how to produce the right set of preprocessor directives to compile on Mac OS X, GCC (Linux), GCC (MinGW) and MSVC? It would be most helpful.

[ojii]

managed to get it to compile (with multithreading) but now it fails to link:

===> LD  build/default/simutrans-experimental
Undefined symbols for architecture x86_64:
  "barrier_init(pthread_barrier_t*, int)", referenced from:
      loadsave_t::set_buffered(bool) in loadsave.o
      karte_ansicht_t::display(bool) in simview.o
      dr_os_open(int, int, int) in simsys_s.o
  "pthread_barrier_wait(pthread_barrier_t*)", referenced from:
      loadsave_thread(void*) in loadsave.o
      loadsave_t::set_buffered(bool) in loadsave.o
      loadsave_t::read(void*, unsigned long) in loadsave.o
      loadsave_t::write(void const*, unsigned long) in loadsave.o
      display_region_thread(void*) in simview.o
      karte_ansicht_t::display(bool) in simview.o
      redraw_thread(void*) in simsys_s.o
      ...
  "pthread_barrier_destroy(pthread_barrier_t*)", referenced from:
      loadsave_t::set_buffered(bool) in loadsave.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [build/default/simutrans-experimental] Error 1

code: https://github.com/ojii/simutrans-experimental/tree/11.0-osx-multithreading-experiments

note that i removed some conditionals in that branch (because i wanted to rule out that they're causing it), gonna clean up everything once it compiles.

[jamespetts]

Thank you for making progress on this - most helpful! These linker errors might be caused by you not having all the relevant pthreads (devel) packages installed on your system: have you checked for that?

[ojii]

the symbols it can't find are defined in utils/threading.h (it's the ones missing from pthread on osx).

[jamespetts]

Yes, they might be declared in the header file, but where is the .cc file or .lib file that defines them? Linker errors like this are caused when methods are declared in a header file, but the actual body of the method cannot be found anywhere (either as source code in a .cc file, or as a pre-compiled library in a .lib or .dll).

[MCollett]

Here is an unthreaded version of 11.3 for Mac OS X:
http://simutrans-germany.com/files/upload/simutrans-experimental.11.3.zip

Best wishes,
Matthew

[ojii]

Yes, they might be declared in the header file, but where is the .cc file or .lib file that defines them? Linker errors like this are caused when methods are declared in a header file, but the actual body of the method cannot be found anywhere (either as source code in a .cc file, or as a pre-compiled library in a .lib or .dll).

headers: https://github.com/ojii/simutrans-experimental/blob/11.0-osx-multithreading-experiments/utils/threading.h
code: https://github.com/ojii/simutrans-experimental/blob/11.0-osx-multithreading-experiments/utils/threading.cc

Note that I'm really not familiar with C++ and this is copy/pasted from all around the internet and then slightly modified by me.

Do I need to modify a Makefile so it compiles threading.cc? YES I do.


Found the issue so far, going to keep digging.

[ojii]

Pro tip: Don't use C++ code snippets from two different sources mixed.

Got it to compile (and link) now, but the generated binary locks up after showing the 'start menu' (new game, load game, ...). I bet it's again because of the weird way I "patched" the game and it causes some sort of threading deadlock (in pthread_barrier_... I'd assume). Will look further into it.

The code is in the branch mentioned above (https://github.com/ojii/simutrans-experimental/tree/11.0-osx-multithreading-experiments) if anyone wants to have a look/help out.

[ojii]

I'm out of ideas. The barrier code makes sense as far as I can see. How do people usually debug C++ applications that just "hang"?

If anyone wants the code without having to mess with git, link to tarball: https://github.com/ojii/simutrans-experimental/tarball/11.0-osx-multithreading-experiments

[jamespetts]

What I normally do to debug hangs is to pause them in MSVC's debugger and see which bits of code  are being executed at the time. I don't know whether that is possible on a Mac, however.

[ojii]

I assume MSVC's debugger is something similar to GDB.

I'm a real noob when it comes to this, but from poking around a few minutes I found that it halts on __psynch_mutexwait, so this is definitely a threading deadlock. Unless someone with more experience has a look at this I'll try to ask some friends more familiar with GDB/C++ and see if I can find more information.

[jamespetts]

The MSVC debugger is in some ways more sophisticated than GDB, as it will show a full stack trace, and one can select all the calls (up to a certain number) that got the programme to where it is, and see which lines in the code that those calls were, and see the values of all the variables in that part of the code, as well as step the code one line at a time manually to see which path that the code follows. One can also edit any variable that is currently on the stack manually. Diagnosing hangs is much easier when one can step the code manually in that way (if it is a threading deadlock, of course, one would not be able to step it: but the fact that it would refuse to step at all would show that it was a threading deadlock rather than an infinite loop).

[prissi]

You can do all that in gdb too, if zou have a debug built. A GUI for gbd can be also very helpful. There are DDD and insight, and various IDE: http://sourceware.org/gdb/wiki/GDB%20Front%20Ends

But a hang on a mutex is something which is difficult to find out with a debugger, since the error is usually in the other thread.

[ojii]

I started spamming dbg messages all over, and actually the main menu does NOT hang (it's just extremely unresponsive), but I can get to the "new game" menu. That's where I get stuck but it might again be that it's just too unresponsive. So this pthread_barriers implementation might just be too slow (too much overhead) to actually work.

I'll need to somehow get debug symbols for the main pthread stuff to get further. I wish I was on linux for this...

[jamespetts]

Version 11.4 is out - would anyone mind compiling at least a single-threaded version of this? I should, as ever, be very grateful.

[dannyman]

MCollett what build settings do you use for single-threaded?  My compiles come out slow as molasses . . .  :/

I've got sdl, color depth 16 and optimise = 1

FLAGS = -I/usr/include

-d

[MCollett]

Version 11.4 is out - would anyone mind compiling at least a single-threaded version of this? I should, as ever, be very grateful.

http://simutrans-germany.com/files/upload/simutrans-experimental.11.4.zip

Best wishes,
Matthew