News:

SimuTranslator
Make Simutrans speak your language.

Possible memory leak

Started by Rollmaterial, February 10, 2020, 09:43:21 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Mariculous

To be more precise:
We don't need to know the routes, we need to know the existence of such a route and its estimated journey time.
Calculating the routes and throwing the results away has not been an issue in the past.
However, I'd really prefer storing and using that already calculated data in an efficient way instead of throwing it away and re-calculating it whenever a car is spawned.

(updated the previous post a few minutes before yours)

Phystam

In a larger map, the frequency of re-calculation is not so high. This fact causes the less response for a newly constructed bypass route. Could you add a priority for such new road? That may help the response.

And one suggestion. I found that the response becomes high when I set by_a_step=0, but it causes less fps. If players can easily switch the mode with a button until finishing calculations for entire route, probably that is helpful for the response.

Matthew

Quote from: jamespetts on March 04, 2020, 06:44:04 PM
Yes,. you have correctly understood what I have asked to be tested: thank you for this.

However, the error given is rather mystifying: I do not understand how it is possible for a thread barrier to raise an arithmetic exception, and I have not been able to reproduce this myself.

I have noticed a thread scheduling problem that seems to be causing thread deadlocks in certain situations, however, so it is possible that these are somehow related.

Can I check whether anyone else is experiencing this issue when testing this branch? It might be helpful to run the test multiple times, as thread errors can be non-constant.

I have run the test multiple times and rebuilt from the same commit. I get a crash with exactly the same error message in gdb every time when I run the British and Japanese paksets. It sometimes appears in the terminal as a "floating point exception" rather than an "arithmetic exception". Interestingly, I can run the Swedish pakset, perhaps because it does not appear to have any private cars.

I can't respond to your technical points except by wishing you good luck when you feel up to revisiting the branch, James. IIRC you find threading bugs particularly difficult. I appreciate your commitment to making Extended run on a wide variety of systems, from my potato maps up to Rollmaterials' Rolls-Royce games.
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

jamespetts

Quote from: Matthew on March 09, 2020, 05:38:34 PM
I have run the test multiple times and rebuilt from the same commit. I get a crash with exactly the same error message in gdb every time when I run the British and Japanese paksets. It sometimes appears in the terminal as a "floating point exception" rather than an "arithmetic exception". Interestingly, I can run the Swedish pakset, perhaps because it does not appear to have any private cars.

I can't respond to your technical points except by wishing you good luck when you feel up to revisiting the branch, James. IIRC you find threading bugs particularly difficult. I appreciate your commitment to making Extended run on a wide variety of systems, from my potato maps up to Rollmaterials' Rolls-Royce games.


Thank you for confirming the consistency of reproduction. I will have to investigate this in detail whenever I have a large amount of time available that I can dedicate to this - I will not be able to incorporate the memory consumption fix without fixing the thread scheduling errors, but this is likely to take a very large amount of work, so I will not be able to start until I have a significant amount of clear time.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

I have realised that there was a change that I had not committed to Github; I am not sure whether this will make any difference, but any re-testing of this would be helpful.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Matthew

Quote from: jamespetts on March 09, 2020, 09:19:26 PM
I have realised that there was a change that I had not committed to Github; I am not sure whether this will make any difference, but any re-testing of this would be helpful.

I'm happy to test, but sadly tonight's commits do not make any difference; I get an "arithmetic exception" or "floating point exception" as before. I also got that crash after loading the Swedish pakset and sometimes (but not always) got crashes when quitting from Sweden.

By the way, when I opened this branch in VS Code, it gave me a warning at lines 2834 and 2904 of simworld.cc: class "karte_t" has no member "private_car_route_mutex". I've never seen such a message when looking at Simutrans before, but I don't know enough to say whether it's significant or helpful. GCC always logs thousands of warnings and Simutrans runs happily nonetheless!  8)

No one expects any quick results from such a difficult bug, but hopefully something in the feedback is helpful when you (James) have more time.
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

jamespetts

Thank you for this. I have pushed what I believe is a fix to this: I should be grateful if you could re-test.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

I have tracked down and fixed some more thread deadlock issues. I should be grateful if anyone who is able to compile the code could test this, as I do need to focus on this fix before moving onto other things if at all possible.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Matthew

At the first attempt, I still get the same floating-point exception.

I notice that there is now a /zstd subdirectory in the source files. Mine is empty. Does anyone (particularly on Linux) have anything in that subdirectory?
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

jamespetts

Thank you for checking this. Can I check precisely what you are compiling and in what environment? When I try to compile this on my Linux computer, I get the following error:


simworld.cc: In member function 'void karte_t::suspend_private_car_threads()':
simworld.cc:1941:29: error: could not convert 'karte_t::private_car_route_mutex' from 'pthread_mutex_t' to 'bool'
  if (private_car_route_mutex)
                             ^
simworld.cc:1951:29: error: could not convert 'karte_t::private_car_route_mutex' from 'pthread_mutex_t' to 'bool'
  if (private_car_route_mutex)
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Spenk009

#45
I am getting the same error in simworld.cc (lines 1941 & 1951), when building from the most recent github release on "upstream/private-car-routing-memory-saving".

Matthew

Quote from: jamespetts on March 18, 2020, 07:11:55 PM
Thank you for checking this. Can I check precisely what you are compiling and in what environment? When I try to compile this on my Linux computer, I get the following error:


simworld.cc: In member function 'void karte_t::suspend_private_car_threads()':
simworld.cc:1941:29: error: could not convert 'karte_t::private_car_route_mutex' from 'pthread_mutex_t' to 'bool'
  if (private_car_route_mutex)
                             ^
simworld.cc:1951:29: error: could not convert 'karte_t::private_car_route_mutex' from 'pthread_mutex_t' to 'bool'
  if (private_car_route_mutex)


The tl;dr is that I also can't build today. I can't think of any sensible reason why I was able to build yesterday, so the most likely explanation is that I was an idiot and failed to build cleanly (i.e. there was code from some other branch in the /build subdirectory). Apologies for wasting everyone's time.

But since you asked, here are my settings. I am using Ubuntu 18.04.3 on an ancient Intel Core i3 and this is not a debug build. I usually use the command make -j2 >pcrms.log 2>pcrms.log.

I just discovered GCC will report its version with gcc -v, so here is that info:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)


I am able to build and run jamespetts/simutrans-extended/master, so whatever I'm doing wrong seems to be specific to this branch.

With a clean build of #e0823a6, I now fail to compile with this (complete logfile on Github):

make: *** [build/default/simworld.o] Error 1
make: *** Waiting for unfinished jobs....
simtool.cc: In member function 'const char* tool_build_station_t::tool_station_aux(player_t*, koord3d, const building_desc_t*, waytype_t, const char*)':
simtool.cc:5187:30: warning: 'cost' may be used uninitialized in this function [-Wmaybe-uninitialized]
  sint64 adjusted_cost = cost * desc->get_x() * desc->get_y();


That message about "unfinished jobs" made me wonder whether it was worth trying compiling in a single thread (of GCC, not Simutrans, so I omit the -j2) and I get a slightly different but equally unhelpful error (logfile):

descriptor/../display/simgraph.h:130:15: warning: 'zoom_factor' defined but not used [-Wunused-variable]
static uint32 zoom_factor = ZOOM_NEUTRAL;
               ^~~~~~~~~~~
common.mk:50: recipe for target 'build/default/simworld.o' failed
make: *** [build/default/simworld.o] Error 1


The last semi-functioning (Swedish pakset only) build I can find was #32b7ac7.

I am still puzzled about why I have that empty /zstd subdirectory though, since the Ubuntu file manager seems to think it was created on Tuesday.
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

Phystam

I have the same problem in a Linux build using g++5.4.0,  but there is no problem in windows cross-compile build using mingw64 g++.
I cannot understand this behaviour; probably it is related to the internal implementation of the pthread library...?

jamespetts

I believe that I have fixed the compile error with GCC based compilers: I should be grateful if you could re-test.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Spenk009

It now successfully compiles and runs on my machine without stutters.

One thing I did notice is that at some point in the game all trees disappeared. Later on, I saved and the save does not load any more. If you're interested, a copy can be found here

jamespetts

Thank you for testing that: that is most helpful. I have also been able to compile this using GDB. However, I am now getting an arithmetic exception relating to thread barriers:


Thread 1 "simutrans-exten" received signal SIGFPE, Arithmetic exception.
__pthread_barrier_wait (barrier=0x555555cf0f80 <karte_t::private_car_barrier>) at pthread_barrier_wait.c:117
117 pthread_barrier_wait.c: No such file or directory.
(gdb) backtrace
#0  __pthread_barrier_wait (barrier=0x555555cf0f80 <karte_t::private_car_barrier>) at pthread_barrier_wait.c:117
#1  0x0000555555924bac in karte_t::suspend_private_car_threads() ()
#2  0x0000555555947b7f in karte_t::load(char const*) ()
#3  0x00005555558dbfbf in simu_main(int, char**) ()
#4  0x00005555558f23e2 in sysmain(int, char**) ()
#5  0x00007ffff6541b97 in __libc_start_main (main=0x5555555ab1d0 <main>, argc=1, argv=0x7fffffffdf88, init=<optimised out>,
    fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7fffffffdf78) at ../csu/libc-start.c:310
#6  0x00005555555ab23a in _start ()
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

#51
Further to the above, I believe that I have now fixed this arithmetic exception error - I should be grateful if people could re-test to confirm that this is now working.
Edit: I have also fixed a thread deadlock issue.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Matthew

Quote from: jamespetts on March 21, 2020, 03:35:32 PM
Further to the above, I believe that I have now fixed this arithmetic exception error - I should be grateful if people could re-test to confirm that this is now working.
Edit: I have also fixed a thread deadlock issue.

I can now compile and run #d33d9dd in pak128.Britain-Ex. I haven't played enough to be sure of stability or performance, but that's definitely progress, so thank you for your work on this difficult bug, James!

EDIT: And as soon as I post I get a freeze, after about 3 hours of in-game time on a tiny map.  :::) GDB backtrace:

Thread 105 "simutrans-exten" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff9d7fa700 (LWP 16161)]
0x0000555555c7d6c0 in vtable for boden_t ()
(gdb) bt
#0  0x0000555555c7d6c0 in vtable for boden_t ()
#1  0x00005555556373a1 in route_t::find_route(karte_t*, koord3d, test_driver_t*, unsigned int, unsigned char, unsigned int, int, unsigned int, unsigned int, bool, route_t::find_route_flags) ()
#2  0x0000555555857c19 in stadt_t::check_all_private_car_routes() ()
#3  0x000055555591e78a in check_road_connexions_threaded(void*) ()
#4  0x00007ffff7bbd6db in start_thread (arg=0x7fff9d7fa700) at pthread_create.c:463
#5  0x00007ffff684888f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

jamespetts

Thank you for this report. Unfortunately, the backtrace does not give the line of code at which this error occurred (I suspect because this is an optimised build), so I cannot understand what happened to cause this problem.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

I have now merged this with the master branch, so this should be in to-morrow's nightly build. My apologies that this issue has taken so long to solve: it has required a very in-depth and sophisticated solution that needed a great deal of testing and debugging.

I should be grateful for any feedback on how private car routing is working after this fix.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Phystam

I had a problem in a server game.
after sending the save data to a client (or sending force-sync using nettool), the client game freezes. ( CPU still has high usage, so I think it is not a segfault.)

save data: https://cdn.discordapp.com/attachments/665986412641779712/691527284577927198/PHNS-SB.sve
open it in pak128.britain-ex.

jamespetts

I am afraid that I am not able to reproduce this: are you sure that this is not the delay of a few seconds whilst the route finding finishes working on the current city when the game is saved?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Phystam

Not a few seconds — freezes more than a minute, probably eternally.

jamespetts

Quote from: Phystam on March 24, 2020, 05:33:50 AM
Not a few seconds — freezes more than a minute, probably eternally.

Very odd - I still cannot reproduce this even with the version downloaded from the Bridgewater-Brunel server.

Can I ask what system that you are using?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

freddyhayward

I've attached a save that freezes or crashes almost immediately after being loaded, showing the same output on GDB as on larger saves I've tested with. The save always freezes when run with GDB, but without GDB it occasionally crashes rather than freezing. https://cdn.discordapp.com/attachments/692376966070009877/692923231882969139/unknown.png.
I'm on a linux machine and the freeze seems to stem from calling await_private_car_threads(). It's also possible that the internal linux code is the problem.

jamespetts

Quote from: freddyhayward on March 27, 2020, 09:56:33 PM
I've attached a save that freezes or crashes almost immediately after being loaded, showing the same output on GDB as on larger saves I've tested with. The save always freezes when run with GDB, but without GDB it occasionally crashes rather than freezing. https://cdn.discordapp.com/attachments/692376966070009877/692923231882969139/unknown.png.
I'm on a linux machine and the freeze seems to stem from calling await_private_car_threads(). It's also possible that the internal linux code is the problem.

Thank you for your report. I believe that I have fixed this; I should be grateful if you could re-test.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

freddyhayward

My previous post seems to have been deleted. Unfortunately the issue persists even under the latest revision.

jamespetts

Quote from: freddyhayward on March 29, 2020, 10:03:05 AM
My previous post seems to have been deleted. Unfortunately the issue persists even under the latest revision.

Thank you for letting me know Can I check whether you are self compiling or using the pre-compiled binaries from the Bridgewater-Brunel nightly build server?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

I have just tried this on my Linux machine and I am afraid that I cannot reproduce this. Can I check what commit date that you are using and whether you are using a pre-compiled binary or a self-compile?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Phystam

Still I have the same problem when loading.
How to produce:
1. Use the attached save data above
2. Start a server game with no window binary
3. Access to the server as a client
(May stop then)
4. Send a nettool force-sync command
(May stop then)

jamespetts

Quote from: Phystam on March 29, 2020, 09:23:34 PM
Still I have the same problem when loading.
How to produce:
1. Use the attached save data above
2. Start a server game with no window binary
3. Access to the server as a client
(May stop then)
4. Send a nettool force-sync command
(May stop then)

Is "the save data above" thread_test.sve?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

freddyhayward

Quote from: jamespetts on March 29, 2020, 12:19:08 PM
Thank you for letting me know Can I check whether you are self compiling or using the pre-compiled binaries from the Bridgewater-Brunel nightly build server?
I had previously used a self compiled build as the nightly build had not yet been uploaded. However I have just tried using the latest nightly build and still have the same problem. So far, the freezes have occurred with fast-forward on in--game. I am now testing to see whether this occurs without fast-forward.

Phystam

James — it is actually "my" save data.
And the issue was produced by the nightly build.

freddyhayward

Quote from: freddyhayward on March 29, 2020, 11:30:06 PM
I had previously used a self compiled build as the nightly build had not yet been uploaded. However I have just tried using the latest nightly build and still have the same problem. So far, the freezes have occurred with fast-forward on in--game. I am now testing to see whether this occurs without fast-forward.
I can now confirm that the freeze also occurs without using fast-forward.

jamespetts

Thank you both for the clarification. I am afraid that I am still unable to reproduce this, which will make it extremely difficult for me to fix this problem.

Can I ask that you try the following:

(1) compile a version with debugging symbols enabled;
(2) run it in GDB; and
(3) whenever the freeze occurs, halt the program with a backtrace?

That would be very helpful. Thank you.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.