very bad memory leak

[Vladki]

Hi, I was wondering for a few weeks why my servers (with pak.britain) keep crashing. There was nothing in the debug output (just that they got killed), and if it happened when I was connected, it just desynced the game... Now I managed to reproduce it. Run this save http://server.exp.simutrans.com/debug-saves/oom-crash.sve (commit baa9703), and wait for Nov 3:49:50 (you can fast forward to 3:48). Watch memory consumption and CPU usage with top or any other tool. It will grow from cca 1 GB to at maybe 10 GB (and cpu to 100%). My computer has 8 GB so it started to swap as hell. The server part gets killed at that moment. The client freezes for a few minutes, but it survived, and slowly recovers to normal. This is the debug output:

Code Select Expand


ERROR: karte_t::sync_step():    delta_t too large: 28622
For help with this error or to file a bug report please see the Simutrans forum:
http://forum.simutrans.com
ERROR: karte_t::sync_step():    delta_t too large: 17041
For help with this error or to file a bug report please see the Simutrans forum:
http://forum.simutrans.com
ERROR: karte_t::step(): delta_t (21799) out of bounds!
For help with this error or to file a bug report please see the Simutrans forum:
http://forum.simutrans.com
ERROR: karte_t::sync_step():    delta_t too large: 38759
For help with this error or to file a bug report please see the Simutrans forum:
http://forum.simutrans.com
ERROR: karte_t::step(): delta_t (12775) out of bounds!
For help with this error or to file a bug report please see the Simutrans forum:
http://forum.simutrans.com


I remember similar (but not that bad) freezes to happen on simutrans standard when the game wanted to expand industries, but could not find a suitable place - the map was full. But this game has a plenty of free space.

I think the same memory leak is is killing the bridgeater-brunel server.

The swedish game is not affected, probably because it is much smaller, not developed and the pakset is limited to trains. No road vehicles, no practical industry chains, only a few cityhouses.

[jamespetts]

I think that I may have fixed this, but it is hard to tell, as the problem is complex. The immediate problem was caused by a huge number of pedestrians being generated when a huge packet of passengers at one stop decided that it would be quicker to walk than take the train/'bus. However, this does not tell me what caused there to be such a huge number of passengers (many tens of thousands) in the first place. I have altered the pedestrian generation algorithm to set it to generate a maximum of 128 at a time, which should at least prevent the game from crashing or becoming unresponsive in this state, but that is not a fix for the problem itself. The errors that you are reporting in the log, incidentally, are likely to be an effect, rather than a cause, of the problem: these excessive delta_t instances occur when the game is bogged down.

I have also pushed a fix to what I think that the ultimate cause of the problem probably is, and this should now be available on the current nightly build. I think that this was a bizarre instance of extra passengers being generated on transfers due to an iterator fault, similarly to a near identical problem reported with goods some time ago. I have pushed a fix to this (which worked for goods), and some preliminary tests suggest that this may have worked, but it is hard to be sure without having a specific instance where the problem can be tracked down without doubt.

I should be grateful if you could let me know whether there appears to be any improvement with this.

[Vladki]

So far so good. Both servers at server.exp.simutrans.com survived for more than 24 hours.