The International Simutrans Forum

 

Author Topic: Heap corruption deleting members of objlist_t  (Read 2406 times)

0 Members and 1 Guest are viewing this topic.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18547
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Heap corruption deleting members of objlist_t
« on: December 11, 2017, 12:37:54 AM »
The Bridgewater-Brunel server has been crashing recently as a result of heap corruption. I am having trouble tracking down the ultimate cause, and any thoughts that anyone might have would be appreciated.

The backtrace for one of the crashes on the server was:

Code: [Select]
(gdb) backtrace
#0  __GI___libc_free (mem=0x800030300) at malloc.c:2951
#1  0x00000000004628a9 in strasse_t::~strasse_t() ()
#2  0x000000000046bb26 in objlist_t::~objlist_t() ()
#3  0x00000000004517e0 in boden_t::~boden_t() ()
#4  0x00000000006dcf17 in planquadrat_t::~planquadrat_t() ()
#5  0x0000000000718cc9 in karte_t::destroy() ()
#6  0x000000000072ce15 in karte_t::load(loadsave_t*) ()
#7  0x000000000073067e in karte_t::load(char const*) ()
#8  0x00000000005a385e in nwc_sync_t::do_command(karte_t*) ()
#9  0x0000000000725ae2 in karte_t::do_network_world_command(network_world_command_t*) ()
#10 0x000000000072618b in karte_t::process_network_commands(int*) ()
#11 0x000000000073d06f in karte_t::interactive(unsigned int) ()
#12 0x00000000006cd6c2 in simu_main(int, char**) ()
#13 0x00000000006e1807 in sysmain(int, char**) ()
#14 0x00007ffff6b45830 in __libc_start_main (main=0x40efb0 <main>, argc=21, argv=0x7fffffffe458, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7fffffffe448) at ../csu/libc-start.c:291
#15 0x000000000040f019 in _start ()

This seems to be able to be reproduced (albeit sporadically) on the clients, too, and Dr. Memory gives me the following output that seems relevant:

Code: [Select]
Error #1: INVALID HEAP ARGUMENT to free 0x08000303
# 0 replace_operator_delete_array                             [d:\drmemory_package\common\alloc_replace.c:2998]
# 1 minivec_tpl<>::~minivec_tpl<>                             [c:\users\james\documents\development\simutrans\simutrans-extended-sources\tpl\minivec_tpl.h:20]
# 2 strasse_t::~strasse_t       
# 3 strasse_t::`scalar deleting destructor'
# 4 objlist_t::~objlist_t                                     [c:\users\james\documents\development\simutrans\simutrans-extended-sources\dataobj\objlist.cc:172]
# 5 grund_t::~grund_t                                         [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\grund.cc:489]
# 6 boden_t::~boden_t           
# 7 boden_t::`scalar deleting destructor'
# 8 planquadrat_t::~planquadrat_t                             [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simplan.cc:55]
# 9 `vector destructor iterator'
#10 planquadrat_t::`vector deleting destructor'
#11 karte_t::destroy                                          [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:655]
#12 karte_t::~karte_t                                         [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:3003]
#13 karte_t::`scalar deleting destructor'
#14 simu_main                                                 [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simmain.cc:1387]
#15 sysmain                                                   [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys.cc:825]
#16 WinMain                                                   [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys_w.cc:1022]
Note: @0:20:28.141 in thread 14116

I have tried to run Valgrind on the server, but I cannot get this to work.

The issue appears to be some sort of heap corruption relating to roads, although it is not clear how this can have arisen or why it occurs sometimes rather than others. Any thoughts would be much appreciated.

Edit: Adding an express destructor to strasse_t makes it a little clearer to find the proximate cause of this issue in the debugger: the problem seems to be in the delete data [] line in minivec_tpl: a minivec of connected buildings is a part of every road tile in Simutrans-Extended. It is not clear at all why this is crashing, especially when (1) the minivec_tpl is a member of strasse_t and not a pointer to another value on the heap; and (2) the data[] in question consist of pointers to buildings, and this error occurs even when I explicitly clear the minivec in the destructor in strasse_t. Is there some known issue with minivec_tpl?

Edit 2: Testing replacing the minivec_tpl with a vector_tpl, this makes no difference: the crash still occurs when deleting the vector. Interestingly, the problem seems always to occur at the same location on the map: a stop at 838,1177.

Edit 3: I am having great difficulties reproducing this reliably enough to run any meaningful tests regarding this. I have, however, just pushed a speculative fix to the server: when the server restarts in about 20 minutes from now with the new version, it will be instructive to see whether the crashes still recur.
« Last Edit: December 11, 2017, 11:14:52 AM by jamespetts »

Offline O01eg

  • *
  • Posts: 79
  • Languages: EN, RU
Re: Heap corruption deleting members of objlist_t
« Reply #1 on: December 11, 2017, 05:22:11 PM »
Could you try to build server with address sanitizers?

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18547
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Heap corruption deleting members of objlist_t
« Reply #2 on: December 11, 2017, 05:27:16 PM »
Could you try to build server with address sanitizers?

Possibly, although that would take a lot of work to set up, as I have not used that before. However, reverting an earlier optimisation appears to have fixed this, although I am not sure why it caused this problem in the first place.

Thank you for the suggestion, however.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18547
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Heap corruption deleting members of objlist_t
« Reply #3 on: December 17, 2017, 02:19:55 AM »
I still seem to be getting this heap corruption occasionally (it occurs locally as well as on the server), although I am unable to reproduce it reliably enough even to begin trying to find a fix for it. It always seems to occur in the same place in the code, however. Oddly, it seems to happen several times in a row with the same saved game, but then, when I specifically go to try to reproduce it, it never seems to happen again.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2633
  • Languages: EN
Re: Heap corruption deleting members of objlist_t
« Reply #4 on: December 17, 2017, 03:17:45 AM »
Tracking such an error is very difficult. Something must be unsafely using a pointer. The reason it is hard to reproduce is it will only crash during a save/load cycle if it has sufficiently corrupted the heap memory management structures. If it corrupts an object it will not cause a crash. This might even explain the out of sync errors, as such corruption will cause an out of sync if it overwrites certain object state instead of the heap memory management structures.

Applying memory safe guards to catch what is responsible might not even be viable due to the complexity of the server game, the overhead might be too much for it.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18547
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Heap corruption deleting members of objlist_t
« Reply #5 on: December 17, 2017, 12:37:30 PM »
I am aware that these errors are fantastically difficult to trap.

I have been able to run Dr. Memory on a local copy of the server game with a debug build, but doing that productively depends on being able to reproduce the error while Dr. Memory is running. I was able to do this at least once (see the first post in this thread), but that gave me no more useful information than did the debugger, as it did not detect any errors other than the final error, after the heap had already been corrupted. Even Dr. Memory, it seems, is unable to find where the corruption is actually occurring.

Incidentally, one thing that I noticed when I was able to reproduce the error several times in a row is that it always occurred in exactly the same place; not just the same place in the code, but in respect of exactly the same piece of road with exactly the same position in the world. This suggests that this may not be general heap corruption, but some specific error (e.g. a double free).

Edit: I notice that this bug seems to occur with particular frequency after a number of players have joined the server. I will have to look into trying to replicate this by setting up a local server and joining it with numerous clients.

Edit 2: Attempting the above has not been a success - I connected a total of 11 clients to a local server without triggering a crash.

Edit 3: Attempting this again with a much larger map, from the Bridgewater-Brunel server, makes no difference: I am up to 4 clients and still no crash (it becomes extremely slow with this number running on one computer).

Edit 4: There is a possibility that a bug identified by AP causing infinite loop crashes when sending vehicles forth from depots on some occasions, which I have now fixed, was responsible for this, although it is hard to be sure. It will be very interesting to know whether this bug recurs after to-morrow's nightly build,
« Last Edit: December 17, 2017, 08:37:09 PM by jamespetts »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18547
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Heap corruption deleting members of objlist_t
« Reply #6 on: December 19, 2017, 11:48:39 PM »
This problem is again recurring on the server when several clients join. This is really quite mystifying.

The backtrace is:

Code: [Select]
[Thread 0x7fffeba2b700 (LWP 11799) exited]
[New Thread 0x7fffeba2b700 (LWP 11986)]
[Thread 0x7fffeba2b700 (LWP 11986) exited]
[Thread 0x7ffff0234700 (LWP 11868) exited]
[Thread 0x7ffff2a39700 (LWP 11864) exited]
[Thread 0x7fffec22c700 (LWP 11875) exited]
[Thread 0x7fffef232700 (LWP 11870) exited]
[Thread 0x7ffff1a37700 (LWP 11866) exited]
[Thread 0x7ffff423c700 (LWP 11862) exited]
[Thread 0x7fffed22e700 (LWP 11873) exited]
[Thread 0x7fffeca2d700 (LWP 11874) exited]
[Thread 0x7fffeda2f700 (LWP 11872) exited]
[Thread 0x7fffeea31700 (LWP 11871) exited]
[Thread 0x7fffefa33700 (LWP 11869) exited]
[Thread 0x7ffff1236700 (LWP 11867) exited]
[Thread 0x7ffff2238700 (LWP 11865) exited]
[Thread 0x7ffff3a3b700 (LWP 11863) exited]
[Thread 0x7ffff4a3d700 (LWP 11861) exited]
[Thread 0x7ffff523e700 (LWP 11860) exited]
[Thread 0x7ffff6240700 (LWP 11859) exited]
[Thread 0x7ffff6a41700 (LWP 11858) exited]
[Thread 0x7ffff5a3f700 (LWP 11857) exited]
[Thread 0x7ffff323a700 (LWP 11856) exited]
[Thread 0x7ffff0a35700 (LWP 11855) exited]
[Thread 0x7fffee230700 (LWP 11854) exited]

Thread 1 "simutrans-exten" received signal SIGSEGV, Segmentation fault.
__GI___libc_free (mem=0x4000a9400) at malloc.c:2951
2951    malloc.c: No such file or directory.
(gdb) backtrace
#0  __GI___libc_free (mem=0x4000a9400) at malloc.c:2951
#1  0x0000000000454885 in minivec_tpl<gebaeude_t*>::~minivec_tpl (this=0x541afb6f, __in_chrg=<optimized out>) at boden/wege/../../tpl/minivec_tpl.h:20
#2  strasse_t::~strasse_t (this=0x541afb20, __in_chrg=<optimized out>) at boden/wege/strasse.h:15
#3  strasse_t::~strasse_t (this=0x541afb20, __in_chrg=<optimized out>) at boden/wege/strasse.h:15
#4  0x000000000045b51e in objlist_t::~objlist_t (this=0x2be6ff00, __in_chrg=<optimized out>) at dataobj/objlist.cc:172
#5  0x0000000000447e64 in grund_t::~grund_t (this=0x2be6fef8, __in_chrg=<optimized out>) at boden/grund.cc:479
#6  0x0000000000444dc2 in boden_t::~boden_t (this=0x2be6fef8, __in_chrg=<optimized out>) at boden/boden.h:19
#7  boden_t::~boden_t (this=0x2be6fef8, __in_chrg=<optimized out>) at boden/boden.h:19
#8  0x00000000006786fd in planquadrat_t::~planquadrat_t (this=0x7fffc0dcd750, __in_chrg=<optimized out>) at simplan.cc:55
#9  0x00000000006af5c3 in karte_t::destroy (this=this@entry=0x11b57e0) at simworld.cc:655
#10 0x00000000006c9b53 in karte_t::load (this=this@entry=0x11b57e0, file=file@entry=0x7fffffffa920) at simworld.cc:8432
#11 0x00000000006ccd9a in karte_t::load (this=this@entry=0x11b57e0, filename=filename@entry=0x7fffffffafe0 "server13353-network.sve") at simworld.cc:8242
#12 0x0000000000560cd8 in nwc_sync_t::do_command (this=0x6ad42bb0, welt=0x11b57e0) at network/network_cmd_ingame.cc:755
#13 0x00000000006b96f2 in karte_t::do_network_world_command (this=this@entry=0x11b57e0, nwc=nwc@entry=0x6ad42bb0) at simworld.cc:10142
#14 0x00000000006b9c1f in karte_t::process_network_commands (this=this@entry=0x11b57e0, ms_difference=ms_difference@entry=0x7fffffffbb8c) at simworld.cc:10088
#15 0x00000000006c8135 in karte_t::interactive (this=this@entry=0x11b57e0, quit_month=quit_month@entry=2147483647) at simworld.cc:10251
#16 0x000000000066e8fe in simu_main (argc=argc@entry=23, argv=argv@entry=0x7fffffffe448) at simmain.cc:1362
#17 0x000000000067c8e9 in sysmain (argc=argc@entry=23, argv=argv@entry=0x7fffffffe448) at simsys.cc:825
#18 0x0000000000724655 in main (argc=23, argv=0x7fffffffe448) at simsys_posix.cc:197

Edit: The odd thing about this is that the error always occurs specifically in relation to the data[] member of the minivec_tpl<> member of the strasse_t class. The other data in the objects appears to be in order. That class is derived from the weg_t class. I have moved the minivec_tpl<> class to the weg_t parent class, and have not been able to reproduce the error locally since doing that.

Given that the error was always difficult to reproduce, this is far from conclusive, but, with the current saved game on the Bridgewater-Brunel, it had become much easier to reproduce, occuring much more frequently, so there is some hope that this might have fixed it.

I shall be interested to see whether the server is more stable to-morrow.

Incidentally, to-morrow I go away for Christmas, and will not have access to my normal computer for coding. I usually concentrate on pakset work during my Christmas break, as I have access only to my small Linux computer, on which I do not have a graphical debugger set up (and have not worked out how to do this).

Edit 2: This does not seem to have assisted as I had thought. Although it seemed to work correctly initially, I have had more warnings of heap corruption, albeit this time not in the same place and not (yet) causing a crash: I get the "crt detected that the application wrote to memory after end of heap buffer" message in another part of objlist_t, albeit still dealing with ways/roads.

Something, somewhere is writing past the end of the heap buffer (possibly on deletion) in objlist and interfering with the objects in there. I am currently running Dr. Memory to try to diagnose this.

Edit 3: Some very interesting results from Dr. Memory:

Code: [Select]
Dr. Memory version 1.11.0 build 2 built on Aug 29 2016 02:42:07
Dr. Memory results for pid 18192: "Simutrans-Extended-debug.exe"
Application cmdline: "C:\Users\James\Documents\Development\Simutrans\simutrans-extended-sources\simutrans\Simutrans-Extended-debug.exe"
Recorded 115 suppression(s) from default C:\Program Files (x86)\Dr. Memory\bin\suppress-default.txt

Error #1: UNADDRESSABLE ACCESS beyond heap bounds: reading 0x5d7941fc-0x5d7941fe 2 byte(s)
# 0 quickstone_tpl<>::is_bound                [c:\users\james\documents\development\simutrans\simutrans-extended-sources\tpl\quickstone_tpl.h:211]
# 1 schiene_t::can_reserve                    [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\wege\schiene.h:75]
# 2 schiene_t::reserve                        [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\wege\schiene.cc:209]
# 3 convoi_t::reserve_own_tiles               [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simconvoi.cc:410]
# 4 convoi_t::step                            [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simconvoi.cc:2087]
# 5 karte_t::step                             [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:5390]
# 6 karte_t::interactive                      [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:10381]
# 7 simu_main                                 [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simmain.cc:1362]
# 8 sysmain                                   [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys.cc:825]
# 9 WinMain                                   [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys_w.cc:1022]
Note: @0:06:32.488 in thread 14516
Note: refers to 0 byte(s) beyond last valid byte in prior malloc
Note: prev lower malloc:  0x5d7941a8-0x5d7941fc
Note: allocated here:
Note: # 0 replace_operator_new               [d:\drmemory_package\common\alloc_replace.c:2899]
Note: # 1 grund_t::rdwr                      [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\grund.cc:350]
Note: # 2 boden_t::boden_t                   [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\boden.cc:23]
Note: # 3 planquadrat_t::rdwr                [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simplan.cc:262]
Note: # 4 karte_t::load                      [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:8715]
Note: # 5 karte_t::load                      [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:8240]
Note: # 6 simu_main                          [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simmain.cc:1228]
Note: # 7 sysmain                            [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys.cc:825]
Note: # 8 WinMain                            [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys_w.cc:1022]
Note: instruction: movzx  (%eax) -> %ecx

Error #2: UNADDRESSABLE ACCESS beyond heap bounds: reading 0x5d7941fe-0x5d7941ff 1 byte(s)
# 0 schiene_t::can_reserve                    [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\wege\schiene.h:75]
# 1 schiene_t::reserve                        [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\wege\schiene.cc:209]
# 2 convoi_t::reserve_own_tiles               [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simconvoi.cc:410]
# 3 convoi_t::step                            [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simconvoi.cc:2087]
# 4 karte_t::step                             [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:5390]
# 5 karte_t::interactive                      [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:10381]
# 6 simu_main                                 [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simmain.cc:1362]
# 7 sysmain                                   [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys.cc:825]
# 8 WinMain                                   [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys_w.cc:1022]
Note: @0:06:32.500 in thread 14516
Note: refers to 2 byte(s) beyond last valid byte in prior malloc
Note: prev lower malloc:  0x5d7941a8-0x5d7941fc
Note: allocated here:
Note: # 0 replace_operator_new               [d:\drmemory_package\common\alloc_replace.c:2899]
Note: # 1 grund_t::rdwr                      [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\grund.cc:350]
Note: # 2 boden_t::boden_t                   [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\boden.cc:23]
Note: # 3 planquadrat_t::rdwr                [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simplan.cc:262]
Note: # 4 karte_t::load                      [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:8715]
Note: # 5 karte_t::load                      [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:8240]
Note: # 6 simu_main                          [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simmain.cc:1228]
Note: # 7 sysmain                            [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys.cc:825]
Note: # 8 WinMain                            [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simsys_w.cc:1022]
Note: instruction: movzx  0x56(%ecx) -> %edx

Error #3: UNADDRESSABLE ACCESS beyond heap bounds: reading 0xb8f28508-0xb8f2850c 4 byte(s)
# 0 _longest_match                         [F:\Develop\vs140\build\zlib-1.2.8\contrib\masmx86\match686.asm:375]
# 1 inflateUndermine           
# 2 deflate                     
# 3 gzungetc                   
# 4 gzwrite                     
# 5 loadsave_t::flush_buffer               [c:\users\james\documents\development\simutrans\simutrans-extended-sources\dataobj\loadsave.cc:678]
# 6 save_thread                            [c:\users\james\documents\development\simutrans\simutrans-extended-sources\dataobj\loadsave.cc:160]
# 7 pthreadVCE2.dll!pthread_setcanceltype +0x4bce   (0x73995eef <pthreadVCE2.dll+0x5eef>)
# 8 MSVCR100.dll!endthreadex              +0x39     (0x5090c6de <MSVCR100.dll+0x5c6de>)
# 9 MSVCR100.dll!endthreadex              +0xe3     (0x5090c788 <MSVCR100.dll+0x5c788>)
#10 KERNEL32.dll!BaseThreadInitThunk      +0x11     (0x76e9336a <KERNEL32.dll+0x1336a>)
Note: @0:10:35.908 in thread 11304
Note: refers to 0 byte(s) beyond last valid byte in prior malloc
Note: prev lower malloc:  0xb8f18508-0xb8f28508
Note: allocated here:
Note: # 0 replace_malloc                             [d:\drmemory_package\common\alloc_replace.c:2576]
Note: # 1 zcalloc                     
Note: # 2 deflateInit2_               
Note: # 3 gzungetc                   
Note: # 4 gzwrite                     
Note: # 5 loadsave_t::write                          [c:\users\james\documents\development\simutrans\simutrans-extended-sources\dataobj\loadsave.cc:660]
Note: # 6 loadsave_t::wr_open                        [c:\users\james\documents\development\simutrans\simutrans-extended-sources\dataobj\loadsave.cc:487]
Note: # 7 karte_t::save                              [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:7694]
Note: # 8 interaction_t::process_event               [c:\users\james\documents\development\simutrans\simutrans-extended-sources\siminteraction.cc:353]
Note: # 9 interaction_t::check_events                [c:\users\james\documents\development\simutrans\simutrans-extended-sources\siminteraction.cc:439]
Note: #10 karte_t::interactive                       [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simworld.cc:10237]
Note: #11 simu_main                                  [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simmain.cc:1362]
Note: instruction: xor    0x04(%edx,%edi) %eax -> %eax

The important part seems to be:

Code: [Select]
Note: # 1 grund_t::rdwr                      [c:\users\james\documents\development\simutrans\simutrans-extended-sources\boden\grund.cc:350]

That line is:

Code: [Select]
weg = new strasse_t(file);

and is in a part of the code that is unchanged from Standard.

I wonder, therefore, whether there is an inherent problem (perhaps due to the way in which freelist works?) with having a minivec as a direct member of weg_t or any derived class.

To test this, I have changed the connected_buildings minivec_tpl into a pointer to the minivec, and created appropriate new and delete operators. I fear that this may adversely impact performance, but it may be necessary in order to make the game run stably.

I am testing again with Dr. Memory and will report back with findings.

Edit 4: This has not helped: I am getting the same error.

Edit 5: For reference, this is the saved game in which I am able to reproduce this bug reliably.

Edit 6: I now have very little time (hours) left before I go away for Christmas and it will be extremely difficult to work on this further until I get back. I had wondered whether the problem was caused by errors in objlist allowing members of that list to overwrite data belonging to other members of that list, so I have updated to the latest objlist code from Standard, but this has not helped in any way.

Edit 7: I have committed a further change to impose a hard limit (based on that in Standard) on the number of vehicles on a tile of water way, having changed this code some time ago to allow for a restriction based on pakset creator settings. This enforces the limit in addition to the user settings, and should not affect gameplay. I have not had a chance to test whether this has helped, but it will be in the next nightly build.
« Last Edit: December 20, 2017, 02:31:59 PM by jamespetts »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18547
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Heap corruption deleting members of objlist_t
« Reply #7 on: December 20, 2017, 08:01:17 PM »
I wonder whether this commit might have something to do with the corruption?


https://github.com/jamespetts/simutrans-extended/commits/master?after=17ddae9d3758e634394f3f382fdbdcd25ad38daa+69

Offline Dwachs

  • DevTeam, Coder/patcher
  • Administrator
  • *
  • Posts: 4564
  • Languages: EN, DE, AT
Re: Heap corruption deleting members of objlist_t
« Reply #8 on: December 21, 2017, 07:31:48 AM »
Valgrind under linux can easily find, where bad memory things happen. E.g., emitting error messages like 'reading memory at code line xx that was freed by code line yy'.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18547
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Heap corruption deleting members of objlist_t
« Reply #9 on: December 21, 2017, 01:49:12 PM »
Valgrind under linux can easily find, where bad memory things happen. E.g., emitting error messages like 'reading memory at code line xx that was freed by code line yy'.

Thank you for that suggestion. I have just tried this. Valgrind spotted a number of uninitialised variable issues that Dr. Memory did not spot (including one that I believe is common to Standard: fix below:

Code: [Select]
marker_t() : bits(NULL) { bits_length = 0; init(0, 0); }

), but unfortunately, my mini-desktop on which I run Linux and which I have with me whilst away from home runs out of memory (8Gb of RAM) if I try to load any of the larger maps, including the one giving the trouble with Valgrind enabled.

Edit: I note that, with the latest fixes, I am no longer able to reproduce the heap corruption on my Linux machine, although it is difficult to test too thoroughly, as running more than about two instances quickly causes my computer to run out of memory.