The International Simutrans Forum

 

Author Topic: Instability on the Bridgewater-Brunel server  (Read 11924 times)

0 Members and 1 Guest are viewing this topic.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Instability on the Bridgewater-Brunel server
« on: September 06, 2018, 03:21:35 PM »
Clients seem to be going out of sync with the server a lot.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #1 on: September 06, 2018, 04:31:23 PM »
Have you any way of telling at present whether these issues are consistent with the server crashing or whether the issue is relating to synchronisation? I am afraid that, if it is the latter, the sort of bug that would cause failures to remain synchronised only on a map as large and developed as this one is likely to be so subtle and complex that it would be virtually impossible to solve, unless people could isolate some specific feature of the game that is only now being used for the first time (although it is hard to think what that might be, since electricity supply and air travel have been in use for an extended period).

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 563
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #2 on: September 06, 2018, 11:20:27 PM »
I am having trouble staying in sync as well.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #3 on: September 06, 2018, 11:26:27 PM »
I connected briefly earlier and noticed no difficulty. Can I check that you all have the latest version of the pakset, which was updated last night?

Offline Junna

  • Devotee
  • *
  • Posts: 1081
Re: Instability on the Bridgewater-Brunel server
« Reply #4 on: September 07, 2018, 03:47:46 AM »
It resetted again when I was replacing 800 road vehicles, and it resetted to way before I started the replacement, once again.. all that work lost... ugh, why do I bother. I even forced a save by reconnecting before starting the replacement in case this happened, but it still resetted to an older save than the one I forced, as well as the hourly save it had done...

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #5 on: September 07, 2018, 04:21:56 AM »
Quote
Have you any way of telling at present whether these issues are consistent with the server crashing or whether the issue is relating to synchronisation? I am afraid that, if it is the latter, the sort of bug that would cause failures to remain synchronised only on a map as large and developed as this one is likely to be so subtle and complex that it would be virtually impossible to solve, unless people could isolate some specific feature of the game that is only now being used for the first time (although it is hard to think what that might be, since electricity supply and air travel have been in use for an extended period).
Surely the crashes should be logged on the server like all application crashes?

I cannot tell from the client side since two cases produce the same cannot connect to server error in the game list.
  • Server has crashed and not yet restarted.
  • Someone beat me to selecting the server and is taking an extremely long time to join it.
Quote
I connected briefly earlier and noticed no difficulty. Can I check that you all have the latest version of the pakset, which was updated last night?
I use my updater so yes it updated the pakset files.

It is very sporadic... For example sometimes I connect and remain connected and in sync for well over an hour before I eventually have to quit to do something else. Other times I connect just to be immediately dropped out of sync the instant the map finishes loading. I can only assume it is out of sync because my client does not crash after I have been disconnected, however that need not be the case if the server is crashing due to running out of resources such as memory.

I know of one case that is highly likely to OOS one after a while. Changing vehicle schedules such that the vehicle will be displaced (eg a train on diagonal or over a junction) seems to not be fully coupled to the server so although the order does go through to the server it begins before or does something differently on the client that issued the order and so eventually a checksum mismatch will occur. This is inherited from standard where I first noticed it. You can login to find trains that you thought you fixed turned out to be stuck over junctions still.

I suspect another case might be related to sending the server orders at the same time the server gets scheduled for a save/load cycle due to someone joining. I am not sure that the orders are being properly discarded from queues and the result might be potentially that all clients get kicked due to these boarderline case orders being executed inconsistently between client and server. It could also be the server crashed during the load cycle, it is hard to tell.

It would be nice if the two cases were distinguished better. If you are disconnected from the server it should say you are disconnected from the server. If you checksum mismatch the server it should say you went out of sync with the server. Currently it uses the same message for both cases. Apparently there is a way to get the clients to log this in console, but I cannot seem to get it to work...

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #6 on: September 08, 2018, 10:20:40 PM »
I have split this from the general topic about the server as this is perhaps more properly treated as a bug report rather than discussion about the maintenance of the server generally.

Firstly, to answer Dr. Supergood's questions: the logs will show crashes, but it is hard to match these to specific instances of reported desyncs. I have just amended one of the scripts to add the date and time of each check whether the server game is still running, so these should in future be slightly easier to correlate.

I suspect that the problem may well be server crashes, but it is difficult to be sure at this juncture. In any event, there appear to be an increasing number of reports of server crashes losing people's work and it is important to try to resolve this.

These crashes appear to be occurring in non-reproducible ways. There is one possible candidate issue currently known that might be the cause of this: there is an occasional crash bug that has persisted for some time which I have never been able to fix because I have never been able to reproduce it reliably enough. It manifests in the following piece of code in simworld.cc:

Code: [Select]
FOR(minivec_tpl<const planquadrat_t*>, const& current_tile_3, current_destination.building->get_tiles())
            {
               const nearby_halt_t* halt_list = current_tile_3->get_haltlist();
               if (!halt_list)
               {
                  continue;
               }
               for (int h = current_tile_3->get_haltlist_count() - 1; h >= 0; h--)
               {
                  halthandle_t halt = halt_list[h].halt;
                  if (halt->is_enabled(wtyp))
                  {
                     // Previous versions excluded overcrowded halts here, but we need to know which
                     // overcrowded halt would have been the best start halt if it was not overcrowded,
                     // so do that below.
#ifdef MULTI_THREAD
                     destination_list[passenger_generation_thread_number].append(halt);
#else
                     destination_list.append(halt);
#endif
                  }
               }
            }

It manifests specifically at this line:

Code: [Select]
halthandle_t halt = halt_list[h].halt;

The way in which it manifests is that halt_list[h] is an invalid memory location of a non-NULL value, indicating that this memory location has been deleted. This usually happens when the variable "h" is a non-zero number.

It is known to occur in the demo game loaded automatically on starting the game: sometimes it occurs a few seconds after starting the game; at other times, it can run for hours without occurring at all. I spent some considerable time earlier this year trying to track it down, and made some progress, but was not able to complete the work because there came a time when this became impossible to reproduce no matter how many times that I loaded the demo game and ran it on fast-forward (which had for a time reproduced this fairly reliably). I did note that it occurred at least sometimes (and possibly all of the time) on multi-tile buildings with holes in them (i.e., which are not contiguous quadrilaterals), but it is not clear whether this is related to the cause (and I did spend some time investigating this aspect of things to no avail).

I have been attempting to reproduce this error whilst writing this post, but to no avail. If anyone can assist in trying to track down the (or a possible) cause of this error, or suggest what might be done to do so (beyond things that I have already tried, such as using Dr. Memory) that would be much appreciated.

As to discerning the cause of server disconnects, the code in this respect is the same as in Standard: I do not think that I know enough about networking to be able to modify this easily. I agree that this would be useful, however.

Incidentally, how has stability been to-day?

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #7 on: September 08, 2018, 10:44:32 PM »
Quote
Incidentally, how has stability been to-day?
Left server running while I made dinner and came back to find client closed (crashed without error message).

Tried to join server just now and was immediately prompted with "you have lost synchronization" or whatever after load and the server resuming.

Finally managed to rejoin just to find all my work I made before making dinner was lost!

Yeh it is one of those days...

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 563
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #8 on: September 08, 2018, 11:32:59 PM »
Can confirm that joining the server is a PITA at the moment...

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #9 on: September 08, 2018, 11:45:15 PM »
I am not sure what the joining issue is, unless the server is now crashing so frequently that people are constantly trying to rejoin. My apologies that the game experience has deteriorated.

I am currently running the server game offline to attempt to reproduce this, although I have not had any success so far.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #10 on: September 09, 2018, 02:01:18 AM »
The server appears resource constrained at the moment. Frequently it hangs or stutters for a few seconds before resuming. This is not connection related since when play resumes I am not far behind the server, instead it must be my client waiting for the server which is falling behind.

Many of the crashes during loading could easily be explained by running out of memory or similar resources.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #11 on: September 09, 2018, 10:49:43 AM »
I have noticed that the game on the server is running at ~94% of RAM, which suggests that this is the issue - however, the amount of RAM on the server is 6GB, whereas loading the game afresh locally gives rise to memory utilisation of only 4GB, suggesting a leak: indeed, when I run Simutrans-Extended locally, the amount of memory used steadily increases with time, also suggesting a leak.

However, running Dr. Memory fails to diagnose this. Whilst it reports many suspected leaks:

Edit: Code too long to paste here

All of these are  in code shared with Standard, and almost all of them are in code run only on startup (mainly the translator and object file reader code). There is one more suspicious leak, which is this:

Code: [Select]
Error #70: POSSIBLE LEAK 32744 direct bytes 0x0000000008c6f820-0x0000000008c77808 + 82702 indirect bytes
# 0 replace_malloc                                  [d:\drmemory_package\common\alloc_replace.c:2576]
# 1 _chvalidator_l                                  [d:\th\minkernel\crts\ucrt\src\appcrt\convert\isctype.cpp:48]
# 2 _chvalidchk_l                                   [d:\th\minkernel\crts\ucrt\inc\ctype.h:161]
# 3 __acrt_locale_changed                           [d:\th\minkernel\crts\ucrt\inc\corecrt_internal.h:658]
# 4 _ischartype_l                                   [d:\th\minkernel\crts\ucrt\inc\ctype.h:186]
# 5 __crt_strtox::is_space                          [d:\th\minkernel\crts\ucrt\inc\corecrt_internal_strtox.h:145]
# 6 __crt_strtox::parse_integer<>                   [d:\th\minkernel\crts\ucrt\inc\corecrt_internal_strtox.h:282]
# 7 xmalloc                                         [c:\users\james\documents\development\simutrans\simutrans-extended-sources\simmem.cc:156]
# 8 freelist_t::gimme_node                          [c:\users\james\documents\development\simutrans\simutrans-extended-sources\dataobj\freelist.cc:106]
# 9 slist_tpl<>::node_t::operator new               [c:\users\james\documents\development\simutrans\simutrans-extended-sources\tpl\slist_tpl.h:42]
#10 slist_tpl<>::insert                             [c:\users\james\documents\development\simutrans\simutrans-extended-sources\tpl\slist_tpl.h:425]
#11 hashtable_tpl<>::set                            [c:\users\james\documents\development\simutrans\simutrans-extended-sources\tpl\hashtable_tpl.h:346]

which is in a part of the code which is used very frequently in Extended (the "set" command in the hashtable), however, this is also in code from Standard. Perhaps the leak actually originates in code from Standard?

Does anyone know how the hashtable "set" implementation works well enough to try to find what is causing this trouble?
Edit: I should note that I am unable to load the Stevenson-Seimens saved game when running Dr. Memory, as this will crash with the following error when doing so (but will load without difficulty when not using Dr. Memory):
Code: [Select]
Dr. Memory version 1.11.0 build 2 built on Aug 29 2016 02:41:18
Dr. Memory results for pid 9212: "Simutrans-Extended-debug.exe"
Application cmdline: "C:\Users\James\Documents\Development\Simutrans\simutrans-extended-sources\simutrans\Simutrans-Extended-debug.exe"
Recorded 115 suppression(s) from default C:\Program Files (x86)\Dr. Memory\bin64\suppress-default.txt

Error #1: UNADDRESSABLE ACCESS: writing 0x0000000000000000-0x0000000000000008 8 byte(s)
# 0 inflate_fast               
# 1 inflate                     
# 2 adler32_combine64           
# 3 gzread                     
# 4 loadsave_t::fill_buffer               [c:\users\james\documents\development\simutrans\simutrans-extended-sources\dataobj\loadsave.cc:777]
# 5 load_thread                           [c:\users\james\documents\development\simutrans\simutrans-extended-sources\dataobj\loadsave.cc:80]
# 6 pthreadVC2.dll!sched_get_priority_max+0x56b2   (0x000007fefbd769b0 <pthreadVC2.dll+0x69b0>)
# 7 MSVCR100.dll!_callthreadstartex       [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c:314]
# 8 MSVCR100.dll!_threadstartex           [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c:292]
Note: @0:01:21.106 in thread 3800
Note: instruction: mov    %r8 -> (%rbx)
This makes it very difficult to assess whether the leak is caused by something present in the Stevenson-Seimens saved game (where I had observed the leak) not present in the demo game from which the output in the original, unedited version of this post was generated.

Edit 2: Compiling in 32-bit rather than 64-bit allows Dr. Memory to work without errors - but I can find no significant leaks using the Stevenson-Seimens saved game with this method, even though memory usage is increasing constantly as reported by Windows - all of the reported leaks originate from the translator or object reader.

The USE_VALGRIND_MEMCHECK preprocessor directive appears to be deprecated, as there is no /valgrind/memcheck.h.
« Last Edit: September 09, 2018, 11:40:50 AM by jamespetts »

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #12 on: September 09, 2018, 07:38:26 PM »
I do not think this will detect logic leaks (if that is the correct term). A conventional leak refers to something be allocated, used, discarded but not destroyed to free resources. A logical leak is when something is allocated, used, not discarded or destroyed and not used again.

The problem with logical leaks is that they will not be detected by memory allocation watchers and such. This is because they might even be cleaned up at certain times, eg on game closure or map reload etc.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #13 on: September 09, 2018, 09:17:43 PM »
I think that this is what the USE_VALGRIND_MEMCHECK is for: to detect at least some of what you describe as "logic leaks" in freelist and the like.

It would be helpful if anyone could attempt running Simutrans-Extended using the saved game from the Stevenson-Seimens server and Valgrind with this enabled (if this is still possible - does anyone know where /valgrind/memcheck.h is?) to see whether this is a detectable leak.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #14 on: September 13, 2018, 11:30:24 AM »
MSVC 2017 has a heap profiling feature which allows you to inspect the number of objects allocated and most importantly compare snapshots of the heap for deltas. Using this it should be possible to spot the leak simply be noting what objects types always have large positive deltas. Or at least that is the theory...

I am trying to run the server game but as you can guess, not going to well. 22 minutes in and still stuck at calculating paths.

I assume the Stevenson-Seimens server is smaller? Anyone got a link to the save game as apparently the server that hosts it is currently down.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #15 on: September 13, 2018, 11:39:20 AM »
Dr. Supergood - that is most helpful. The Stevenson-Seimens game is much, much smaller and yet still exhibits this problem. I may well have to repair my computer (you may well be right about the PSU - but replacing one of those takes a lot of time, which is somewhat limited at present; I will have to do it sooner or later, however) before looking into this intensively. If you are able to make progress with heap profiling with MSVC 2017, that would be most useful. Indeed, if this feature assists, it might be worth my while upgrading.

Offline Vladki cz

  • Devotee
  • *
  • Posts: 2611
    • My addons, mostly roadsigns
  • Languages: EN, CS
Re: Instability on the Bridgewater-Brunel server
« Reply #16 on: September 13, 2018, 01:01:03 PM »
I assume the Stevenson-Seimens server is smaller? Anyone got a link to the save game as apparently the server that hosts it is currently down.
Try this one: https://uran.webstep.net/~vladki/simutrans/autosave08.sve

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #17 on: September 13, 2018, 04:52:20 PM »
I cannot detect any gameplay leaks in the Stevenson-Seimens save. After 30 minutes of fast forward simulation it gained 294 MB.

Of which ~288MB of this was allocated by...
void route_t::INIT_NODES(uint32 max_route_steps, const koord &world_size)
After roughly 12 minutes and came in the form of 6 route_t::ANode[] each of exactly 48,000,244 bytes size and were ultimately generated by stadt_t::check_all_private_car_routes(). The game ended with 14 of these identical sized arrays, with the other 8 having been generated around map loading from sources such as for player vehicles. I am guessing this memory is recycled so cannot be considered a leak.

However due to the server game being so massive and getting more and more public cars it could quite well be possible that dozens more of such arrays are needed/used. If that is the case those would push up memory usage over time potentially using many gigabytes of memory. If these represent cached routes there might need to be a line drawn as to how many routes are allowed to be cached at any given time. To keep performance reasonable more staggering could be performed as more routes are required.

6.6MB was allocated in the form of transferring_cargo_t[] as part of vector expansion and 3.8MB in the form of void as part of tile object list expansion. These can potentially be considered logic leaks since the vectors and object list do not dynamically shrink from what I can tell. The transferring_cargo_t[] produces few large arrays as each element is a complex object of well over 32 bytes, this could be combated to some extent by turning it into an array of pointers and having each transferring_cargo_t allocated from the free lists so each element is at most 8 bytes (pointer length). Object list expansion seems to suffer the most from pedestrian creation and movement, with some object lists being observed to hit the 255 element maximum limit which is over 4kb for a single tile.

Both these logical leaks could be solved by adding shrinking logic to the vectors. Transfering cargo could be made to shrink the backing list if it is larger than 8 elements and currently is utilizing less than 1/4 (bit right shift of 2) of its capacity Logic behind this is that a busy transfer will always be busy transferring so one would only want to downsize in response to exceptional loads. Object list could use something similar but can likely be set more aggressive to cull large arrays when they have 4 free capacity, since outside of pedestrians most tiles will have under 4 objects on them. Numbers obviously could be fine turned for performance and memory trade offs. Doubt this will make much of a difference as these lists are likely purged during save/load which occurs frequently. Still might cause a fluctuation of 100MB odd on the server.

I ran save/load cycles and have spotted a leak... Every save/load cycle quickstone_hashtable_tpl<haltestelle_t,haltestelle_t::connexion *>[] increased in allocation by 1,243 instances totalling 3,087,612 bytes. This leak is easily repeatable, with each save/load cycle of the map adding exactly the same number.

The allocations below were observed...
Code: [Select]
Simutrans-Extended Normal x64 Debug (SDL 2).exe!path_explorer_t::compartment_t::step() - Line 674 C++
  Simutrans-Extended Normal x64 Debug (SDL 2).exe!haltestelle_t::haltestelle_t() - Line 427 C++
Both sources appear to be leaking between save/load cycles. As the bridgewater server is a lot more complex and is subject to a lot of save/load cycling this could easily explain why it runs out of memory. In this simple map 3 MB is leaked per save/load cycle. In a more complex map that could easily be 30 or even 100MB. 4 players joining would be 400MB, etc...

It is worth noting that the route_t::ANode[] were not destroyed between save/load despite being created initially after first load ran for a bit. This is probably not a leak as I am guessing it gets recycled and reused.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #18 on: September 15, 2018, 05:27:06 PM »
Thank you - that is very helpful. I have just pushed two possible fixes for memory leaks on saving/loading. The first is relating to private car route finding: the cause of this was, I think, that the ANODE arrays were thread local, created on the heap, and not explicitly destroyed when the threads were terminated (as they are on loading a saved game).

The second was, I think, some logic errors in the code for deleting the connexions* object which prevented this from occurring when loading a saved game.

Additionally, I have now had a chance to check the system log messages, which confirms that at least most of the crashes are associated with running out of memory. I shall be very interested to see whether there is an improvement with the next nightly build.

Thank you again very much for your investigative work, and apologies for the difficulties on the server.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #19 on: September 16, 2018, 01:21:08 PM »
Some testing to-day: players are reporting greater stability with this version. However, there may still be some memory leaks left: what I notice after joining the game multiple times is that, memory usage gradually increases after a player joins until it reaches a plateau, where it fluctuates sligthly. When another player joins, memory usage falls considerably again, but not quite back to the level where it was when the previous player joins, and then increases to a slightly higher plateau.

Some examples from recent testing are that the free memory on the server was 376MB after the first join, falling to 128-129MB free after a few minutes. After the second join, it increased to 371MB free, falling to 115-116MB free after a few minutes.

However, curiously, a third join seems to have increased free memory to 493MB immediately after the join, falling to 257MB, so the pattern does not seem to be entirely consistent.

Offline ACarlotti

  • *
  • Posts: 405
Re: Instability on the Bridgewater-Brunel server
« Reply #20 on: September 17, 2018, 05:14:22 AM »
In light of the above, it seems likely (to me, at least) that a primary cause of the incomplete saves was that the server was running out of memory mid-save. However, there must have been other contributing factors involved. In particular, these incomplete saves should not be replacing complete saves. In karte_t::save there is code to support using a temporary filename while writing the save; once the save has been successfully written this file is then renamed to replace the previous save. However, this tempory file is used only if the filename passed to karte_t::save beings with "save/". So I have two questions.
1. Why is it only used in those cases? (Prissi, who wrote that feature in 2012, would presumably know.)
2. Are the saves that are made by the server for backup purposes written to a filename beginning "save/"?

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #21 on: September 17, 2018, 09:51:07 PM »
Thank you for this - I was not aware of this restriction to the save file failsafe code, nor do I understand the reason for it. I do know, however, that the server saved games are not stored in a "/save" folder.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 9438
  • Languages: De,EN,JP
Re: Instability on the Bridgewater-Brunel server
« Reply #22 on: September 18, 2018, 12:56:53 PM »
Writing to other directories usually did not overwrite an existing save game respective it is meaningless to keep an old save for the next client. However, for an exisitng game, you press the same filename. If simutrans crashes during saving, you have lost everything. So only deleting the old one after finishing saving.

Servers usually do not use autosave, since saving causes an out of sync error with the clients (if they are not saving too then).

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #23 on: September 22, 2018, 08:24:26 PM »
I think that the memory leak issue has now gone, as there are no reports of out of memory crashes since last week. I note the reasons given by Prissi for restricting the safe saving behaviour to filenames with /save, but, since, on a server, the saved game file is overwritten whenever that game is loaded/saved when a player joins (or by an autosave facility created by use of a script with nettool, as is used on the Bridgewater-Brunel server, to avoid desyncing when saving), it is useful to avoid overwriting the saved game until the save has confirmed to have been completed successfully, so I have removed the code restricting the use of this feature to files in the /save directory.

There is a new source of instability on the server, however: the game crashes with a segfault shortly after loading. I have not been able to reproduce this on a Windows debug build, but it can be reproduced (without being able to be traced) on the cross-compiled Windows release build.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #24 on: September 22, 2018, 09:59:17 PM »
Quote
There is a new source of instability on the server, however: the game crashes with a segfault shortly after loading. I have not been able to reproduce this on a Windows debug build, but it can be reproduced (without being able to be traced) on the cross-compiled Windows release build.
?!

It is not crashing at all for me. I just get dropped out of sync after 5-10 minutes of being connected. I can then immediately rejoin and it resumes where it left off (no progress lost). Makes playing with more than 1 person connected impossible.

Using nightly build from server as downloaded by my tool, as always.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #25 on: September 23, 2018, 09:11:37 AM »
This is the first confirmation that there is a desync issue independent of the fixed memory leak issue. When I tested this yesterday, the game crashed immediately after losing synchronisation with the server, so I inferred that the problem was that the server was crashing; however, testing again to-day, I see that synchronisation with the server is lost when neither server nor client crashes.

Finding and fixing this problem will be a gargantuan undertaking (as any desync bug subtle enough only to manifest itself only now is likely to be almost impossible to find), which will be likely to delay by a very long time progress on fixing other bugs or adding new features. I am not likely to be able to start in earnest until my own computer hardware problems are resolved, which I hope that they will be when I have time to install the new power supply that I ordered last week, whenever it arrives.

Any assistance from anyone in narrowing this down in any way would be very much appreciated (for example, testing whether old versions of the saved games remain in sync with the current build, or whether the current version of the saved game remains in sync when loaded with an older build, checking whether Linux clients are equally affected, checking whether the problem remains after deleting all aircraft on an offline test platform, checking whether the problem remains after removing city and/or industry electrification, etc.).

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #26 on: September 23, 2018, 09:24:55 AM »
I could possibly do the tracking down but I lack hardware for it. At best I could do would be a windows server with windows configuration. However I would need exactly the server configuration (sync time, time per step etc) to rule that out as a variable.

Another more interactive approach would involve running the server with builds with particular components disabled. For example all power net updating could be turned off, or all aircraft movement disabled, etc. This could potentially be quicker than manually deleting all aircraft or modifying the save.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #27 on: September 23, 2018, 09:39:46 AM »
That would be very helpful - thank you.
For reference, here is the server's simuconf.tab:
Code: [Select]
# simuconf.tab
#
# Low-Level values and constants
#
# Lines starting with '#' or any non_character letter will be ignored!
# To actually set a value, remove the leading '#'!
#
# This file can be found in many places:
#
# simutrans/config/simuconf.tab (this one)
# ~/simutrans/simuconf.tab (in the user directory, if singleuser_install == 0 or not present, first only pak-file path will be read)
# simutrans/pakfilefolder/config/simuconf.tab
# ~/simutrans/simuconf.tab (readed a second time, s.a.)
#
################################# Base settings ##################################
#
# This simuconf.tab will be read first => we set meaningful defaults here.
#
# load/save the files in the users or the program directory directory? (default: 0 = user directory)
# ATTENTION!
# will be only used if this file is located in the program directory at config/simuconf.tab!
singleuser_install = 1
#
#
#
# Do not delete these comment line! (Needed for installer)

progdir_overrides_savegame_settings = 0
pak_overrides_savegame_settings = 0
userdir_overrides_savegame_settings = 0

# Default pak file path
# which graphics you want to play?
# Nothing means automatic selection
# ATTENTION!
# This will be only used if this file is located in the program directory at config/simuconf.tab!
# and will be overwritten by the settings from simutrans/simuconf.tab in the user directory
#
#pak_file_path = pak/
#pak_file_path = pak.german/
#pak_file_path = pak128/
#pak_file_path = pak.japan/
#pak_file_path = pak.winter/
#pak_file_path = pak.ttd/

# The maximum number of position tested during a way search
# Consumes 16*x Bytes main memory, where x is the "max_route_steps" value.
max_route_steps = 1500000

# How many tiles to check before giving up on finding a free bay at a stop or free alternative route?
# Default: 200
# Unlimited: 0
max_choose_route_steps = 0

# size of catchment area of a station (default 2)
# older game size was 3
# savegames with another catch area will give strange results
station_coverage = 16

# Max number of steps in goods pathfinding
# This should be equal or greater than the biggest group
# of interconnected stations in your game.
#
# If you set it too low, some goods might not find a route
# if the route is too complex. If you set it too high, the
# search will take a lot of CPU power, particularly if searches
# often fail because there is no route.
#
# Depending on your CPU power, you might want to limit the search
# depth.
#
# prissi: On a 512x512 map with more than 150000 people daily, the saturation
# value for "no route" was higher, around 8000. Using 300 instead almost doubled
# the value of "no route"
#
max_hops = 2000

# Passengers and goods will change vehicles at most "max_transfer"
# times to reach their destination.
#
# It seems that in good networks, the number of transfers is below
# 5. I think 6 is a good compromise that allows complex networks but
# cuts off the more extreme cases
#
# You can set this lower to save a little bit of CPU time, but at some
# point this means you'll get less passengers to transport
#
# This value is less critical than "max_hops" from above.
#
# T. Kubes: I would say 8 for now, but this definitely should be difficulty
# dependent setting.
#
max_transfers = 12

# way builder internal weights (defaults)
# a higher weight make it more unlikely
# make the curves negative, and the waybuilder will built strange tracks ...
#
way_straight=1
way_curve=2
way_double_curve=6
way_90_curve=15
way_slope=10
way_tunnel=8
way_max_bridge_len=15
way_leaving_road=25

# These settings are used to calculate adjusted figures based on the length of the month.
# To assume a base month length based on the settings in Simutrans-Standard, use 1,000
# base_meters_per_tile and 18 base_bits_per_month.
#
# To assume a base month length of 24 hours (to allow various settings to be calibrated
# as if months were days), use a base meters per tile of 7500
base_meters_per_tile = 1000
base_bits_per_month = 18

# This setting determines the rate at which jobs replenish. At the default, 100, all
# of a building's jobs replenish in exactly 1 month. At 200, it takes 2 months, and at
# 50, it takes 1/2 a month. This is useful when calibrating the month/year scale as
# against the hour/minute scale. If a month is timed as 6.4 hours (6:24), for example,
# a day (24 hours) consists of 3.75 "months". For jobs to replenish once every 24 hours,
# therefore, set this to 375 if the length of a month is 6:24.
job_replenishment_per_hundredths_of_months = 100

# Save the current game when quitting and reload it upon reopening
reload_and_save_on_quit = 1

############################### Passenger and mail settings ##############################
# also pak dependent

# town growth multiplier factors
# The greater the factor, the greater the exent to which the thing to which the factor
# makes reference influences growth
passenger_multiplier = 40
mail_multiplier = 15
goods_multiplier = 20
electricity_multiplier = 20

# town growth is size dependent. There are three different sizes (<1000, 1000-10000, >10000)
# the idea is, that area increase by square but growth is linear
growthfactor_villages = 400
growthfactor_cities = 200
growthfactor_capitals = 100

# Enable this to use the old algorithm for city growth from Standard.
# NOTE: The renovation_percentage in cityrules.tab should be increased if this be done.
quick_city_growth = 0

# if enabled (default = 0 off) stops may have different capacities for passengers, mail, and  freight
seperate_halt_capacities = 0

# three modes (default = 0)
# 1: the payment is only relative to the distance to next interchange, 2 to the trips destination (default 0 is distance since last stop)
pay_for_total_distance = 0

# things to overcrowded destinations won't load if active (default off)
avoid_overcrowding = 0

# do not create goods/passenger/mail when the only route is over an overcrowded stop
no_routing_over_overcrowded = 0

# These settings determine the population, visitor demand, jobs and mail per "level" of building.
# Each of these things can be set independently in the buildings' .dat files, but for older paksets
# or paksets from Standard, only a "level" will be supplied, so these conversion factors are
# important in those cases.
population_per_level = 3
visitor_demand_per_level = 3
jobs_per_level = 2
mail_per_level = 1

# These settings determine the number of passenger trips that each person makes per game month
# and the number of items of mail that each unit of mail demand produces per month, in 1/100ths.
# This does *not* include onward and return trips, however, and is *before* adjustment for the
# meters per tile and bits per month scales.
passenger_trips_per_month_hundredths = 200
mail_packets_per_month_hundredths = 10

# This setting determines the maximum number of onward trips that passengers may make on a journey.
# The actual number of onward trips for any given packet of passengers is a random number of anywhere
# between 1 and this figure. This is only applicable if passengers are in fact going to make an onward
# trip, the distribution_weight of which is determined by a different setting (see below).
max_onward_trips = 3

# This figure determines how likely that it is that passengers will make any onward trips at all. It
# is expressed in percentage terms.
onward_trip_chance_percent = 25

# This is the distribution_weight, in percentage, that any given passenger journey will be a commuting trip. Any
# trip that is not a commuting trip is classed as a visiting trip.
commuting_trip_chance_percent = 67

# The following settings determine the way in which individual packets of passengers decide
# what their actual journey time tolerance is, within the above ranges. The options are:
#
# 0 - Even distribution
# Every point between the minimum and maximum is equally likely to be selected
#
# 1 - Normal distribution (http://en.wikipedia.org/wiki/Normal_distribution)
# Points nearer the middle of the range between minimum and maximum are more likely
# to be selected than points nearer the end of the ranges.
#
# 2 - Positively skewed normal distribution (squared) (http://en.wikipedia.org/wiki/Skewness)
# Points nearer the a point in the range that is not the middle but is nearer to the lower
# end of the range are more likely to be selected. The distance from the middle is the skew.
#
# 3 - Positively skewed normal distribution (cubed)
# As with no. 2, but the degree of skewness (the extent to which the modal point in the range
# is nearer the beginning than the end) is considerably greater.
#
# 4 - Positively skewed normal distribution (squared recursive)
# As with nos. 2 and 3 with an even greater degree of skew.
#
# 5 - Positively skewed normal distribution (cubed recursive)
# As with nos. 2, 3 and 4 with a still greater degree of skew.
#
# 6 and over - Positively skewed normal distribution (cubed multiple recursive)
# As with nos. 2, 3, 4, and 5 with an ever more extreme degree of skew. Use with caution.

random_mode_commuting = 2
random_mode_visiting = 2

################################## Industry settings #################################

# when a city reaches 2^n times of this number
# then a factory is extended, or a new factory chain is spawned
#industry_increase_every = 2000

# smallest distance between two adjacent factories
min_factory_spacing = 6

# max distance for connected factories
# if percentage>0, it will be in percent of the largest map dimension
# percentage also overrides the absolute value
max_factory_spacing_percentage = 25
#max_factory_spacing = 40

# allow all possible supplier to connect to your factories?
# best to leave it in default position. (only on for simuTTD)
crossconnect_factories = 0

# how big is the distribution_weight for crossconnections in percent
# (100% will give nearly the same results as crossconnect_factories=1)
crossconnect_factories_percentage = 33

# how much is the total electric power available (in relation to total production) in parts per thousand
electric_promille = 1100

# true if transformers are allowed underground (default)
allow_underground_transformers = 1

# with this switch on (default), overcrowded factories will not recieve goods any more
just_in_time = 1

# How much amount in transport is sent before further distribution stops
# This is only enabled when "just_in_time" is enabled
# The limit is given in percent of factory storage (0=off)
#
# This number is (as of Simutrans-Extended 11.16) scaled to the ratio
# of the time that it takes the industry to consume its stock to the average
# lead time for new deliveries. Values of slightly over 100% are recommended.
maximum_intransit_percentage = 110

# use beginner mode for new maps (can be switched anyway on the new map dialog)
first_beginner = 0

# number of periods for averaging the amount of arrived pax/mail at factories for boost calculation
# one period represents a fixed interval of 2^18 ms in-game time
# value can range from 1 to 16, inclusive; 1 means no averaging; default is 4
factory_arrival_periods = 4

# whether factory's pax/mail demands are enforced or not; default is on
factory_enforce_demand = 1

################################# Display settings ################################

# if defined, the default tool will try to calculate straight ways (like OpenTTD)
straight_way_without_control = 0

# player color can be fixed for several players or set totally random
# in the latter case each player will get unique but random coloring
# (default 0)
random_player_colors = 0

# when set here, the player will always get these player colors, even
# if random colors were enabled.
# color values range from 0 to 27 for first and second player color
# and there are 0...15 player in the game
player_color[1] = 1,4

# remove companies without convois after x month (0=off, 6=default)
remove_dummy_player_months = 6

# remove password of abandoned companies (wihtout any building activity) after x month (0=off default)
unprotect_abondoned_player_months = 0

# how long is a diagonal (512: factor 2=1024/512, old way, 724: sqrt(2)=1024/724
# THIS WILL BE ONLY USED, WHEN THIS FILE IS IN THE pakxyz/config!
#diagonal_multiplier = 724

# how height is a tile in z-direction (default 16, TTD 8)
# THIS WILL BE ONLY USED, WHEN THIS FILE IS IN THE pakxyz/config!
#tile_height = 16

# (=1) drive on the left side of the road
drive_left = 0

# (=1) signals on the left side
signals_on_left = 0

# Do you want to have random pedestrians in town? Look nice but needs some
# CPU time to process them. (1=on, 0=off)
# Impact on frame time: ~10% (16 cities on a new standard map)
random_pedestrians = 1

# Do you want to have random pedestrians after pax are reaching this
# destination? May generate quite a lot. (1=on, 0=off)
stop_pedestrians = 1

# there are some other grounds (like rocky, lakes etc. )
# which could be added to the map (default 10)
# show random objects to break uniformity (every n suited tiles)
random_grounds_probability = 10

# show random moving animals (n = every n suited tiles, default 1000)
random_wildlife_probability = 1000

# animate the water each intervall (if images available)
# costs some time for the additional redraw (~1-3%)
water_animation_ms = 250

# Show info windows for private cars and pedestrians?
# (1=on, 0=off)
pedes_and_car_info = 0

# How much citycars will be generated
citycar_level = 5

# After how many month a citycar breaks (and will be forever gone) ...
# default is ten years
default_citycar_life = 36

# Show infos on trees?
# (1=on, 0=off)
tree_info = 1

# Show infos also on bare ground?
# (1=on, 0=off)
ground_info = 1

# Show passenger level of townhalls?
# (1=on, 0=off)
townhall_info = 1

# do not show the button to add an inverted schedule for rail based convois
# (1=hide, 0=show anyway)
hide_rail_return_ticket = 1

# always open only a single info window for the ground,
# even if there are more objects on this tile
#only_single_info = 1

# show a tooltip on convois at several conditions
# 0 no messages
# 1 (default) only no_route and stuck
# loading and waiting at signals too
#show_vehicle_states = 1

# show (default) tiles with a halt when editing a schedule
#visualize_schedule = 1

# Should stations get numbered names? (1=yes, 0=no)
#numbered_stations = 0

# Show name signs and statistic?
# 0 = don't show anything
# 1 = station names
# 2 = statistics
# 3 = names and statistics
# The visual style is added to this number:
#   0 = black name in color box
#   4 = name in player color with outline
#   8 = box left of name in yellow outline
show_names = 3

# Color of cursor overlay, which is blended over marked ground tiles
# The available colors and their numbers can be found on
#    http://simutrans-germany.com/wiki/wiki/tiki-index.php?page=en_FactoryDef#mapcolor
# Suggested values (155 is the default)
# -- pak64, pak.german, pak128
#cursor_overlay_color = 155
# -- pak128.japan
#cursor_overlay_color = 149

# Color of background default: COL_GREY2 (210)
#background_color = 210

# there are three different ways to indicate an active window

# first: draw a frame with titlebar color around active window
#window_frame_active = 0

# second: draw the title with a different brighness (0: dark ... 6: bright)
front_window_bar_color = 1
bottom_window_bar_color = 3

# third (best together with 2nd):use different text color for bar
# some colors are 215-white, 240-blck 208-214- all shades of gray
front_window_text_color = 215
bottom_window_text_color = 209

# when moving, you can use windows to snap onto each other seamlessly
# if you do not like it, set the catch radius to zero
window_snap_distance = 8

# show tooltips (default 1=show)
show_tooltips = 1

# tooltip background color (+-1 arounf this index is used), taken from playercolor table
tooltip_background_color = 4

# tooltip text color (240=black, 215=white)
tooltip_text_color = 240

# delay before showing tooltip in ms (default 500ms)
tooltip_delay = 500

# duration in ms during tooltip is visible (default 5000ms=5s)
tooltip_duration = 5000

# show graphs old style (right to left) or rather left to right (default)
left_to_right_graphs  = 1

# if run on a mobile device, setting hide_keyboard=1 will only show the keyboard
# when there is text to input in a dialoge. Other than that textinput will not
# be possible (no keyboard short cuts).
# Currently only supported with SDL2
hide_keyboard = 0

# 1=top, 2=vertical centre, 3=bottom, 4=left, 8=horizontal centre, 12=right
# default for minimap is 1+12=13
compass_screen_position = 0

# Should either account (default) or net wealth be shown blow the screen?
player_finance_display_account = 1

################################### Finance settings ##################################
#
# These values are usually set in the pak files
# You can adjust all the cost in the game, that are not inside some pak file
#

# Starting money of the player. Given in Credit cents (1/100 Cr)
#starting_money = 20000000

# New system of year dependent starting money. Up to ten triplets are
# possible. The entries are of the form:
# startingmoney[i]=year,money(in 1/10 credits),interpolate (1) or step(0)
# starting_money[0]=1930,20000000,1
# starting_money[1]=2030,35000000,1

# allow buying obsolete vehicles (=1) in depot
allow_buying_obsolete_vehicles = 1

# vehicle can loose a part of their value, when the are once used
# the loss is given in 1/1000th, i.e 300 mean the value will be 70%
used_vehicle_reduction = 0

# lowest possible income with speedbonus (1000=1) default 125 (=1/8th)
bonus_basefactor = 125

# if a convoi runs on a way that belongs to another player, toll may
# be charged. The number given is the percentage of the running cost
# of the convoi or the way cost (include electrification if needed).
# (default 0)
toll_runningcost_percentage = 0
toll_waycost_percentage = 0

# Maintenance costs of buildings
#maintenance_building = 2000

# first stops: the actual cost is (cost*level*width*height)
#cost_multiply_dock=500
#cost_multiply_station=600
#cost_multiply_roadstop=400
#cost_multiply_airterminal=3000
#cost_multiply_post=300
#cost_multiply_headquarter=1000

# cost for depots
#cost_depot_air=5000
#cost_depot_rail=1000
#cost_depot_road=1300
#cost_depot_ship=2500

# other construction/destruction stuff
#cost_buy_land=100
#cost_alter_land=1000
#cost_set_slope=2500
#cost_found_city=5000000
#cost_multiply_found_industry=20000
#cost_remove_tree=100
#cost_multiply_remove_haus=1000
#cost_multiply_remove_field=5000
#cost_transformer=2500
#cost_maintain_transformer=20

# in beginner mode, all good prices are multiplied by a factor (default 1500=1.5)
beginner_price_factor = 1500

 # The number of months of maintainance that the make public tool costs to use
cst_make_public_months = 60

################################### Miscellaneous settings ##################################
#
# also pak dependent
#

# minimum distance between two townhalls
#minimum_city_distance = 16

# Minimum distance of a city attraction to other special buildings
#special_building_distance = 3

# Max. length of initial intercity road connections
# If you want to speed up map creation, lower this value.
# If you want more initial intercity roads, raise this value.
# If the value is too large then very long bridges might be created.
# 200 seems to be a good compromise between speed and road number
# note: this will slow down map creation dramatically!
#
#intercity_road_length = 200

# This is the maximum number of tiles that a road, river or canal
# that is a public right of way may be diverted for an existing
# public right of way to be deleted. Diversion allows players to
# change the course of public rights of way to accomodate, for
# example, railways whilst protecting their integrity.

max_diversion_tiles = 16

# This is the fraction of the way's total wear capacity below which
# the way will count as degraded and be automatically renewed or,
# if the player has insufficient money or auto-renewal has been
# disabled for the way in question, will enter a degraded state
# in which the speed limit will be reduced. (At a state of 0,
# the way will become totally impassable).
# Default: 7 (approx. 14%).

way_degradation_fraction = 7

# These two settings determine the default relationship between
# the weight of vehicles and their way wear factors. Air and
# road vehicles use the "road_type", and all others apart from
# maglev and water (which are hard coded to zero) use the
# "rail_type". This only applies to vehicles whose way wear
# factor is not specified in the individual vehicle definitions.
# The default for road is 4, based on the "forth power law":
# http://www.pavementinteractive.org/article/equivalent-single-axle-load/
# The default for rail is 1.

way_wear_power_factor_road_type = 4
way_wear_power_factor_rail_type = 1

# This is the setting to calibrate the way wear system. This
# is only effective for vehicles which do not have their way
# wear factor set in their individual .dat files. For an
# explanation of the standard axle load (in tonnes), see
# the link above.
# The default is 8, which is the UK standard for road.

standard_axle_load = 8

# This is the way wear factor exerted on roads by all
# "citycars" (that is, the automatically generated
# private road traffic) in the game, measured in
# 1/10,000ths of a standard axle load.
# Default = 2

citycar_way_wear_factor = 2

# Type of intercity roads - must be available as PAK file.
# Intercityroad with timeline: intercity_road[number]=name,intro-year,retire-year
# .. number = 0..9 - up to ten different intercity_roads possible
# .. name = name of an existing pak file
#intercity_road[0] = asphalt_road,0,1990
# default: city_road
# (old, 102.2.2 and before) intercity_road_type = asphalt_road

# Type of city roads - must be available as PAK file.
# Cityroad with timeline: city_road[number]=name,intro-year,retire-year
# .. number = 0..9 - up to ten different city_roads possible
# .. name = name of an existing pak file
#city_road[0] = dirt_road,1920,1940
# default: asphalt_road
# (old, 102.2.2 and before) city_road_type = city_road

# now river stuff
# first river type (should be defined in pak dependent file)
# The highest number is the smallest. A river with max_speed==0 is not navigavable by ships.
#river_type[0] = river
#river_type[1] = small_river
#river_type[2] = just_the source

# river number (16 gives a few nicely branched rivers)
#river_number = 16

# min length
#river_min_length = 16

# max length
#river_max_length = 320

# This is the distance in meters at which train drivers can see signals ahead.
# Trains have to brake in time for signals which might be at danger, so this
# distance affects train speed.
sighting_distance_meters = 250

# This is the maximum speed at which rail (including narrow gauge, monorail and maglev)
# vehicles may travel in the drive by sight working method. 0 = as fast as they can
# stop in time with no other limit. Note that this does not apply to trams.

max_speed_drive_by_sight_kmh = 35

# The following settings apply to time interval signalling only.
#
# time_interval_seconds_to_clear is the time, in seconds, after a train has completely passed a time interval
# signal that it will reset its aspect to clear.
#
# time_interval_seconds_to_caution is the time, in seconds, after a train has completely passed a time interval
# signal that it will reset its aspect to caution.
#
# Default: clear - 600 (10 minutes); caution - 300 (5 minutes)

time_interval_seconds_to_clear = 600
time_interval_seconds_to_caution = 300

# Corners of greater than 45 degrees have their radius calculated. However, because of
# the rigid tile system in Simutrans, this produces unrealistic results when applied
# to corners of 45 degrees. A value is thus specified here. This affects the speed
# at which vehicles can take 45 degree corners.
#
# If this is set to 0, 45 degree corners are treated as straight.
#
# If this is set to 9999, adjacent pairs of degree corners are treated as half the
# radius of a 90 degree corner (and non-adjacent pairs scaled according to the distance
# between them). This is techincally correct but does not work well in Simutrans because
# of the inability to have gentler corners.
#
assumed_curve_radius_45_degrees = 1000

# Towns have speed limits. This value, expressed in km/h, is the speed limit
# value for urban roads in the game.
# Default: 50
# Set to 0 to disable town speed limits (roads will have their base speed limits in towns).
# This only applies to roads.

town_road_speed_limit = 50

# This allows for height maps with more height levels
new_height_map_conversion = 0

# disable companies from making ways public with the appropiate tool
# even when disabled companies can still make stops public
# does not affect public service provider player
disable_make_way_public = 0

################################# Tree settings #################################
#  please be careful in changing them, I spent lot of time finding optimals.
#  those values have impact on no. of spawned trees -> memory consumption
#
# Number of trees on square 2 - minimal usable, 3 good, 4 very nice looking
max_no_of_trees_on_square = 3

# Base forest size - minimal size of forest - map independent
forest_base_size = 36

# Map size divisor - smaller it is the larger are individual forests
forest_map_size_divisor = 38

# Forest count divisor - smaller it is, the more forest are generated
forest_count_divisor = 16

# Determins how dense are spare trees going to be planted (works inversly)
forest_inverse_spare_tree_density = 400

# climate with trees entirely (1: water, 2:desert, 4:tropic, 8:mediterran, 16:temperate, 32:tundra, 64:rocky, 128:arctic)
# zero (default) means no climate with at least one tree per tile
tree_climates = 4

# climates with no trees at all (desert and arctic at the moment)
no_tree_climates = 130

# if set, no trees will be created at all (save about 30% memory and
# the season change will be much smoother on small machines)
#no_tree = 0

################################### Time settings ###################################

# Enforce vehicle introduction dates?
# 0 = all vehicles available from start of the game
# 1 = use introduction dates
# 2 = (default) use settings during game creation, new games off
# 3 = use settings during game creation, new games on
#
use_timeline = 3

# Starting year of the game:
# Setting it below 1850 is not recommended for 64 set!
# You will have problems with missing vehicles, do not complain if you do so!
# Setting it above 2050 will render game bit boring - no new vehicles.
#
# other recommended vaule for 64 is 1956
#
starting_year = 1930

# Starting month of the game for people who want to start in summer (default 1=January)
starting_month = 1

# Should month be shown in date?
# (0=no, 1=yes, 2>=show day in japan format=2, us format=3, german=4, japanese no season=5, us no season=6, german no season = 7, hours/minutes scale = 8)
# This is most useful, if you use longer months than the default length (see below)
# The hours/minutes scale shows the time in hours/minutes as used for determining journey times and other short times. It is the recommended setting.
show_month = 8

# Global time multiplier (will be save with new games)
# 2^bits_per_month = duration of a game month in milliseconds real time
# default before 99.x was 18. For example, 21 will make the month 2^3=8 times longer in real time
# production and maintainance cost will be adjusted accordingly.
#
bits_per_month = 20

################################# System settings #################################

# compress savegames?
# "binary" means uncompressed, "zipped" means compressed
# "bzip2" uses another compression algorithm
# other options are "xml", "xml_zipped" and "xml_bzip2"
# xml detects more errors of broken savegames but files are much larger
# bzip2 savegames are smaller than zipped but saving/loading takes longer
saveformat = zipped

# Alternate format for faster autosaves
autosaveformat = zipped

# autosave every x months (0=off)
autosave = 0

# display (screen/window) width
# also see readme.txt, -screensize option
#display_width  = 704

# display (screen/window) height
# also see readme.txt, -screensize option
#display_height = 560

# show full screen
fullscreen = 0

# For versions of Simutrans compiled to work in a multithreaded system (which will improve
# certain aspects of performance on multiple core/processor machines), this is the number
# of threads that will be used. Maximum: 12.
threads = 4

# maximum size of tool bars (0 = no limit)
# if more tools than allowed by height,
# next and prev arrows for scrolling appears
toolbar_max_width = 0
toolbar_max_height = 0

# How many frames per second to use? Display may look pretty until 10 or so
# (depends very much on computer, game complexity and graphics driver)
frames_per_second = 15

# during zooming out simutrans may get slow due to the very high number
# of tiles visible. If the tiles become equal or smaller than the tile size
# below, a simpler clipping algorithm will be used, which will give some
# clipping errors, but is faster. (default size = 4)
# However, during nromal gaming this will be detrimined automatically, so you
# usually you do not need to set this manually.
simple_drawing_tile_size = 4

# you can force fast redraw for fast froward by this (default off)
simple_drawing_fast_forward = 1

# How much faster should the game proceed with fast forward (limited by your computer and size of the map)
fast_forward = 100

################################### Network settings ##############################
#
# Synchronized networking is always a trade off between fast respone and safe
# connections. A more relaxed timing will cause delay of commands but is more
# likely to compensate for clients running slightly faster than the rest.
#

# Sets the local addresses Simutrans should listen on and use for making outgoing connections
# By default it will use all local IPv4 and IPv6 addresses
# This setting has no effect if Simutrans has been compiled with the USE_IP4_ONLY flag set!
# The addresses listd will be tried in the order specified
# A DNS name may be specified, this will be resolved and Simutrans will attempt to listen
# on all of the addresses returned.
listen = 46.32.231.222

# How much delay before comands are executed on the clients.
# A larger number will catch even clients running slightly ahead but cause delay.
# This is set by the server side.
#
# The sum of server_frames_ahead and additional_client_frames_behind should be
# evenly divisble by server_frames_between_checks for best network performance.
server_frames_ahead = 4

# How much extra delay in command execution on the client side, on top of server_frames_ahead.
# A larger number can compensate for larger fluctuations in communication latency.
# This is set by the client side.
#
# The sum of server_frames_ahead and additional_client_frames_behind should be
# evenly divisble by server_frames_between_checks for best network performance.
additional_client_frames_behind = 0

# In network mode, there will be a fixed number of screen updates before a step.
# Reasonable values should result in 2-5 steps per second.
#
# This is the number of sync steps for every step. Sync steps handle user interaction
# and things that update regularly: steps handle things that take much computational
# effort, such as routing. Each step takes much more time than each sync step.
# This setting is only active in network mode: the timing is automated in single
# player mode.
server_frames_per_step = 8

# The server sends after a fixed number of steps some information to the clients.
# Large values here means: reduced server communication (if that is of importance...)
# Small values should improve the timing of the clients.
server_frames_between_checks = 32

# Automatically announce server on the central server directory (http://servers.simutrans.org/)
# 0 (default) = off, 1 = on
server_announce = 1

# Interval of server announcement (if enabled)
# Value is number of seconds between server announcements, default is 900 (15 minutes)
# Minimum value is 60 (1 minute), for accurate listing it is recommended not to increase
# this value to greater than 3600 (1 hour)
# To disable announcements set server_announce to 0
server_announce_interval = 900

# Fully Qualified Domain Name (FQDN) or IP address of your server (IPv6 or IPv4)
server_dns = bridgewater-brunel.me.uk

# Name of server in server listing
server_name = Bridgewater-Brunel 1

# Additional information about your server (for display on the list server)
server_comments = A server for online play of large maps with the latest versions of Simutrans-Extended

# Email address of server maintainer (for display on the list server)
server_email = [REDACTED]

# The password required for administering the server.
# NOTE: This should be changed from the default when used, for
# obvious reasons.
server_admin_pw  = [REDACTED]

# Pakset download URL (for display on the list server)
server_pakurl = http://bridgewater-brunel.me.uk/downloads/nightly/pakset/pak128.britain-ex-nightly.tar.gz

# Server info URL (for display on the list server)
server_infurl = http://www.bridgewater-brunel.me.uk

# Pause server when no clients are connected
pause_server_no_clients = 1

# Nickname when joining network games
nickname = Minister of Transport

# Server saves savegame when being killed (default=0 off)
server_save_game_on_quit = 1

# Chat window transparency (0=off, 25, 50 75 are possible)
chat_transparency = 75

# The number of game months before a player making no changes
# to its company is unlocked automatically to allow other players
# to take over.
unprotect_abandoned_player_months = 180

# The number of game months before a player that has never built
# anything substantive is deleted automatically.
remove_dummy_player_months = 24

# Here you can add a message about your server (It will read this file on each joining anew)
server_motd_filename = motd.txt

Offline ACarlotti

  • *
  • Posts: 405
Re: Instability on the Bridgewater-Brunel server
« Reply #28 on: September 25, 2018, 08:54:33 AM »
I've pushed a further improvement for partial saves to Github, which was discussed for Standard here. The improvment is that the temporary filename is derived from the given filename, making it safe to run multiple instances of Simutrans on one filesystem simultaneously.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #29 on: September 25, 2018, 10:12:11 PM »
Thank you for that - I had not thought of that issue, but this is a sensible improvement.

Now incorporated.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #30 on: September 27, 2018, 07:41:34 PM »
I have begun some testing in relation to the desync bug. The first test was to run a local client/server pair on the saved game from the 16th of September (early 1937 in game) with the latest cross-compiled release build from the nightly server. The result of this test, just completed, was that this ran for over an hour without losing synchronisation.

The next step is to try with a later saved game. I note, incidentally, that a 21st of September saved game will crash the release build, but I cannot reproduce this in the debug build.

Edit: The second round of testing with a saved game from earlier to-day also failed to show any desync when run in a server/client pair locally for about an hour (until October 1940 in-game).

Can all those who are having desync problems let me know what operating systems that they are using? It is possible that the issue is occurring only when Windows clients are connected to a Linux server.
« Last Edit: September 27, 2018, 08:55:39 PM by jamespetts »

Offline Ves

  • Devotee
  • *
  • Posts: 1638
  • Languages: EN, SV, DK
Re: Instability on the Bridgewater-Brunel server
« Reply #31 on: September 27, 2018, 09:04:38 PM »
I notice alot of talk about the overtaking patch and that it might be the issue.
When I made my airports, I utilized that feature for some of the bustations to the airports, particularly the "parallel stop mode". That was one of the last things I did before it became more unstable and I had to do other real world stuff.

I could try to rebuild those sections with just normal  way mode and see if that helps. That is, if I ever get on to the server....
Edit:
Got on the server and rebuilt all such stretches that I remember I have built.

Although currently connected and not yet desynced, it happened earlier the previous weeks, and I run Windows 10 with the 64 bit executor. The 32 bit executor has been unable to connect to the servergame without getting an out of memory within seconds.

edit2:
Just desynced....
« Last Edit: September 27, 2018, 09:18:13 PM by Ves »

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 563
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #32 on: September 27, 2018, 09:13:32 PM »
I have been using the parallel stop mode rather extensively since it was introduced. It seemed to work fine.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18502
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #33 on: September 27, 2018, 09:33:57 PM »
Thank you both for your feedback - that is useful. The 32-bit executable will not work as the game takes too much memory for this to be usable.

I have tried just now to connect with the latest build to see for how long that I can stay connected, but I had blue screen crashes on two separate occasions of trying, despite having replaced my PSU (with visibly melted capacitors) earlier in the week. I suspect that I may need a more comprehensive computer upgrade before carrying out intensive work on Simutrans, which might take a number of months to prepare and execute.

Ves - thank you for the test, but it seems from your results and Rollmaterial's later observations that this test is inconclusive (the loss of synchronisation still occurring being not inconsistent with the problem either being or not being with the new overtaking code, which is capable of having effect even if the features are not explicitly used).

If anyone could check whether it is possible to stay in sync for an hour or so with a Linux client, that would be very helpful.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2617
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #34 on: September 28, 2018, 03:12:41 AM »
The overtaking code appears to be typecasting to "(volatile uint8)" in 4 places.
Quote
'type' : top-level volatile in cast is ignored
The compiler detected a cast to an r-value type which is qualified with volatile, or a cast of an r-value type to some type that is qualified with volatile. According to the C standard (6.5.3), properties associated with qualified types are meaningful only for l-value expressions.
I suspect from a compiler perspective this makes no sense. Volatile is used to annotate how the compiler should access a storage location, not a value.
Code: [Select]
for(  uint8 pos=1;  pos<(volatile uint8)sg[i]->get_top();  pos++  ) {The value returned by get_top is a value. This value cannot be volatile as it is a value so has no associated storage location to manipulate as specified by volatile.

The fact volatile is being used there raises alarm bells of possible race conditions.

I also spotted the following which might or might not be bad.
Code: [Select]
Severity Code Description Project File Line Suppression State
Warning C4701 potentially uninitialized local variable 'btyp' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\descriptor\reader\building_reader.cc 637
Warning C4701 potentially uninitialized local variable 'f_desc' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\display\font.cc 220
Warning C4701 potentially uninitialized local variable 'f_height' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\display\font.cc 219
Warning C4701 potentially uninitialized local variable 'g_desc' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\display\font.cc 91
Warning C4701 potentially uninitialized local variable 'h' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\display\font.cc 91
Warning C4701 potentially uninitialized local variable 'g_width' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\display\font.cc 127
Warning C4701 potentially uninitialized local variable 'image' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\gui\components\gui_convoy_assembler.cc 1226
Warning C4701 potentially uninitialized local variable 'image' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\gui\components\gui_convoy_assembler.cc 1484
Warning C4701 potentially uninitialized local variable 'byte_length' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\gui\components\gui_textinput.cc 187
Warning C4701 potentially uninitialized local variable 'byte_length' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\gui\components\gui_textinput.cc 216
Warning C4701 potentially uninitialized local variable 'n' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\gui\settings_stats.cc 1113
Warning C4701 potentially uninitialized local variable 'to' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\vehicle\simroadtraffic.cc 223
Warning C4701 potentially uninitialized local variable 'ri' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\simsys_s2.cc 231
Warning C4701 potentially uninitialized local variable 'cost' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\simtool.cc 5041
Warning C4701 potentially uninitialized local variable 'gd' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\vehicle\simvehicle.cc 2276
Warning C4701 potentially uninitialized local variable 'gd' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\vehicle\simvehicle.cc 2254
Warning C4701 potentially uninitialized local variable 'this_direction' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\vehicle\simvehicle.cc 3711
Warning C4701 potentially uninitialized local variable 'station_signal' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\vehicle\simvehicle.cc 7128
Warning C4701 potentially uninitialized local variable 'this_stop_signal_index' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\vehicle\simvehicle.cc 7197
Warning C4701 potentially uninitialized local variable 'next_next_signal' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\vehicle\simvehicle.cc 6879
Warning C4701 potentially uninitialized local variable 'tolerance' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\simworld.cc 6292
Warning C4701 potentially uninitialized local variable 'walking_time' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\simworld.cc 7108
Warning C4701 potentially uninitialized local variable 'car_minutes' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\simworld.cc 7261
Warning C4701 potentially uninitialized local variable 'best_journey_time' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\simworld.cc 6801
Warning C4701 potentially uninitialized local variable 'ts' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\squirrel\sqstdlib\sqstdstring.cc 138
Warning C4701 potentially uninitialized local variable 'ti' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\squirrel\sqstdlib\sqstdstring.cc 139
Warning C4701 potentially uninitialized local variable 'tf' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\squirrel\sqstdlib\sqstdstring.cc 140
Warning C4701 potentially uninitialized local variable 'name' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\gui\times_history.cc 113
Warning C4701 potentially uninitialized local variable 'best' used Simutrans-Extended Normal d:\simutransbuild\simutrans extended\simutrans-extended\bauer\vehikelbauer.cc 306
Obviously some of them might logically always be initialized. However does that apply to all of them? Uninitialized values leaking in and modifying behaviour could easily cause out of sync errors.