News:

Beta test the new forum at https://simutrans.forum/
Use the "Forum Search"
It may help you to find anything in the forum ;).

"Lost synchronisation with server" report thread

Started by freddyhayward, September 24, 2020, 08:48:53 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

zook2

I think I was just sending a "no Route/too complex" ship to another waypoint when I crashed:

Warning: sint64 convoi_t::calc_revenue:   Average speed (33) for (9411) Fly-boat horses exceeded maximum speed (18); falling back to overall average
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (616) Friesian horses  (four-in-hand) with short stage coach at Sherhill Town Hall Stop
Warning: nwc_tool_t::rdwr:   rdwr id=8 client=0 plnr=8 pos=4231,1469,0 tool_id=8216 defpar=f,10547 init=1 flags=0
Warning: nwc_tool_t::rdwr:   rdwr id=8 client=2 plnr=8 pos=4231,1469,0 tool_id=8216 defpar=f,10547 init=1 flags=0
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 14709.975 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 400000 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 5883.98999 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 400000 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
Warning: sint64 convoi_t::calc_revenue:   Average speed (19) for (4451) Friesian horses (pair) exceeded maximum speed (12); falling back to overall average
Warning: sint64 convoi_t::calc_revenue:   Average speed (19) for (4451) Friesian horses (pair) exceeded maximum speed (12); falling back to overall average
Warning: sint64 convoi_t::calc_revenue:   Average speed (13) for (4451) Friesian horses (pair) exceeded maximum speed (12); falling back to overall average
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 14709.975 / 0
<snip>

https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 14709.975 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 400000 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 5883.98999 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 400000 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 14709.975 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 400000 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 5883.98999 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 400000 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (3828) Friesian horses (four-in-hand) with strap stage coach at Zenlyn Lilac Road Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (9352) Shire horses (pair) at Wallow River Markets & DIY
Warning: void convoi_t::laden():   (6473) Yorkshire coach horses (six-in-hand) with braked mail coach trying to load at Gwesty'r Dwycawl Abbey when not at a halt; rerouting
Warning: nwc_tool_t::rdwr:   rdwr id=8 client=0 plnr=8 pos=4226,1471,1 tool_id=8216 defpar=V,10547 init=1 flags=0
Warning: nwc_tool_t::rdwr:   rdwr id=8 client=2 plnr=8 pos=4226,1471,1 tool_id=8216 defpar=V,10547 init=1 flags=0
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (7679) Shire horses (pair) at Fen Sixdale Hardware & DIY
Warning: sint64 convoi_t::calc_revenue:   Average speed (19) for (8461) Yorkshire coach horses (pair) exceeded maximum speed (18); falling back to overall average
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (3310) Hackney horses (pair) at Pewton Monument Stop
Warning: nwc_routesearch_t::rdwr:   rdwr limits=(1024, 98304, 34880, 16777216, 256) apply_limits=0
Warning: karte_t::interactive:   nwc_routesearch_t object created and sent to server: sync_step=59977 map_counter=8671108 limits=(1024, 98304, 34880, 16777216, 256)
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (3795) Friesian horses (four-in-hand) with strap stage coach at Zenlyn County Hall Stop
Warning: nwc_routesearch_t::rdwr:   rdwr limits=(1024, 98304, 34880, 5558528, 256) apply_limits=0
Warning: karte_t::interactive:   nwc_routesearch_t object created and sent to server: sync_step=59990 map_counter=8671108 limits=(1024, 98304, 34880, 5558528, 256)
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (10052) Fly-boat horses at Kingserpool Bridge dock
Warning: nwc_routesearch_t::rdwr:   rdwr limits=(1024, 98304, 34880, 5558528, 212672) apply_limits=0
Warning: karte_t::interactive:   nwc_routesearch_t object created and sent to server: sync_step=60000 map_counter=8671108 limits=(1024, 98304, 34880, 5558528, 212672)
Warning: void convoi_t::laden():   (6473) Yorkshire coach horses (six-in-hand) with braked mail coach trying to load at Gwesty'r Dwycawl Abbey when not at a halt; rerouting
Warning: void convoi_t::laden():   (6201) Irish draught horses (pair) trying to load at Moel-y-Groes Dairy & Pub when not at a halt; rerouting
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (3213) Friesian horses (four-in-hand) with strap stage coach at Zenlyn Broadway Inn
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (4133) Friesian horses (four-in-hand) with short stage coach at Safle Dwycawl Trindod Sanctaidd
Warning: nwc_routesearch_t::rdwr:   rdwr limits=(1024, 26584, 22144, 5558528, 212672) apply_limits=1
Warning: nwc_routesearch_t::execute:   to be applied: limits=(1024, 26584, 22144, 5558528, 212672)
Warning: nwc_routesearch_t::execute:   add to world command queue (sync step: command=60057 world=60052)
Warning: nwc_routesearch_t::do_command:   apply limits=(1024, 26584, 22144, 5558528, 212672)
Warning: sint64 convoi_t::calc_revenue:   Average speed (26) for (8469) Yorkshire coach horses (pair) exceeded maximum speed (18); falling back to overall average
Warning: sint64 convoi_t::calc_revenue:   Average speed (20) for (8469) Yorkshire coach horses (pair) exceeded maximum speed (18); falling back to overall average
Warning: sint64 convoi_t::calc_revenue:   Average speed (20) for (8469) Yorkshire coach horses (pair) exceeded maximum speed (18); falling back to overall average
Warning: sint64 convoi_t::calc_revenue:   Average speed (20) for (8469) Yorkshire coach horses (pair) exceeded maximum speed (18); falling back to overall average
Warning: sint64 convoi_t::calc_revenue:   Average speed (26) for (8469) Yorkshire coach horses (pair) exceeded maximum speed (18); falling back to overall average
Warning: sint64 convoi_t::calc_revenue:   Average speed (26) for (8469) Yorkshire coach horses (pair) exceeded maximum speed (18); falling back to overall average
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (8342) Yorkshire coach horses (pair) at Pewton Town Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (5416) Cleveland bay horses (six-in-hand) with mail coach at Safle Neuadd Plwyf Y Maesffordd | Maesffordd Parish Hall Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (8502) Friesian horses (four-in-hand) with short stage coach at Sherhill Town Hall Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (4763) Friesian horses (pair) at Neddbury Jones Gardens Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (8624) Yorkshire coach horses (pair) at Wallow Hill Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (3940) Friesian horses (four-in-hand) with short stage coach at Moel-y-Groes Parish Hall Stop
Warning: sint64 convoi_t::calc_revenue:   Average speed (34) for (9861) Fly-boat horses exceeded maximum speed (18); falling back to overall average
Warning: sint64 convoi_t::calc_revenue:   Average speed (34) for (9861) Fly-boat horses exceeded maximum speed (18); falling back to overall average
Warning: sint64 convoi_t::calc_revenue:   Average speed (34) for (9861) Fly-boat horses exceeded maximum speed (18); falling back to overall average
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (939) Friesian horses (six-in-hand) with rigid stage coach at Pollveor East Bank Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (7249) Friesian horses (four-in-hand) with short stagecoach at Safle Dwycawl Trindod Sanctaidd
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (4393) Friesian horses (pair) at Norburn Balgownie Street Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (9644) Friesian horses (pair) at Norburn dock
Warning: nwc_routesearch_t::rdwr:   rdwr limits=(1024, 98304, 18680, 5558528, 212672) apply_limits=0
Warning: karte_t::interactive:   nwc_routesearch_t object created and sent to server: sync_step=60194 map_counter=8671108 limits=(1024, 98304, 18680, 5558528, 212672)
Warning: sint64 convoi_t::calc_revenue:   Average speed (17) for (6563) Cleveland bay horses (six-in-hand) with braked stage coach exceeded maximum speed (14); falling back to overall average
Warning: nwc_routesearch_t::rdwr:   rdwr limits=(1024, 98304, 18680, 1660672, 212672) apply_limits=0
Warning: karte_t::interactive:   nwc_routesearch_t object created and sent to server: sync_step=60205 map_counter=8671108 limits=(1024, 98304, 18680, 1660672, 212672)
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (10522) Tayleur express engine 2-2-2 at Finerminster Princess Loreley Dock
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (9454) Friesian horses (pair) at Norburn Balgownie Street Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (5520) Friesian horses (four-in-hand) with short stage coach at Philron Dukesgate Stop
Warning: sint64 convoi_t::calc_revenue:   Average speed (18) for (4418) Short stage coach exceeded maximum speed (12); falling back to overall average
Warning: haltestelle_t::liefere_an():   6 Passengers delivered to Tannacaber Fields dock were intended for a building that has been deleted.
Warning: nwc_routesearch_t::rdwr:   rdwr limits=(1024, 35445, 18680, 1660672, 212672) apply_limits=0
Warning: karte_t::interactive:   nwc_routesearch_t object created and sent to server: sync_step=60252 map_counter=8671108 limits=(1024, 35445, 18680, 1660672, 212672)
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (3838) Friesian horses (four-in-hand) with short stage coach at Zenlyn Canal Street Quay
Warning: nwc_routesearch_t::rdwr:   rdwr limits=(1024, 35445, 34278, 1660672, 212672) apply_limits=0
Warning: karte_t::interactive:   nwc_routesearch_t object created and sent to server: sync_step=60265 map_counter=8671108 limits=(1024, 35445, 34278, 1660672, 212672)
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (4263) Short stage coach at Pewton Town Stop
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (6396) Shire horses (pair) at Hayford Princess Victoria (Vicky) Railway Station
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (10469) Fly-boat horses at Finerminster Town dock
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (7376) Shire horses (pair) at Zenlyn Royal Railway Station of the Century
Warning: sint64 convoi_t::calc_revenue:   Average speed (12) for (1246) Norfolk wherry (passengers) exceeded maximum speed (10); falling back to overall average
Warning: nwc_routesearch_t::rdwr:   rdwr limits=(1024, 30381, 34278, 1660672, 212672) apply_limits=1
Warning: nwc_routesearch_t::execute:   to be applied: limits=(1024, 30381, 34278, 1660672, 212672)
Warning: nwc_routesearch_t::execute:   add to world command queue (sync step: command=60322 world=60316)
Warning: nwc_routesearch_t::do_command:   apply limits=(1024, 30381, 34278, 1660672, 212672)
Warning: void convoi_t::laden():   (5404) Yorkshire coach horses (six-in-hand) with braked mail coach trying to load at Moel-y-Groes Cambrian Dock when not at a halt; rerouting
Warning: void convoi_t::hat_gehalten(halthandle_t halt):   Arrival time in the future for (7151) Shire horses (pair) at Norburn Prince Dennis the Doubtful Dock
Warning: nwc_tool_t::rdwr:   rdwr id=8 client=0 plnr=8 pos=4239,1471,0 tool_id=8218 defpar=b,4231,1469,0,10550,0 init=1 flags=0
Warning: nwc_tool_t::rdwr:   rdwr id=8 client=2 plnr=8 pos=4239,1471,0 tool_id=8218 defpar=b,4231,1469,0,10550,0 init=1 flags=0
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 14709.975 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: float32e8_t::operator / (const float32e8_t & x) const:   Division by zero in: 400000 / 0
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
Warning: route_t::intern_calc_route():   Too many steps (1500000>=max 1500000) in route (too long/complex)

jamespetts

If we can get a reproduction case for the float3288_t divisions by zero, that would be helpful. I have no idea whether or not this relates to the losses of synchronisation, but it would certainly be helpful to try to fix this and see whether that assists.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Matthew

#37
I have tried to join B-B several times since today's (2022-02-20) Nightly and reset. I desync as soon as the game unpauses.

I am using the latest B-B Nightly client (#36c9cd5) on Ubuntu 20.04.

Each time the checklist differences are debug_sums[5] ("Passengers/mail generated this step") and [7] ("Transferring cargoes after passenger generation"). For example:

   sums=761526,690114314,89800,111679,6,969,50834,50958,28,112
   sums=761526,690114314,89800,111679,6,971,50834,50961,28,112

BTW I noticed that some of the code for producing the checklists also changed yesterday. I don't understand all those changes, but I guess that makes it possible that the bug is in the checklist itself? I don't how plausible that is.

P.S. My client number incremented every time I tried to join. Is that expected? As far as I could see from the in-game interface and list.extended.simutrans.org, no other clients were connected. Probably a total red herring.

P.P.S. I could reproduce the desyncs on a Windows 10 VPS.
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

jamespetts

Confirmed reproducible with my Linux client. I think that this may be related to the "heavy mode" changes from yesterday, but I will have to undertake further testing to narrow down when this occurred.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

#39
Testing has produced some very odd results: I will consistently lose synchronisation with the server seconds after unpausing on both Bridgewater-Brunel and Stephenson-Siemens servers. However, running a client/server pair locally, no such issue arises, whether the server be a command line server or a graphical server. I have been able to remain connected for >20 minutes with a local client/server pair.

Edit: Further testing shows that downloading the binary compiled on the Bridgewater-Brunel server and running that locally, connecting a client to it locally, still does not reproduce the loss of synchronisation - this only seems to occur when client and server are on different computers.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

I have now attempted testing running a server/client pair on two different physical computers on the same home network. Using demo.sve, I cannot reproduce the loss of synchronisation in this environment. Note that the other computer's Simutrans was built from source from the latest commits on the master branch of the Github repository.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

I think that I have managed to narrow this down and solve the immediate problem, albeit I have not yet found where exactly in the code that this problem was manifested. Updating en.tab on the server (the base version, not the pakset version) to the latest version has allowed me to connect stably without losing synchronisation for >10 minutes. I encourage others to connect and to test this.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Mariculous

updated to latest nightly, doesn't seem to work.

Ranran

Is it possible to check the difference in the existence of all files such as en.tab?
In that case, the difference in font files and city namelist may be taken into consideration. The missing Chinese font has recently been revived in extended. So some people may not have it.

jamespetts

Quote from: Sirius on February 20, 2022, 09:13:44 PM
updated to latest nightly, doesn't seem to work.

Can I clarify - are you losing synchronisation the moment that you connect?

Quote from: Ranran on February 20, 2022, 09:23:36 PM
Is it possible to check the difference in the existence of all files such as en.tab?
In that case, the difference in font files and city namelist may be taken into consideration. The missing Chinese font has recently been revived in extended. So some people may not have it.

This is possible in theory, but very difficult in practice, as there are many files (e.g. config.default) which need to differ, and there is no straightforward interface for checking this automatically.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Mariculous

Quote from: jamespetts on February 20, 2022, 09:29:00 PMCan I clarify - are you losing synchronisation the moment that you connect?
Exactly.

Seems to be working now.

prissi

At least in standard, the city and stop names are provided by the servers, so different citylists should not matter. It it possible to play in Japanese and German together, while all ingame automatic names are in English. Local settings should not matter for servergames, unless there is a setting that was forgotten to be seved with the servergame and was instead added to the enviromnet.

jamespetts

#47
Further testing shows that it is not any divergence in en.tab that causes losses of synchronisation: adding "Test 1" and "Test 2" strings to the bottom of my local en.tab and then connecting results in no loss of synchronisation.

Edit: Further testing shows that adding a single town name syllable locally does not lead to losses of synchronisation.

Edit 2: Further testing shows that adding a number of additional town name syllables locally does not lead to losses of synchronisation. This suggests that there is some specific part of the en.tab file difference in which causes loss of synchronisation.

Edit 3: In further testing, I have not been able to find any modification to my local en.tab that will cause losses of synchronisation with the server. The problem appears to be reproducible only in very particular circumstances. As such, I am discontinuing further investigations and noting this issue as dormant for the present. However, if anyone does manage to find the cause of this, that would be helpful.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

prissi

The town name and everything else is generated in the game on the server and sent as command to the clients. As such the local langugae tab is irrelevant, since I can even join an English server with German or Japanese langugage selected. (and have all error messages etc. in my language.)

Matthew

Quote from: prissi on February 24, 2022, 11:46:57 AM
The town name and everything else is generated in the game on the server and sent as command to the clients. As such the local langugae tab is irrelevant, since I can even join an English server with German or Japanese langugage selected. (and have all error messages etc. in my language.)

That is not how it works in Extended network games. Both client and server generate town names, so there is an inconsistency until the client fully disconnects and a savegame is sent from the server.

But as far as I know that doesn't cause desyncs.
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

zook2

An hour ago I had a desynch when deleting three tiles of road in Alnington, single clicks (not the road removal tool). Freakh was online, too; maybe he did something else that could have caused it. I rejoined a few minutes later and the server had reverted to an earlier save.

Mariculous


Matthew

Hi James. I have a couple of comments relating to the multiplayer mystery bug. I'm not sure if this the best thread but it was the closest one I could find.

Firstly, TransShipmentEnvoy has said on Discord that when his(?) server goes down, Extended has usually filled the memory, so he suspects a memory leak. If you have not already ruled this out as a symptom of the B-B server freezes, then you might want to check the memory situation next time you have to reset the server.

Secondly, does anyone know whether an update is sent to the server list when a player desyncs?
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

neroden

#53
Bridgewater-brunel is consistently desyncing (probably crashing?) and resetting to previous save at 1 PM in December 2012.  No idea why, it doesn't seem to be related do doing anything.  Memory leak?

Now doing the crash at Feb 2013.  Not sure what's going wrong, but it's impossible to play right now.

jamespetts

Apologies for not having had a chance to look at this sooner. This does not seem to be fully consistent or reproducible, since the list server records one client connected and the date at May 2015.

If we can get a reproduction case for this, that would be helpful.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

neroden

OK, I have a log trace for your during a desync
Warning: karte_t:::do_network_world_command:    sync_step=196751  server=[ss=196751 st=98375 nfc=1 hash=0 rand=3120262428 halt=385 line=1 cnvy=1
        ssr=2410088546,0,2410088546,2410088546,3120262428,3120262428,3120262428,0
        str=4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894
        exr=4163797894,2410088546,2410088546,0,0,0,0,0
        sums=129712,50607159,6307,6599,6,0,0,0,7,26]
client=[ss=196751 st=98375 nfc=1 hash=0 rand=3120262428 halt=385 line=1 cnvy=1
        ssr=2410088546,0,2410088546,2410088546,3120262428,3120262428,3120262428,0
        str=4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894,4163797894
        exr=4163797894,2410088546,2410088546,0,0,0,0,0
        sums=129712,50607159,6307,6599,6,0,0,0,7,26]

Warning: karte_t:::do_network_world_command:    sync_step=196752  server=[ss=196752 st=98376 nfc=0 hash=0 rand=170881881 halt=385 line=1 cnvy=1
        ssr=3120262428,0,3120262428,3120262428,3928131591,3928131591,3928131591,0
        str=3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591
        exr=3928131591,170881881,170881881,0,0,0,0,0
        sums=128469,50317759,6307,6599,6,91,25606,25623,16,20]
client=[ss=196752 st=98376 nfc=0 hash=0 rand=1689447484 halt=385 line=1 cnvy=1
        ssr=3120262428,0,3120262428,3120262428,3928131591,3928131591,3928131591,0
        str=3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,3928131591,356449376
        exr=356449376,1689447484,1689447484,0,0,0,0,0
        sums=128469,50317759,6307,6599,6,91,25606,25623,16,20]

I don't know what to make of this -- a bit cryptic.

I've also been getting some really weird things where the server is claiming I have a mismatch even though I *just* downloaded both pak and executable.  This is not reproduceable yet.  I'm trying re-downloading the themes and config directories.

neroden

Here's another desync.  And I was the only one logged in.  This happened when trying to use the station relocation tool

Warning: karte_t:::do_network_world_command:    sync_step=218863  server=[ss=218863 st=109431 nfc=1 hash=0 rand=2619890734 halt=385 line=1 cnvy=1
        ssr=790818943,0,790818943,790818943,2619890734,2619890734,2619890734,0
        str=790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943
        exr=790818943,790818943,790818943,0,0,0,0,0
        sums=136393,54160268,6684,6899,6,0,0,0,6,31]
client=[ss=218863 st=109431 nfc=1 hash=0 rand=2619890734 halt=385 line=1 cnvy=1
        ssr=790818943,0,790818943,790818943,2619890734,2619890734,2619890734,0
        str=790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943,790818943
        exr=790818943,790818943,790818943,0,0,0,0,0
        sums=136393,54160268,6684,6899,6,0,0,0,6,31]

Warning: karte_t:::do_network_world_command:    sync_step=218864  server=[ss=218864 st=109432 nfc=0 hash=0 rand=3670556178 halt=385 line=1 cnvy=1
        ssr=2619890734,0,2619890734,2619890734,2104222973,2104222973,2104222973,0
        str=2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973
        exr=2104222973,3670556178,3670556178,0,0,0,0,0
        sums=136204,53876190,6686,6899,6,103,25058,25068,11,34]
client=[ss=218864 st=109432 nfc=0 hash=0 rand=669551320 halt=385 line=1 cnvy=1
        ssr=2619890734,0,2619890734,2619890734,2104222973,2104222973,2104222973,0
        str=2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,2104222973,3711126139
        exr=3711126139,669551320,669551320,0,0,0,0,0
        sums=136204,53876190,6686,6899,6,103,25058,25068,11,34]

Warning: karte_t::network_disconnect(): Lost synchronisation with server. Random flags: 0
Warning: nwc_routesearch_t::reset:      all static variables are reset
Warning: karte_t:::do_network_world_command:    Disconnected due to checklist mismatch

I have no idea what to make of any of this since I haven't looked through the code to understand what the log messages mean.

TurfIt

Look in checklist_t::print() to decode which rand index. For your desync, first difference is between rand[22] and rand[23] - last number printed in str= line.
Look in simworld. cc to find rands[22] and rands[23] surrounding haltestelle_t::step_all();
Your desync was due to your computer calling simrand() while stepping the halts, the server did not - mismatch.
IIRC the only thing halts call randoms for is generating pedestrians...

neroden

Well *that's* interesting.  Perhaps halts call randoms for something other than generating pedestrians now.

But if not... then it's worth noting that the setting for whether halts generate pedestrians is in simuconf.tab under display settings and not in the pak.  Hmmmmm.

ceeac

There were also lots of thread synchronization bugs (data races) the last time I checked (which was more than a year ago I think) - to reproduce, just compile and link the headless server with -fsanitize=thread and load a savegame of your choice. There are also some data races in the display subsystem, but those are also in Standard and don't affect the synchronization of the simulation.

Ranran

#60
Simutrans Extended cannot completely suppress the occurrence of pedestrians by changing the display settings. They seem to occur based on arrival.
In another case, even when a private car arrives at its destination, it is destroyed and then purged two.
Also, as Freddy previously reported, the location of pedestrians differs depending on the client, so blowing it up with the remove tool will cause a dysync.

EDIT:
Quote from: prissi on June 19, 2022, 03:23:03 AMHence, r10664+10660+10659 made passengers independent from the game state. So a client can switch off passengers (or the server can switch off them.)
r10664 may not have been merged yet. It might be worth looking into it.

jamespetts

Thank you all for your posts and analysis. As to the pedestrians: the problem is very unlikely to be just the display of pedestrians itself. Rather, losing synchronisation on the display of pedestrians is likely to be a symptom of the problem: this is likely simply to be the first call to simrand() after the game states on client and server have diverged. Those game states could have diverged anywhere between the simrand() call that triggered the loss of synchronisation and an arbitrary point in the past. It need not even have occurred since the last simrand() call if the state divergence had no direct effect on the part of the game that effected the previous simrand() call such that the game state affecting when that call was made remained undiverged. This is what makes debugging these loss of synchronisation problems so challenging. What I have found that works in the past is laboriously narrowing it down to a particular part of the code by disabling that part of the code for testing and seeing whether the losses of synchronisation still occur, and then examining that part of the code carefully and making educated guesses as to what in that code might be the cause of the problem.

In this case, the fact that some of the losses of synchronisation at least fairly reliably occur after doing something on a stop and the divergence is first detected in code relating to stops suggests that we should probably be looking at something in the code relating to stops first.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

Quote from: ceeac on May 26, 2024, 06:04:16 AMThere were also lots of thread synchronization bugs (data races) the last time I checked (which was more than a year ago I think) - to reproduce, just compile and link the headless server with -fsanitize=thread and load a savegame of your choice. There are also some data races in the display subsystem, but those are also in Standard and don't affect the synchronization of the simulation.
Thank you for this suggestion. I have been attempting to do this, but, unfortunately, I get the error,

FATAL: ThreadSanitizer: unexpected memory mapping 0x57a2c0f6b000-0x57a2c107a000
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

Note for those interested in looking into the losses of synchronisation immediately on connexion in some cases: a possible cause was identified a few months ago here, but was not able to be fixed at the time.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

neroden

Quote from: Ranran on May 26, 2024, 06:16:49 AMSimutrans Extended cannot completely suppress the occurrence of pedestrians by changing the display settings. They seem to occur based on arrival.
In another case, even when a private car arrives at its destination, it is destroyed and then purged two.
Also, as Freddy previously reported, the location of pedestrians differs depending on the client, so blowing it up with the remove tool will cause a dysync.

EDIT:r10664 may not have been merged yet. It might be worth looking into it.
It hasn't been.  It doesn't merge cleanly; the first problem is that r10520 hasn't been merged.  *sigh*  These are all after the directory structure reorg, so I'm going to go back to attempting to clean up the merges from standard from before r10440

Ranran

#65
https://github.com/Ranran-the-JuicyPork/simutrans-ex-fix/commit/71764efd7111550b4466874b287d79452f0238fd
I tried incorporating it before. However, the consecutive commits from r10520 did not work properly, so they were left alone.
But I think this itself is just related to the folder structure.

https://github.com/Ranran-the-JuicyPork/simutrans/commit/4f4c9e947c2380593ed631f5975efed59f756e45
As for r10664, it is after the folder structure has been changed and lot of changes and doesn't seem to be related to the UI(and chat window/font stuff), so I'm basically skipping stuff like this as declared.
But I suspect this commit is not that related to r10520.

ceeac

Quote from: jamespetts on May 26, 2024, 10:36:41 AMThank you for this suggestion. I have been attempting to do this, but, unfortunately, I get the error,

FATAL: ThreadSanitizer: unexpected memory mapping 0x57a2c0f6b000-0x57a2c107a000

That is a known issue: https://stackoverflow.com/questions/77850769/fatal-threadsanitizer-unexpected-memory-mapping-when-running-on-linux-kernels
On current Arch stable setting sysctl vm.mmap_rnd_bits=30 temporarily is sufficient, but depending on the distribution ASLR needs to (temporarily!) disabled entirely.

neroden

Looks like the answer in here for turning off address randomization for one program is still valid:

https://stackoverflow.com/questions/5194666/disable-randomization-of-memory-addresses

setarch `uname -m` -R ./yourProgram

jamespetts

Thank you both for that - I will look into that when I get a moment. 

In the meantime, I should note one particular observation: when testing the 2403-std branch just now, loading an old saved game from before piers were introduced and then connecting to that as a client resulted in the same loss of synchronisation immediately on connexion and then remaining in synchronisation stably for some time on the second attempt as we have often seen elsewhere, suggesting that the piers may not be responsible - or at least, not exclusively responsible - for this issue.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Ranran

The problem that the game time does not progress after the first connection was introduced at some point in Standard, and it was the same in Standard. That may be related.
It's possible that I introduced an error in the merge regarding the flag that advances game time.