This patch adds a new network desync debugging tool: Heavy Mode (the name is inspired from here (https://www.factorio.com/blog/post/fff-63), although the implementation is silghtly different). In a nutshell, this allows comparing the game state of the client and server for every sync_step. There are now three modes which can be selected via -DSIMUTRANS_HEAVY_MODE=<number> in CMake or HEAVY_MODE:=<number> in config.default:
- 0: Disabled. No changes from the current behaviour. Default mode.
- 1: Stores the adler32 hash of the game state for every sync_step and compares the hashes every server_frames_between_checks sync_steps. Essentially this replaces the current checklist mechanism by a game state hash comparison.
- 2: Same as 1, but additionally, both client and server save the game for every sync_step to <user_dir>/save/heavy/ and abort on connection loss/desync to allow comparing the save at the exact sync_step where the desync occurred. As this generates a lot of disk IO, using a ramdisk is recommended. ;)
Simply compiling server and client in Heavy Mode and connecting with no user interaction should work, but there are still two cases of "desyncs" (i.e. desyncs in Heavy Mode that do not occur with Heavy Mode disabled as far as I know):
- In the depot window, when pressing the schedule button to give a convoi an individual schedule, the convoi gets an empty schedule which is not synchronized over the network.
- zeiger_t's and dummy grounds for elevated way previews and such are saved in the save game which causes a desync. I'm not sure if we can get away with not saving them since zeiger_t's are not only used for UI purposes but also as markers for the old C++ AI. Not sure yet how to solve this properly...
Nice idea! would it be possible to also record all commands in a savegame, to allow a full replay of the game?
Quotezeiger_t's and dummy grounds for elevated way previews and such are saved in the save game which causes a desync. I'm not sure if we can get away with not saving them since zeiger_t's are not only used for UI purposes but also as markers for the old C++ AI. Not sure yet how to solve this properly...
This should not happen: they are deleted by calls to cleanup, see simworld.cc:4775.
I also wrote a desync debug tool some years ago: it logged calls to simrand, and allowed to find the call that got wrong. However it was not very helpful, as looking for the cause was an entire different story :/
Quote from: Dwachs on November 06, 2021, 10:10:38 AMwould it be possible to also record all commands in a savegame
Yes, but that is orthogonal to this patch. Also this will not work with variable sync_step sizeof single player mode, unless the step size is saved or one uses the fixed step size to record a replay.
Quote from: Dwachs on November 06, 2021, 10:10:38 AMthey are deleted by calls to cleanup
I explicitly don't want to do that because that might hide desyncs (for example, hovering over a full sea tile with the mouse when a ship enters the tile).
Could you please commit this patch?
Edit: The comparison of two savegames could be done by a derived class from loadsave_t: it takes two (original) loadsave_t objects and throws an error if the result of byte-reading differs between the two.
Patch submitted in r10297.
Save file comparison is a good idea - maybe using a hidden command line parameter -compare a b or similar.
I would like to turn the heavy-mode setting into a command-line parameter. Then I do not have to recompile to switch between modes 1 and 2.
I am fine with this. :)
Submitted: heavy mode and comparing savegames from command line