The International Simutrans Forum

 

Author Topic: New desync debugging tool: Heavy Mode  (Read 271 times)

0 Members and 1 Guest are viewing this topic.

Offline ceeac

  • Devotee
  • *
  • Posts: 280
New desync debugging tool: Heavy Mode
« on: November 06, 2021, 09:41:37 AM »
This patch adds a new network desync debugging tool: Heavy Mode (the name is inspired from here, although the implementation is silghtly different). In a nutshell, this allows comparing the game state of the client and server for every sync_step. There are now three modes which can be selected via -DSIMUTRANS_HEAVY_MODE=<number> in CMake or HEAVY_MODE:=<number> in config.default:
  • 0: Disabled. No changes from the current behaviour. Default mode.
  • 1: Stores the adler32 hash of the game state for every sync_step and compares the hashes every server_frames_between_checks sync_steps. Essentially this replaces the current checklist mechanism by a game state hash comparison.
  • 2: Same as 1, but additionally, both client and server save the game for every sync_step to <user_dir>/save/heavy/ and abort on connection loss/desync to allow comparing the save at the exact sync_step where the desync occurred. As this generates a lot of disk IO, using a ramdisk is recommended. ;)

Simply compiling server and client in Heavy Mode and connecting with no user interaction should work, but there are still two cases of "desyncs" (i.e. desyncs in Heavy Mode that do not occur with Heavy Mode disabled as far as I know):
  • In the depot window, when pressing the schedule button to give a convoi an individual schedule, the convoi gets an empty schedule which is not synchronized over the network.
  • zeiger_t's and dummy grounds for elevated way previews and such are saved in the save game which causes a desync. I'm not sure if we can get away with not saving them since zeiger_t's are not only used for UI purposes but also as markers for the old C++ AI. Not sure yet how to solve this properly...

Offline Dwachs

  • DevTeam, Coder/patcher
  • Administrator
  • *
  • Posts: 4931
  • Languages: EN, DE, AT
Re: New desync debugging tool: Heavy Mode
« Reply #1 on: November 06, 2021, 10:10:38 AM »
Nice idea! would it be possible to also record all commands in a savegame, to allow a full replay of the game?

Quote
zeiger_t's and dummy grounds for elevated way previews and such are saved in the save game which causes a desync. I'm not sure if we can get away with not saving them since zeiger_t's are not only used for UI purposes but also as markers for the old C++ AI. Not sure yet how to solve this properly...

This should not happen: they are deleted by calls to cleanup, see simworld.cc:4775.

I also wrote a desync debug tool some years ago: it logged calls to simrand, and allowed to find the call that got wrong. However it was not very helpful, as looking for the cause was an entire different story :/

Offline ceeac

  • Devotee
  • *
  • Posts: 280
Re: New desync debugging tool: Heavy Mode
« Reply #2 on: November 22, 2021, 07:29:45 AM »
would it be possible to also record all commands in a savegame
Yes, but that is orthogonal to this patch. Also this will not work with variable sync_step sizeof single player mode, unless the step size is saved or one uses the fixed step size to record a replay.

they are deleted by calls to cleanup
I explicitly don't want to do that because that might hide desyncs (for example, hovering over a full sea tile with the mouse when a ship enters the tile).