Author Topic: Desync issue (devel-new-2) with Linux Server/Windows client (Read 35874 times)

jamespetts · « **Reply #35 on:** October 26, 2016, 01:50:49 AM »

Dr. Supergood makes an interesting point. Ves - are these GCC builds? If so, I will have a go with the latest one connecting to the servers and see whether there are any desyncs.

Edit: Unfortunately, testing http://server.exp.simutrans.com/Devel-new-builds/Simutrans-Experimental_161024_15851bf.exe by connecting to the Bridgewater-Brunel server results in a desync within a few seconds of connecting.

Ves · « **Reply #36 on:** October 26, 2016, 01:11:24 PM »

They are compiled on vs2015 on windows 10. I maybe might be able to produce some more builds later today.

jamespetts · « **Reply #37 on:** October 26, 2016, 01:49:57 PM »

Ahh, yes, I see. Thank you: that is most helpful.

Vladki · « **Reply #38 on:** October 26, 2016, 06:36:05 PM »

Quote from: jamespetts on October 25, 2016, 04:07:29 PM

Ahh, that makes more sense. Can you try instead the immediately previous commit in that case, SHA 1cb53908a88570042717df64be86828fe917c8d6 ? Thank you very much.

So the stack smashing was probably by trying to load new savegame by older version. Now the british test server is running commit 1fb72c0.

jamespetts · « **Reply #39 on:** October 26, 2016, 08:57:20 PM »

Thank you very much for testing this: this is most helpful. However, the issue with the saved game is preventing a proper comparison: this is the saved game from your server as running now, which produces no desync even on the latest version. This saved game, however, which is a more developed and later version than what is on the server, but still, I think, old enough to be of the old saved game version that should work with the old version that you are running does produce desyncs with the latest version.

Can you try this version on your server so that I can connect with the binary of the old version and see whether it makes any difference? Thank you.

Vladki · « **Reply #40 on:** October 26, 2016, 11:13:24 PM »

Server restarted with the requested savegame

jamespetts · « **Reply #41 on:** October 27, 2016, 12:21:23 AM »

Thank you very much for uploading that. Connecting to that with the 1fb70... Windows executable compiled by Ves produces a near instant desync, so the problem does not appear to be of recent origin at all.

The next step is painstakingly to work out which features of the saved game cause the desync by testing either with certain features disabled in the code, or with saved games with those features not used. The latter may be a good place to start, as we have two very similar saved games, one which does and one which does not cause a desync.

Vladki - I wonder whether you could start by sending all the trains to the depot (you will be able to connect with Linux), and let me know when you have done that so that I can attempt to connect again.

Can anyone remember whether the game in the state in which it now exists has ever been playable on the server without desyncs when connected from a Windows client?

For reference, the date of this saved game was originally the 18th of September, and is the saved game that was uploaded to demonstrate the problems with trains apparently teleporting, which was since fixed.

Thank you both very much for your help so far.

Edit: One thought occurs to me - does anyone have a Windows executable from the last month or two built with Visual Studio 2012? It would be worthwhile trying that with a matching server version to see whether that makes any difference.

Vladki · « **Reply #42 on:** October 27, 2016, 06:47:34 AM »

I won't be at my home pc until sunday evening. So if someone else with linux would be able to help, please do.

Ves · « **Reply #43 on:** October 27, 2016, 08:07:08 AM »

James, I will have quite limited time in the forthcoming days, do you have any specific commits you want me to
compile, or a range of them?

jamespetts · « **Reply #44 on:** October 27, 2016, 11:09:43 AM »

Can you try the very last commit of the original devel-2 (using Visual Studio 2012), which is ff5373424e0fee6b5163ea59ae464fe943c2ed60, and also 197f14910b8395cc72a6c76595d1205b02b30120 from devel-new-2 in Visual Studio 2015?

Thank you very much.

Ves · « **Reply #45 on:** October 27, 2016, 11:01:09 PM »

They are both online now (they where in fact already compiled, just removed from server!).

Compiled using msvs2015, win10.

jamespetts · « **Reply #46 on:** October 28, 2016, 01:31:00 AM »

Thank you very much. Vladki, are you able to set your server to run ff5373424e0fee6b5163ea59ae464fe943c2ed60?

Vladki · « **Reply #47 on:** October 28, 2016, 09:45:18 AM »

Server commit ff5373... cannot load the savegame server-test-control-3:

FATAL ERROR: loadsave_t::read - savegame corrupt, not enough data. Restarted again with 1fb72c...

jamespetts · « **Reply #48 on:** October 28, 2016, 10:59:01 AM »

That is unfortunate. Does anyone have any saved games created on or before the 1st of September based on the map from this same server game? Edit: I have found one myself here. Can we try with ff5373424e0fee6b5163ea59ae464fe943c2ed60 and that saved game?

Thank you very much for trying.

Vladki · « **Reply #49 on:** October 28, 2016, 11:36:21 AM »

Quote from: jamespetts on October 28, 2016, 10:59:01 AM

That is unfortunate. Does anyone have any saved games created on or before the 1st of September based on the map from this same server game? Edit: I have found one myself here. Can we try with ff5373424e0fee6b5163ea59ae464fe943c2ed60 and that saved game?

Thank you very much for trying.

Running.

jamespetts · « **Reply #50 on:** October 28, 2016, 12:11:40 PM »

Splendid, thank you very much.

Edit: This has been extremely helpful: to Vladki's server with ff5373424e0fee6b5163ea59ae464fe943c2ed6, I am able to stay in sync. However, to the Bridgewater-Brunel server running the 1585... build on server and client, I am not able to stay in sync with the same saved game (server-test-3.sve).

This has narrowed down the difficulties considerably; thank you.

Ves · « **Reply #51 on:** October 28, 2016, 01:33:28 PM »

Glad that there is progress on the issue!

jamespetts · « **Reply #52 on:** October 28, 2016, 08:19:01 PM »

Vladki - to narrow this down further, can you build commit 197f14910b8395cc72a6c76595d1205b02b30120 on the server and run it with the same saved game as is on there now?

Vladki · « **Reply #53 on:** October 30, 2016, 10:23:10 AM »

server restarted whit commit 197f149

jamespetts · « **Reply #54 on:** October 30, 2016, 11:01:12 AM »

Thank you very much - this is most helpful. This might well be a slow process of trying various builds on client/server to find the point at which it stops working. The usual technique is to find the point half-way between the known working and not working point and see whether that works, giving a new known working or not working point, and then repeating the process until the exact commit causing the failure is identified.

Edit: Ves - do you have an archive Windows build for this commit?

Vladki · « **Reply #55 on:** October 30, 2016, 11:46:22 AM »

If you have a list of commits that are interesting for you, I may compile them at once, and run on different ports.

jamespetts · « **Reply #56 on:** October 30, 2016, 11:48:35 AM »

That is helpful; but the trouble is that which commits will need to be compiled next will depend on the result of testing each previous one on the basis of the system that I describe above (which I believe is a well-known fault finding methodology).

Vladki · « **Reply #57 on:** October 30, 2016, 12:12:56 PM »

Yes, but if you have now at least two commits - one working and one not, I could compile several commits in between to more precisely find the midpoint. This will narrow the range more quickly than waiting for you to test and then waiting for me to compile one more chosen commit.

jamespetts · « **Reply #58 on:** October 30, 2016, 01:34:36 PM »

Ahh, I see. That is helpful. ff5373424e0fee6b5163ea59ae464fe943c2ed6 is the last known working build at present. 1fb72c0... is, I think, the first known non-working build; so, if you can extrapolate from those and produce builds at half, quarter, and eighth points between them (it need not be exact), that would be helpful.

Vladki · « **Reply #59 on:** October 30, 2016, 02:39:49 PM »

Uff that's quite a lot of commits. I notecied that there are some commits from BerndGabriel...
And I think he made a typo in 30574d19e928f908425b0f584f5e30987f0acfe4

windows build for the currently runnig commit is here: http://server.exp.simutrans.com/Devel-new-builds/
Server commit dd8d3b2 running on port 13354, windows binary is available for download on the above link as well.
And commit 9b7bfe9 running on port 13355 (swedish pak server stopped now.)

Server commit 197f149 crashed with:

Code: [Select]

FATAL ERROR: vector_tpl<T>::[] - 7koord3d: index out of bounds: 72 not in 0..70
Aborting program execution ...

For help with this error or to file a bug report please see the Simutrans forum at
http://forum.simutrans.com
FATAL ERROR: vector_tpl<T>::[] - 7koord3d: index out of bounds: 72 not in 0..70
Aborting program execution ...

For help with this error or to file a bug report please see the Simutrans forum at
http://forum.simutrans.com
Aborted (core dumped)

I think I have seen this error before...

jamespetts · « **Reply #60 on:** October 30, 2016, 03:09:04 PM »

Yes, there are a lot of commits; those were mainly things from Standard in that period. That is why I suggested doing them one by one rather than all at once.

However, this appears to work without disconnecting, although I cannot test it for very long, as I get a crash after a few minutes (a bug since fixed, I think).

We now need to go to the (approximate) half-way point between this working build (17 September) and the earliest known non-working build (13 October). May I suggest that we try 69ff5d7d2d1bface984c5c0546bc4004e90b63c4, which is the first build using Visual Studio 2015 from the 25th of September?

Thank you very much for your help with this.

Vladki · « **Reply #61 on:** October 31, 2016, 06:52:42 PM »

comit 69ff5d7 running on the standard port.

How about the previously posted commits?
./simutrans-experimental-dd8d3b2 -server 13354
./simutrans-experimental-9b7bfe9 -server 13355

jamespetts · « **Reply #62 on:** October 31, 2016, 07:00:48 PM »

Thank you very much for your help: this is much appreciated. There does not seem to be a Windows build for 69ff5d7 available, and I have had trouble compiling older vesrions myself for some reason; would anyone be able to produce a copy?

As for the other two builds:

dd8d3b2 does desync; and
9b7bfe9 also does desync.

Ves · « **Reply #63 on:** October 31, 2016, 07:01:02 PM »

./simutrans-experimental-dd8d3b2
./simutrans-experimental-9b7bfe9

Is in the devel-new folder.

comit 69ff5d7 I cannot compile, due to (now solved) timespec-issues..

jamespetts · « **Reply #64 on:** October 31, 2016, 07:06:23 PM »

dd8d3b2 is from the 8th of October and 9b7bfe9 is from the 3rd of October, so the problem occurred somewhere between the 17th of September and the 3rd of October. May I suggest trying 41d2457cc0763c8d6449fd0a5e398b6ddd1a119d, which is one commit before the change to Visual Studio 2015, from the 19th of September?

Thank you very much.

Ves · « **Reply #65 on:** October 31, 2016, 07:12:28 PM »

The 41d2457cc0763c8d6449fd0a5e398b6ddd1a119d is now on devel-new!

I realize, the patch that solved the timespec issue for me was 9b7bfe9 (3 oct) and you changed two files. Do you think it is valid if I fetch one of those commits in the middle (eg 69ff5d7) and change those two files? Will it disturb anything you think?

edit:
Anyway, I took the liberty to do it and have added 69ff5d7 (the msvs2015 upgrade) to the server!

edit2:
Get immediate desync with my version of 69ff5d7!

jamespetts · « **Reply #66 on:** October 31, 2016, 07:43:38 PM »

Excellent, thank you very much for testing: that is extremely helpful. The next thing to check is the immediately previous version to see whether the problem is the switch to Visual Studio 2015 (i.e. 41d2457cc0763c8d6449fd0a5e398b6ddd1a119d).

Vladki · « **Reply #67 on:** October 31, 2016, 08:36:05 PM »

./simutrans-experimental-41d2457 is now running on standard port

Ves · « **Reply #68 on:** October 31, 2016, 08:48:01 PM »

Quote from: Vladki on October 31, 2016, 08:36:05 PM

./simutrans-experimental-41d2457 is now running on standard port

7 mins, still running.....
12 mins, still running....

edit
accidentally closed the game window and then the game had been running for 34 minutes without a desync!

jamespetts · « **Reply #69 on:** October 31, 2016, 09:18:06 PM »

Thank you - this is very interesting. Either the problem is something that occurs as a result of the one substantive change in the code in commit 69ff5d7d2d1bface984c5c0546bc4004e90b63c4, or the problem is with Visual Studio 2015 itself. Let me check reverting the one substantive change, which is:

Code: [Select]

-                        schiene_t* const sch = obj_cast<schiene_t>(way);
+         //schiene_t* const sch = obj_cast<schiene_t>(way);
+         schiene_t* const sch = way->is_rail_type() ? (schiene_t*)way : NULL;

and see whether that helps.

Edit: I have just pushed the change to Github - let us see whether this helps.

Edit 2: Latest version now running on the Bridgewater-Brunel server.

Edit 3: It does stay in sync between instances on a Windows machine, so if it does not work between Windows and Linux, this is the original problem continuing, not some new problem subsequently introduced.

Edit 4: A desync still occurs between the Windows client and Linux server (Bridgewater-Brunel) with the latest version. I wonder whether the problem could be caused by Visual Studio 2015 itself?