The International Simutrans Forum

 

Author Topic: Instability on the Bridgewater-Brunel server  (Read 13658 times)

0 Members and 1 Guest are viewing this topic.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #105 on: October 27, 2018, 05:27:15 PM »
I have been in the process of testing by starting again with the saved game and by withdrawing vehicles only on the last route before the game apparently became stable: you joined, I think, after this process had started.

I am aware that there are stuck vehicles: however, this should not cause the clients to lose synchronisation with the server. It is unlikely that these are the cause as:

(1) there have been many times in the past when there have been stuck vehicles and/or vehicles with no route on the server and it has not lost synchronisation with clients; and
(2) removing only the Bay Transport Group prevents loss of synchronisation, but does cause some stuck vehicles.

My tests are now revealing bizarre and inconsistent results: starting from the FRC - Roxingstoke - Templecaster (local) line and working down to the bottom of the list, I have not been able to achieve the stability that I had earlier achieved starting at the bottom and working up to the FRC - Roxingstoke - Templecaster (local) line. I am now working up from that line towards the top of the list, but this is an extremely slow process.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2727
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #106 on: October 27, 2018, 05:29:45 PM »
One cannot rule out that after enough time is allowed to progress the game magically goes back into sync due to how many broken lines there are due to the removal of all other companies. One might have to test with all companies present so that all lines continue normal operation.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #107 on: October 27, 2018, 06:08:30 PM »
Having to reset the game every time would double or triple the time taken to test. This has already taken literally all day, so this sort of testing would have to be an absolute last resort.

Also, this seems unlikely, as the broken lines are stably broken and so do not change much with time.

However, the latest test shows that removing up to and including the Northern Frontier Express seems to have lead to stability. I should be grateful if people could log on for further testing in the current state. This line in particular is promising because the rolling stock seems to have been replaced in 1938.

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 572
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #108 on: October 27, 2018, 07:18:16 PM »
I have been able to remain in sync for ~1 hour.

Offline Phystam jp

  • *
  • Posts: 248
  • Pak256.Ex developer
  • Languages: JP, EN, EO
Re: Instability on the Bridgewater-Brunel server
« Reply #109 on: October 27, 2018, 08:46:07 PM »
I could connect to the server for ~1 hour.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #110 on: October 27, 2018, 08:55:14 PM »
Thank you both - that is very helpful.

May I ask now that one of you could try adding trains back to the withdrawn Bay Transport lines one by one, then waiting ~10 minutes after each repopulation to see whether it causes loss of synchronisation?

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 572
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #111 on: October 27, 2018, 11:09:11 PM »
I resent the trains out on the Northern Frontier Express and after a few minutes... desync!

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #112 on: October 28, 2018, 12:13:17 AM »
Very interesting - thank you. It might help to test a few times by withdrawing them and then adding them to check that the behaviour is consistent. If it is, this is a very interesting development, although quite what is special about that line remains to be investigated.

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 572
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #113 on: October 28, 2018, 01:39:41 AM »
Sending the trains to depot again seems to prevent a desync. I have managed to remain connected for over 30 min.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2727
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #114 on: October 28, 2018, 09:44:47 AM »
Since most trains work, it is likely something to do with the signalling. Try altering signalling, especially around any complex parts. Since one knows that line is the cause, any lines sharing the same ways should be checked if they can be terminated without stopping the desync to make testing easier.
« Last Edit: October 28, 2018, 11:37:48 AM by DrSuperGood »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #115 on: October 28, 2018, 11:41:00 AM »
Thank you for that testing: that is very helpful.

One thing that might be worth checking is to send out a freight train using the same line to try to distinguish whether the issue is with signalling or similar or whether it is with passenger routing.

Offline SuperTimo gb

  • *
  • Posts: 62
  • Languages: English, French
Re: Instability on the Bridgewater-Brunel server
« Reply #116 on: October 28, 2018, 04:47:04 PM »
I have sent out a freight train on that route and so far encountered no issues.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #117 on: October 28, 2018, 05:20:22 PM »
Splendid, thank you for testing. To make this test more rigorous, it would be helpful if you could:

(1) send out some more freight trains (in case the issue is signalling and requires trains to interact to cause problems); and
(2) try to stay connected for at least 45 minutes.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #118 on: October 29, 2018, 12:37:20 AM »
I have added a number of freight trains to this line, and have been able to stay connected for >1 hour, so it seems unlikely that the problem is signalling or similar.

Offline SuperTimo gb

  • *
  • Posts: 62
  • Languages: English, French
Re: Instability on the Bridgewater-Brunel server
« Reply #119 on: October 29, 2018, 10:20:17 AM »
Splendid, thank you for testing. To make this test more rigorous, it would be helpful if you could:

(1) send out some more freight trains (in case the issue is signalling and requires trains to interact to cause problems); and
(2) try to stay connected for at least 45 minutes.

Sorry James I intended to do some more thorough testing with freight trains but got caught up yesterday evening.

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 572
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #120 on: October 29, 2018, 06:16:56 PM »
Tested sending out a single new passenger train with different rolling stock. No desync.

Edit: There actually seems to be a systematic desync after a longer period of time than before.
« Last Edit: October 29, 2018, 08:23:32 PM by Rollmaterial »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #121 on: October 29, 2018, 08:55:39 PM »
Thank you for testing - can I ask what you mean by a "systematic" desync?

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 572
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #122 on: October 29, 2018, 09:01:32 PM »
That the game consistently loses sync after a period of time that is longer than for the originally observed desync. It suggests that the desync occurs individually on any train of the line.
Edit: I have managed to stay in sync for well over an hour.
« Last Edit: October 29, 2018, 10:32:43 PM by Rollmaterial »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #123 on: October 29, 2018, 10:41:20 PM »
Interesting - what is the train doing when you are in sync for so long? I wonder whether there is a pattern in its activity and its connection with when you lose synchronisation?

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #124 on: October 31, 2018, 10:25:13 PM »
Some additional testing this evening shows that using the A4 class locomotive used on the Fronteir Express but hauling fast freight wagons does not trigger a loss of synchronisation, so any cause specific to the A4 class locomotive can be ruled out.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #125 on: November 03, 2018, 10:22:49 PM »
Further testing yields some interesting results: I have re-instituted passenger services on the Northern Frontier Express without any loss of synchronisation. However, I am using different rolling stock: a GWR Hall class and GWR express carriages (just passenger, no mail).

I should be grateful if anyone else could log in and double check for long term stability. The next phase of testing would be to add mail to these trains and see whether that makes any difference. The trains are successfully carrying passengers along this line currently without loss of synchronisation, and I have been connected for over an hour. There are 5 trains in total running.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #126 on: November 04, 2018, 11:05:25 AM »
I have now added some mail only trains to the line. After doing so, I experienced a loss of synchronisation after a time. I tried to reconnect and run the test again for confirmation, but my hardware problems caused my computer to crash while running the game on this occasion, so I could not test fully. The game is still running with mail trains on the Northern Frontier Express - I should be very grateful if anyone could log in and test stability with the game in this state.

Offline SuperTimo gb

  • *
  • Posts: 62
  • Languages: English, French
Re: Instability on the Bridgewater-Brunel server
« Reply #127 on: November 04, 2018, 05:47:24 PM »
I've connected 3 times and each time I have de-synced from the server.

edit: I am connected at the moment and it seems to be saying in sync, I will see how long that lasts.
« Last Edit: November 04, 2018, 06:04:34 PM by SuperTimo »

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 572
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #128 on: November 04, 2018, 06:22:47 PM »
I have connected twice and desynced both times after 20-30 min.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #129 on: November 04, 2018, 08:18:03 PM »
That is extremely interesting - thank you both for that test. This suggests that there may well be a problem specific to mail. SuperTimo - was Bay Transport originally your company - if so, can you remember whether mail was conveyed on the Northern Frontier Express before 1938?

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 572
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #130 on: November 04, 2018, 08:27:46 PM »
From what I remember of previous testing, the line didn't carry mail after 1938 either. At least it didn't carry mail when I tested and it was desyncing after 5-10 minutes.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #131 on: November 04, 2018, 08:31:24 PM »
Interesting - can I ask what locomotives and rolling stock that you used for that test?

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 572
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #132 on: November 04, 2018, 08:34:12 PM »
I think it was the stock originally used on the line, that being LNER A1's or A3's with the latest GNR corridor carriages.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #133 on: November 04, 2018, 09:07:53 PM »
I think it was the stock originally used on the line, that being LNER A1's or A3's with the latest GNR corridor carriages.

Do you remember which carriages specifically? Some brake carriages that go with this set (I think the LNER rather than the GNR set by 1938) carry mail rather than passengers.

Offline SuperTimo gb

  • *
  • Posts: 62
  • Languages: English, French
Re: Instability on the Bridgewater-Brunel server
« Reply #134 on: November 04, 2018, 11:42:49 PM »
That is extremely interesting - thank you both for that test. This suggests that there may well be a problem specific to mail. SuperTimo - was Bay Transport originally your company - if so, can you remember whether mail was conveyed on the Northern Frontier Express before 1938?

Bay transport wasn't my company (Great Highland Railway was mine), but the owner started carrying post as I requested them to as it would be beneficial to both of us (I think bay transport introduced a couple of mail only trains following this). This was not long before the de-sync started I think, around the time that Green Quantinglow airport was built which I think was around 1939.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #135 on: November 04, 2018, 11:48:11 PM »
This is very interesting. This does suggest that the carriage of mail in particular on this line is likely to be a cause of the problem. It is very odd that this line in particular carrying mail is a problem rather than it always being a problem when mail is carried, however. I know that there was a comprehensive mail network for over 100 game years on the server before this problem started happening, so this is very odd indeed.

Offline Rollmaterial fi

  • Devotee
  • *
  • Posts: 572
  • Languages: EN, FR, DE, FI, SE
Re: Instability on the Bridgewater-Brunel server
« Reply #136 on: November 05, 2018, 12:48:06 AM »
Do you remember which carriages specifically? Some brake carriages that go with this set (I think the LNER rather than the GNR set by 1938) carry mail rather than passengers.
I remember there were no mail carriages. Simply the last model of GNR passenger carriages, with a restaurant carriage in each train.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2727
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #137 on: November 05, 2018, 07:26:13 AM »
Mail generally has long waiting times compared with passengers. This means the total jounrney time or other such metrics used to decide route travelled would be considerably longer than with passengers. If due to an incorrect type choice an integer were to overflow one might run into platform specific behaviour that is not warned by compilers. In C/C++ integer overflow/underflow is not defined for signed types, and additionally a type might not be the same number of bits between platforms. This could result in mail taking different routes between platforms.

This would only be a problem with new mail lines because before there were not many competing excessivly long mail routes so there was only 1 choice. Recently before the desyncs started (within 10 years before) a cross map mail ship line was added, on top of an air mail network as well as additional train mail networks. There is probably well over 3 paths for mail to reach some destinations, with some of them being stupidly slow due to waiting times.

This is speculation but could be worth looking into.

Offline SuperTimo gb

  • *
  • Posts: 62
  • Languages: English, French
Re: Instability on the Bridgewater-Brunel server
« Reply #138 on: November 05, 2018, 10:17:39 AM »
This would only be a problem with new mail lines because before there were not many competing excessivly long mail routes so there was only 1 choice. Recently before the desyncs started (within 10 years before) a cross map mail ship line was added, on top of an air mail network as well as additional train mail networks. There is probably well over 3 paths for mail to reach some destinations, with some of them being stupidly slow due to waiting times.

My line connected Bay Transport's (BT) and Far Eastern and Western's Railway lines (FE&WR). When BT started carrying post the combination of mine and BT routes this would have likely created a faster route than FE&WR's. The combination of the three companies would have also generated a great deal of additional mail journeys. I had one post train which ran once every month so this would lead to particularly long waiting/journey times for mail going via the Great Highland Railway.

In addition to this, around when the de-sync started occurring BT's trains on the Northern Frontier express were constantly getting stuck due to a lack of one way signs, causing trains to reserve routes for stupidly long distances in on the wrong line. This would have caused massive waiting times for post to be picked up/delivered

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Instability on the Bridgewater-Brunel server
« Reply #139 on: November 05, 2018, 10:34:47 AM »
The integer overflow hypothesis does not seem plausible: journey times are now stored in unsigned 32 bit integers with a resolution of 10ths of minutes, giving a maximum storable journey time of 7,158,278 hours (being 298,261 days, or 817 years).

Rollmaterial - can you be specific as to exactly which carriages that you used to test and how many of them that there were?