Topic: Bridgewater-Brunel no. 1 - Great Britain sized map (no. 2)

[jamespetts]

I am away this week without access even to my secondary desktop, so I am afraid that I cannot fix this at present - please let me know whether it resolves itself to-morrow.

[DrSuperGood]

Seems to have resolved itself with the nightly deploy.

[DrSuperGood]

Server offline again... Again seems around time of reboot, or slightly earlier.

[jamespetts]

My apologies for the difficulty: the saved game appears to have become corrupted. I have now restored from the backup.

[Rollmaterial]

The passwords are gone in the backup save. Apparently there has been some progress since the reboot. I would be fine with rebooting with a correct save, would anyone oppose that?

[jamespetts]

Can you elaborate on what you mean by "rebooting with a correct save"?

[Rollmaterial]

I meant to restart the server with a backup where the passwords have been kept. It has happened before that the server got stuck with a corrupt save and was restored with a backup without passwords.

[jamespetts]

I am not sure that the server has any left with passwords now and it would take a very long time laboriously to check each one; would it not be easier for players simply to restore their passwords?

[DrSuperGood]

You have loaded the wrong save. The one restored to is ancient, losing many hours of peoples work.

Bellow is a link to the most recent save game before the server crashed. Or at least it was the most recent one I have. It is not corrupt.
https://www.dropbox.com/s/o8dxwltzod25jm2/Extended%20Server%20Game%2030082018.sve?dl=0

This save is considerably in the future compared with the current one. I completely redid the transport for several islands including setting up complex bus routes.

[jamespetts]

I had not restored "the wrong" saved game - I had restored the latest apparently uncorrupted game available in my set of backups. I note that there are people online now - I will have to check with them before now resetting whatever they are doing now and re-loading this game.

[DrSuperGood]

I had not restored "the wrong" saved game - I had restored the latest apparently uncorrupted game available in my set of backups.

So the server does not keep a copy of the most recent made multiplayer saves like the clients do? Or was it that all of them were corrupted?

[jamespetts]

The server keeps a copy of the single latest save. That was corrupted. The server has also been configured to have a rotating backup of 6 prior versions. The latest in time of these was also corrupted, so I restored the second latest in time, being the most recent apparently uncorrupted saved game available.

I am now having trouble connecting to the server to try to speak to the people who are playing.

[ACarlotti]

Have you ever been able to (or tried to) diagnose why saves on the server are getting corrupted?

[jamespetts]

I have not tried this, as I am not sure quite where I would start in the diagnosis. One hypothesis - so far untested - is that the corruption might happen if the server restarts to apply an update to Linux, which is set to occur automatically for security reasons, during a load/save cycle.

If you have any ideas for looking into this, I should be most interested.

[DrSuperGood]

Restarting during or shortly after save/load cycle would explain the corruption. Files are not written out immediately to disks unless explicitly ordered to do so, instead they will be streamed out over time thanks to file caches.

The solution would be to schedule Linux updates to happen when the server is restarting with the nightly.

• Build Nightly.
• Warn of impending server shutdown.
• Shutdown Server.
• Apply Linux Updates (if any).
• Restart system (if updates applied, or maybe anyway).
• Start Server.

Linux is cryptic so I do not know how one would do this. However surely it must be possible...

[ACarlotti]

For that hypothesis you could try logging the times at which saves are created and loaded, and compare these to the times at which the server is restarted. You could also try applying mitigation along the lines DrSuperGood suggests.

However, I don't see how that cause should lead to earlier saves being corrupted too, unless those earlier saves were made not long before the most recent one (so were still in a cache when the server made the next save). I think having earlier saves corrupted too does rule out a bug in Simutrans, but I'm not certain of this.

Another thing to consider (although I don't know how testable this is) is whether the corruption is down to small errors (perhaps indicative of a dodgy/failing disk), or whether the save is significantly damaged or truncated (which would be more indicative of it having never been fully written in the first place).

[jamespetts]

The server is professionally hosted and corruption usually consists of a file a fraction of the size of the full save (e.g. ~10MB instead of ~76MB). The backups are made by discarding the oldest backup, copying the second oldest backup to the name of the oldest backup and so forth until the current saved game is copied to the newest backup slot. There are a total of six slots and this rotation does not occur unless the fresh saved game has changed, so this would, I imagine, explain the way in which the very latest backup becomes corrupt. This would mean that the latest uncorrupted backup is in fact the most recent uncorrupted game that was ever saved by the server.

[DrSuperGood]

Except that could not be the case as the save I provided was more recent than the one restored to. I am pretty sure I was still connected when playing...

Are we going to restore to the version I provided? If not I will just have to spend more hours rebuilding what I lost sigh...

[ACarlotti]

copying the second oldest backup to the name of the oldest backup

I hope you're just renaming the old files, and not making new copies of them? Otherwise that would explain why older saves are getting lost, and also provides a fairly simple fix for that issue.

Except that could not be the case as the save I provided was more recent than the one restored to

I don't see any contradiction here: the fact that you have an uncorrupted save stored locally does not mean that a corresponding save was successfully written to the server's disk.

[DrSuperGood]

I don't see any contradiction here: the fact that you have an uncorrupted save stored locally does not mean that a corresponding save was successfully written to the server's disk.

And why would it not be successfully written to the server? I played for many minutes after the save was made which should be more than enough time for all data to be written out. Of course this assumes that the file cache is set to periodically write out changes and not just wait until cache eviction to write them out, which I guess would be a sensible settings for some server applications.

[jamespetts]

Thank you for your thoughts. Firstly, as to the backups - I have looked at the old backup rotation script that I had written back in 2012 to deal with backing up saved games. This used the Linux "cp" command. I have now modified the script to replace the "cp" command with the "mv" command for all but the first backup (as it is important that the first backup is indeed a copy, as the original file needs to remain untouched). However, this would not address the issue described here, as the only corrupt backup was the first, which was (correctly) a copy of the current server save file.

I have looked over the logs and cannot find anything of significance: what appears to have happened on the 31st is that the saved game became corrupted at some point (I do not have the exact time as many of the logs were not timed - I have now modified this) and there came a point when the game could not be restarted. If Dr. Supergood's save was an automatic rather than a manual save from his end, I am not clear on why this failed on the server other possibly than that this was relating to cache settings, which I am not entirely sure how to adjust (a brief search has not yielded clear results - do either of you know how to adjust these?).

I have now restarted the server with the restored saved game provided by Dr. Supergood; I have also taken the opportunity to increase the maximum commuting time tolerance to 3 hours and also increase the maximum visiting time tolerance to 6,015 minutes and the minimum visiting time tolerance to 12 minutes (up from 8) to reflact changes made to the pakset since the server game was originally created.

I do note, however, that Dr. Supergood's saved game also does not appear to have password data retained. I have backed up the saved game that was in use on the server until this restoration in case of difficulty.

[ACarlotti]

The 'sync' command (which optionally can be given a list of files to sync) should "Synchronize cached writes to persistent storage". So calling that after the copy (and perhaps also once after finishing all operations) would be a good idea. 'sync' is intended for exactly this sort of situation.

The other thing you could do is to first copy the save to a file with a different name to the ultimate backup, then sync the write, and then rename the save to the actual backup name. This should ensure that you never end up with a partial backup save file (except under the temporary name).

Also, I would expect replacing cp with mv (where possible) to significantly improve the time taken to save the backups.

[DrSuperGood]

I do note, however, that Dr. Supergood's saved game also does not appear to have password data retained. I have backed up the saved game that was in use on the server until this restoration in case of difficulty.

Obviously it does not as I am not the server... Only the server has the password data for security reasons (if I had it, I could login as anyone I wanted to). Password data is stored in a separate file next to the game save. The game save file is transferred to clients so for the passwords to be kept secure they must be stored separately in an unrelated file that is not transferred to clients but is used by the server to authenticate passwords.

Also, I would expect replacing cp with mv (where possible) to significantly improve the time taken to save the backups.

A modern drive system should be writing large files at over 20MB/sec so even the massive server save is at most 5 seconds, probably far less.

[jamespetts]

Thank you for that suggestion: I have now added sync commands where sensible, and also replaced one instance of a forced shutdown with a shutdown using nettool in case this was responsible for the difficulties.

[Rollmaterial]

The server has been down for the last 4 hours apparently.

[jamespetts]

Thank you for the report. This seems to have been another corrupt saved game incident. I have restored from the backup, but I am not sure what is causing this.

[Rollmaterial]

What is the value of max_diversion_tiles? I am trying to replace the bridges over Crandon station but cannot remove one of them, even though I have built a 12-tile long diversion.

[DrSuperGood]

What is the value of max_diversion_tiles?

I think about 8-10 tiles. A diversion must also be of equal or better axel load. So for example you cannot divert a cobblestone road over Tar MacAdam road despite a higher speed limit but you can divert it over Tar MacAdam (heavy) road.

Honestly I do not like the public way system at all. It happily turns your bridges into public ways inside a city but it does not take the maintenance away from them in the process. It also is a real pain to upgrade bridges to higher tonnage as a result. It also will not let you downgrade public ways to at least the current public way type (inner city for pavemented roads, otherwise between city for non pavement).

[Rollmaterial]

A diversion must also be of equal or better axel load. So for example you cannot divert a cobblestone road over Tar MacAdam road despite a higher speed limit but you can divert it over Tar MacAdam (heavy) road.

Thank you, I am perfectly aware of that

Anyway, I lost my first attempt in a crash, and my second attempt succeeded. However, whether a bridge will be removed seems a bit random at times.

[DrSuperGood]

However, whether a bridge will be removed seems a bit random at times.

Unless the bridge itself is a diversion it will not start out as a public way. City buildings cause the ramp/span ways to become public ways if they are created in the surrounding times. If a bridge is inside a city it need not necessarily become a public way and as such can be deleted at the owner's leisure. However if it does then the annoying diversion rules apply.

[SuperTimo]

All the extended servers seem to be down at the moment. It is not even possible to access the server list page: http://list.extended.simutrans.org:8080/list

[jamespetts]

That suggests an error with the listing server, which I do not run. As a workaround, you can access thus server by typing net:bridgewater-brunel.me.uk in the load game dialogue.

[SuperTimo]

Thanks, the server is indeed running, I tried putting the URL but put in the https:// url which obviously didn't work.

[DrSuperGood]

Cannot connect to the server. Not sure if its because people are joining or because it has crashed.

[jamespetts]

Thank you for the report. We seem to have had saved game corruption again: I have restored from the latest backup, so this should be running again shortly.