cannot load save data

[Phystam]

In some cases, the saved data cannot be loaded.
Finally, I produced such data. Please find it.
https://www.dropbox.com/s/0q5v957jd8rpbwv/PHNS-SB-bak15.sve?dl=0

[jamespetts]

Thank you for the report. Generally, problems with loading saved data are problems caused during saving rather than during loading, and thus I cannot reproduce the problem from the corrupt save data. What I need is a reproduction case of the circumstances in which the corrupt save data were generated in the first place. Are you able to assist with that at all?

[Phystam]

This save data was produced during force-sync by nettool. In some cases, after saving such a broken data, the reloading process may stop as I have reported before...

[jamespetts]

This save data was produced during force-sync by nettool. In some cases, after saving such a broken data, the reloading process may stop as I have reported before...

Is using force-sync with nettool a means of reproducing this reliably under any circumstances? If not, I will need sufficient instructions for a reliable reproduction case.

[turenar]

I also encounter this problem.

https://private.turenar.xyz/private-files/54b70baeb9580646e0c12324f27f88a2e4361e6fc1f49d87598e6c32dc3246be/extended3.sve

If I save immediately after loading this attached save, the newly saved game cannot be loaded.
After playing the attached save for hours without pause, however, it seems I can save correctly.

I don't know what's the cause or other reproduction cases but I hope this info helps your investigation.

[jamespetts]

Turenar - thank you for that. Unfortunately, however, I am not able to reproduce this either with a debug build or the released nightly build. Can you be any more specific as to the necessary conditions for causing this error to occur?

[turenar]

Sorry for late response.

umm... I've confirmed this issue on windows (my usual environment) and linux (with #0a95ad3, http://bridgewater-brunel.me.uk/downloads/nightly/packages/Simutrans-Extended-Complete.zip modified on 2020-04-20 05:19).
Just load my save and save to a new file immediately, then load the new save. The result is freezes with "Unloading map...", which can be confirmed with loading PHNS-SB-bak15.sve.

I don't have other conditions, sorry.

[turenar]

freeze (with cpu core 100% load)

[wlindley]

I had traced this awhile back, and found that after the old map was removed, and the new one started being read, that the number of industries in a city should have been something like 100 but instead the for-loop in the code wound up trying to read four billion cities.

I do not remember the exact subroutine or variable names, nor do I understand why the protections against reading past end-of-file were not causing the program to error or crash rather than freeze.  (It also would be nice if the screen dialogue were updated after the unload was really complete.)

Does that help?

[wlindley]

p.s., I really wish the file I/O was not so ancient and brittle.  Even the alleged XML format instead of having "<city>..." has just a bunch of integers: "<i8>3</i8><i16>22</i16>" which is entirely against the point of having an XML format.  The file format should be clear, easily understood, and easily accept when variables are added or removed.  Perhaps we need an entirely new file format that is more UNIX-like, perhaps JSON:

Code Select Expand

{
    "name": "factory",    "built": 1950,    "capacity": 200
}
Surely there must be a library we can use that does that all for us?

[jamespetts]

Sorry for late response.

umm... I've confirmed this issue on windows (my usual environment) and linux (with #0a95ad3, http://bridgewater-brunel.me.uk/downloads/nightly/packages/Simutrans-Extended-Complete.zip modified on 2020-04-20 05:19).
Just load my save and save to a new file immediately, then load the new save. The result is freezes with "Unloading map...", which can be confirmed with loading PHNS-SB-bak15.sve.

I don't have other conditions, sorry.

I can reproduce the failure to load that saved game, but that only shows that the PHNS-SB-bak15.sve game has already been corrupted (the number of industries being excessive is no more than a symptom of save file corruption). As already explained, unless I can reproduce the corruption being caused, I cannot realistically find and fix this problem, and there is no practical way to reproduce the corruption being caused when the only information available is the already corrupted save file.

I cannot reproduce this problem with a non-corrupted game.

As to the ancient and brittle save file format, no doubt there are libraries: if anyone with the relevant skills/knowledge is willing to spend the hundreds or thousands of hours necessary to convert the save file format in every place in the game into something more robust, whilst simultaneously maintaining full backwards compatibility and not significantly increasing the save file size (which would be a serious problem for online games), then that would be splendid; but if somebody wanted to spend that much time improving Simutrans-Extended, it is likely that it could more efficiently be spent on things relevant to economic balance.

[Mariculous]

and not significantly increasing the save file size

That means XML and JSON are definitely NOT an option.
We have to stick with a binary/bytestream format but it could be improved cutting it into chunks, where each chunk has a few header bytes stating the type, size and eventually a checksum of the following chunk, thus the game can detect broken chunks on load.
Still, the save would be corrupted and we still don't know what happened on save.
Such a binary format would indeed be more robust, but still not much more human readable.

Further, as James mentioned, implemeting this would be a very huge thing, in fact implementing all rdwr methods from scratch, retaining and calling the old code to load old savegames.

[prissi]

In principle there should be an integrety test, since most games are compressed and hence there coul;d be a checksum from the compression. A broken game will be broken in any savegame format, because if save information is corrupt, it cannot be guessed, since otherwise there would be no need to save it ...

[Mariculous]

That is partially true for extended.
Things like like pasenger routing is kind of redundant information. It does not neeed to be stored in the save as it can be recalculated from stops and lines. However, it speeds up loading times quite a lot.
Same goes for stored privartecar routes.
Other informations are not that important like last months stats and stuff like that.

However, the main reason why many file formats usually store data in blocks with metadata is not to attempt to somehow fix broken blocks but to allow for more flexibility and to reliably detect file corruption.
E.g. such a save format could be designed to support upward and downward compatibility of saves.
Everything we don't even know it existed will be ignored, everything missing will get the a default value assigned.

We should not try to somehow load broken saves, but we should know where our data should be located in that file without iterating the whole loading process up to that point.


Anyway, again I don't think implementing a new savegame format currently makes sense except if someone is really very bored...