Loading Sourcefiles Directly Into The Game

[Flemmbrav]

This has been a wish of me for a long time.
What do I want? I want to have a way to load a Pakset with sources being readable for users still.
Currently, Paksets have to be shipped containing a lot of .pak-files for ingame objects, as well as some "trunk"-data-files, that are open source already.
Compiling objects to pak-files takes time and some initial effort (you might have to find makeobj first, understand how to use it and stuff).
I wish for Simutrans to be able to read objects in a way, that does not need extra tools.
I imagine things like a zip containing .dat and .png-files. Or even raw folders with these inside.
This shall not replace the already existing pak files.
This is more about an additional way to load objects into Simutrans.

Benefits of this would be:

- easy testing of new objects
- easy and fast adjustment of already existing objects
- users of Paksets that distribute in that way could edit their objects without extra software resulting in a lower initial barrier of creating your own objects for Simutrans
- objects that are not open source can be distributed the old way still

The third point is my main goal here. I'd love to see people being able to edit Simutrans with on board tools of their own OS. This is how I got myself into editing games first. The game "Battle for Wesnoth" provides exactly this. I could just edit my own objects without any further help. Even tho I might have used this to cheat the system first, I actually ended up drawing myself my own army.
I would not have done that, if I'd have to figure how to get my objects first.
Also, my first objects for Simutrans have been made with Online-Dat because back then I did not know how to manipulate a program to run with commands. This might sound silly to me by now, but back then I've been super glad about this. In the end, a low initial barrier to work on an open source program should be one of the main goals of such a program, if it wants to keep being alive. I understand that this is hard on the coding side of things, as you really want Simutrans to run from compiled code. But editing graphics might be a nice first step for future devs.

As you might have figured before, I'm not here to show you a working patch. Mainly, this is because I don't really want to get too deep into the development of Simutrans itself, but rather work on our Pakset. Still, I thought of some things that might not be a given:

- where exactly should the pak-loader be bypassed? Would it be enough to look for .dat-files in the folder, and just read them in with the same routine that does it with .pak-files after unpacking them?
- does this require the .png-files to be RGB555?
- would it take significantly longer to load objects this way? What can be done to get a good performance out this still?
- could this support shortened dat-files?

[prissi]

It would take significantly longer, since one would need to handle much more files, and each image would need to be transformed into the ingame format. I am also somewhat concerned that memory leaks may be in makeobj.

I would estimate that loading a pak from source would take between 5 to 20 times longer than from paks, using the current makeobj source code (since for each pak the png file is opened again etc.) Unless reqriting the code and keep all pngs open (since you do not know which images will be needed again), I see little to speed up very much. You still need to parse the files and decode/encode the images. Using the existing code, adding this capability is easy though. Therefore, this capability would be reserved for addons.

Finally, I think if we talk addons only, then a simple folder with the files would be the way to go. Simutrans has no zip support, Z and bz and zstd are all block format, encoding single data blocks.

[Ters]

Loading readily editable files directly also means that Simutrans would have to expect the files to be broken. Simutrans doesn't have particularly good error handling. This also increases complexity and running time.

I think that anyone who can properly create the images and get the dat-syntax right, will have no problem running makeobj. And tools helping out with those things, will also benefit from have makeobj as a free-standing tool that they can invoke.

The only advantage I can see with plain text files directly, is that player can edit the price, capacity and power, but I suspect that would be most used for cheating.

[Vladki]

What about adding third option in the start dialog where pakset is chosen. Now we have the option to load pakset with or without addons. Third option would be: run "makeobj pak<size>" on addons/<chosen pak>/ folder and then load whatewer is produced by that?  Output could be logged somewhere for further investigation.

[DrSuperGood]

A lot of the concerns about loading performance could be addressed with optimizations over time. For example image conversion results could be automatically cached if conversion is too slow and to some extent files could be processed in a multi-threaded way. Technically images do not even need to be loaded until required which could have the potential of shortening initial loading time in exchange for potential stutter in the form of asset stalls during gameplay. For complete, ready to use, paksets there is the possibility of bundling them as single compressed archives to minimize filesystem overhead, storage footprint and I/O bandwidth.

The main issue is such a change is non-trivial as it would require a lot of code to be reviewed, modified, tested and iteratively optimized. There are also many edge cases that have to be considered such as how multiplayer handle these new files and how to deal with malformed files.