unpak-function [How important is Open Source?]

[Leartin]

There are already a bunch of threads about unpaking pak files with different opinions. Clearly, since Simutrans is Open Source, paking files is not a way to effectively encrypt them and hasn't been for the last decade, yet there is resentment against the idea of such a tool. Or is there? Kind of been a while...

You'll notice this is not an extension request - this is a community discussion. Because I wonder: How do people feel about opening sources? Is it still a no go, or should that be a regular part of Simutrans/Makeobj to embrace Open Source as a [Community Mentality]? In this thread, let's assume there are no technical limitations, because we should know what we want before we check what we can do.


My opinion on the matter:
Simutrans is an Open Source Game, and I think we should be an Open Source Community with an according mindset.
Since it's not a 3D game, all the pixels of all assets are drawn 1:1 on the screen. Even without any kind of programming it's hilariously easy to 'rip' some graphics if you want to steal them, and there is hardly anything else worth 'hiding' from the public.
As such, I don't think there is a benefit in closing those sources. On the other hand, being able to open a file, change a value, and continuing on is perhaps the easiest entry in active development you could think of, but still hard enough that nobody should accidentally change something and bug us about why the game doesn't work (as would be the case if Simutrans read sources directly)

Out of respect to the pak128.German devs, I'd only allow new paks to be depaked, and of those only these which were not paked with some new tag. This means people who really don't want their paks to be opened can still ensure this, even while it's discouraged. A kind of backward compatibility that does not stand in the way of a general, more open-minded environment. Perhaps that tag does not even require documentation...

[prissi]

You cannot change retroactively license of pakset. Like MHz cannot make his graphics open source, due to the way he obtained the underlying 3D models. So there must be no shareable paks. End of story.

You can of course save the graphics easily, although animations is another story. Just dumping the images is useless for anything that is not extremely simple. Restoring as useful dat file requires quite some work, since reading dat files and making pak is as much effort as reading pak and writing dat/png files, i.e. the entire content in description read needs to be done as to write out dats. So onless somebady writes the pak to dat/png converter, it is all mood.

Since for newer paks with proper license we have the dat/png files anyway, Thus I doubt there is somebody who will do the work.

[Leartin]

You cannot change retroactively license of pakset. Like MHz cannot make his graphics open source, due to the way he obtained the underlying 3D models. So there must be no shareable paks. End of story.

Not end of story, because that's simply not how licenses work. If his graphics had to be encrypted, then that license was violated once Simutrans became Open Source and thus the key available. But if it's fine that the key is known to everyone so long as it's still encrypted, that would still be the case.

[prissi]

Well, anyway a pakset unpaker is as much effort as rewriting makeobj. So I doubt somebody will do this anyway.

[Roboron]

Simutrans is an Open Source Game, and I think we should be an Open Source Community with an according mindset.

I must start saying that I agree with the premise, but I'm not convinced that this particular idea help with that.

In this thread, let's assume there are no technical limitations, because we should know what we want before we check what we can do.

It is not that we should assume that there are no technical limitations to open sourcing: it is that there are not such technical limitations. You want to go open source? You publish the code, the same you do with the resulting pak file. Done. Easy. We should think of technicalities [note: I don't know if that is the correct translation of the word I'm trying to use] as a way to boost that attitude, but definitely not as a limitation.

In this regard "open-sourceness" is a philosophical problem, and it should be approached as such. If we want people to embrace our open source values, we have to give them incentives to do so. On the add-ons thread, I talked about using an open license as a requirement to upload an add-on to our integrated add-on platform. This is an incentive. You want to make use of our infrastructure and appear in the in-game add-on manager? You can do so, but you need to adhere to the values of that very game and community...

For your proposal I can not see how it could push those values, really. Pretending that developing a un-pak tool will do something for open source is like pretending that developing a decompiler will boost the Free Software Movement. No it won't. The license won't magically change just because you reversed the creation process, and thus the content produced (if it was copyrighted) is not open source and not allowed to be modified.

It can only be of some use if the pak had an open license from the start. But in such a case, the sources should be available... And if they are not, then again, incentivise (or asking) the creator to publish the sources (as I also suggested in the add-on thread) is the correct way to approach this. Maybe if we are not going to ask for sources in the add-on manager this un-paker will become a more attractive idea - but like prissi said it is much more effort than the contrary.

Clearly, since Simutrans is Open Source, paking files is not a way to effectively encrypt them

As such, I don't think there is a benefit in closing those sources. On the other hand, being able to open a file, change a value, and continuing on is perhaps the easiest entry in active development you could think of

You talk like if makeobj were a tool designed to prevent access and modification of source files. I am ignorant about makeobj and I never have been interested in it. But as a programmer I would expect it to be designed to apply compression and thus boost performance when loading pak files. If this is not the case (or is it of too little benefit with today's computers), then something that would help more than un-paking is that Simutrans could be able to load source files directly. Now, having sources directly available for the end user(s) could be very beneficial, since they could see how they are made, modify them easily, and maybe in that process getting interested in creating their own and eventually release the sources back to the community. That would be a better way to embrace open source, in my opinion.

Also much easier to implement, since the code is already there. But I am getting technical!

[Mariculous]

Oh well copyright law on the international scale :O
That's difficult, I won't even try!

According to German law, we first have to determine if a pak is a computerprogram or not.
I don't think it is, but who knows...
If not, modification is permitted, but sharing the result is not!
If it is a computer program, modifications are generally not permitted, but there exist a few exceptions. I don't think any of these apply here.

Thus, de-makeobj might be useful to a few nerds adjusting addons to suit their needs (I had been one of such before I joined the extended community), but it won't help open-sourceness at all and given the required amount of work, I don't think it's worth the effort.

[prissi]

makeobj does some sanity checks and mostly compresses colors. It can also have several object in a single file, which can speed up the first loading dramatically.

Without makeobj, one would always to load simutrans to find if there is a mistake.

Loading likely takes much longer, just because the need to handle two files instead of one. Also if there are more than one dat file using the same PNG, the entire PNG has to processed several times during load from libpng. (Maybe with libzip integrated, instead paks one could consider a zip with many PNG and dat as a full pak, without losing too much of loading speed and probably even save hardidsk/ space).

[Leartin]

You talk like if makeobj were a tool designed to prevent access and modification of source files. I am ignorant about makeobj and I never have been interested in it. But as a programmer I would expect it to be designed to apply compression and thus boost performance when loading pak files.

Since it's old enough to be created in a closed source environment, I do not know if preventing access wasn't also part of the consideration. It was certainly treated as a tool to prevent access later on.

And this is the weird thing: Why does an Open Source game require a file format that intentionally prevents access?
You are right that one solution to that question would be to read source files directly, which was suggested before -> https://forum.simutrans.com/index.php/topic,19925
Another is to see pak files merely as a 'sanitized compression format' that does not intentionally prevent access. In that case, there would be nothing wrong with 'decompressing' them. So, why be against it?

Too much effort? Well, you don't have to actively get involved in something to be in favor of it. That reason can only work as an excuse why one personally won't push it, but not to dismiss the concept.
Copyright law with old paks? I already proposed not to allow opening old paks, which would circumvent that issue entirely, so it's just one of the technicalities that does not dismiss the concept.
Note that this is still Community Discussion, not Extension Request. It's not about whether or not any actual code is ever written, but about perspective.

Naturally, there is a third option: Introduce a new 'sanitized compression format' without access prevention, that can be read by Simutrans and 'unpaked/decompressed' by users, as the open standard format for Simutrans object files in the Open Source spirit. Then, old paks are still locked, those who prefer to hide sources can still use the old format, and only new creations by those willing to use an open format can actually be unlocked, yet it would still be the default.
[Kudos to those who can see this is the same as the original proposal just described from a different angle]


Of note here, despite being a technicality:
Some Open Source licenses require distributors to provide access to the sources. Addons using such licenses thus could not be shared in pak file only, as it would infringe on the license. If the paks were compressed sources and, ideally, a new makeobj version allows to embed the type of license directly in the pak file, it would just be completely fine. [note that we already follow the credit-requirement of some licenses via the (unfortunately named) copyright parameter]

[prissi]

Some Open Source licenses require distributors to provide access to the sources. Addons using such licenses thus could not be shared in pak file only, as it would infringe on the license. If the paks were compressed sources and, ideally, a new makeobj version allows to embed the type of license directly in the pak file, it would just be completely fine.

Providing access is not the same as distribution with the program. Actually some company some time ago did on request mail you a printout of the sourcecode of their contribution to some hardware (was GPL). This completely fulfills the requirements. Thus distribution pak files requires only a licenses files with a hint where and how to get the source. If you sent back images and screenshot of the pak files by paper mail does not matter. Also there are even open source licenses which does not allow for modification.

But this is not about how to fulfill open source obligations. OpenTTD has the grf files and a compiler/decompiler from there very beginning (because they started by reusing copyrighted material).

In Simutrans, makeobj was developed (as the third format) for simutrans to speed up loading and have pak file libaries. Later on especially Hajo voted against a decompiler, due to copyright concerns of images. Still, for makeobj I even wrote a patch (it is in the source but maybe needs updating) with a crude image dump fuction, but no one ever invested any amount of work to develop this any further and full restore png and dat files, because simply no one ever saw the need. Most people have the pak files or can download the source. Because such demakeobj-tool would for sure create one pat and many png per object, since the original files names are not stored.

makeobj has the dump function. You can use this as inspriration to extend it to a proper demakeobj. You would need to add a dump_node function to each node (i.e. the *_writer_t classes). However, this is not trivial due to the "automagic C++" code of makeobj, since you would also have to trace you psoition in the object tree and so on and for images, what kind of image it is. It would take longer than a week of work for me for sure. So I will certainly no work on this.

[Leartin]

Providing access is not the same as distribution with the program. [...] Thus distribution pak files requires only a licenses files with a hint where and how to get the source.

The point is that you need to be able to get the source somehow, though. Which originally could be in a forum post, facebook post, discord message... and those are volatile. If they disappear and you only have the pak, it's still lost to the world, as you can't provide access to the sources and thus are no longer allowed to distribute.
Granted, so far, addons tend to come without licenses anyway, and an addon hub which is discussed in another thread would tackle this as well.

I will certainly no work on this.

Even though you won't do it, and you don't believe anyone else will: IF someone wrote an unpak patch [or a new unpakable pak format], would it become part of Simutrans/makeobj?
- If yes, we are in agreement that access to pak files should not be intentionally prevented, and that the current state is merely a result of Simutranses history. You don't see it as an issue that needs fixing, but are not opposed to someone else doing it.
- If no, you are opposing the idea per se. I'd prefer to hear your reasons for that, since most of what you stated so far goes into "not worth it" territory.
- if you are unsure, let's keep the discussion open. Perhaps someone else has arguments in either direction that sway you. Would be a bit unfair to have someone do all the work and only then start discussing whether it's actually welcome.

[Mariculous]

Some Open Source licenses require distributors to provide access to the sources.

Which license do you have in mind?
Most licenses do not require anyone to ship the sources with the product.
All you have to do is to ensure that people can somehow obtain a copy of the sources (for free).
And you have to tell them that they have this right and where they can get that copy.

So, why be against it?

Personally, I am not against it. I just fail to see the point in spending much effort into this as there is not much you can do with sources resulting from demakeobj, if the original file was not shipped with an open-source license.
Surely, you are permitted to unpak such files, modify them and pak them again (at least according to German law and only if it's not cosidered a computer program), but the result can only be used privately. You are not permitted to publish it.

Open-source paksets or addons on the other hand usually use github or something simmilar.
Sources are available in a more-or-less reliable location, so there is not really a point in using demakeobj on such.

In case of git, each dev has his own copy of that repository, so sources won't get lost suddenly, even if github decides to shut their service down.

So in my holy opinion we should rather spend work towards a reliable pakset and addon distribution platform as discussed elsewhere.

[prissi]

Since you asked me directly: If a useful unpack option (unlike the current image dump patch), why not. Anybody can write this, since makeobj is open source. I certainly find other thing more pressing, like the better cornering of vehicles, Android support, and making a quick release of the current state soon.

[Roboron]

So, why be against it?

Too much effort? Well, you don't have to actively get involved in something to be in favor of it. That reason can only work as an excuse why one personally won't push it, but not to dismiss the concept.

Oh no, I'm not personally against it in the sense that I will definitely not oppose such contribution, if implemented. It is in any case an improvement over the current situation, if only for personal use, so for me it would be welcome.

But if I would have to do it myself I would dismiss it in favour of other options that I think are more aligned with the Open Source philosophy. That's the difference.

Naturally, there is a third option: Introduce a new 'sanitized compression format' without access prevention, that can be read by Simutrans and 'unpaked/decompressed' by users, as the open standard format for Simutrans object files in the Open Source spirit. Then, old paks are still locked, those who prefer to hide sources can still use the old format, and only new creations by those willing to use an open format can actually be unlocked, yet it would still be the default.
[Kudos to those who can see this is the same as the original proposal just described from a different angle]

Different angles matter for this kind of things :-)

Currently makeobj can't unpak files, that's a technical limitation.
But if makeobj could unpak files and did not because of copyright, it would be not a technical limitation, but a DRM. I'm against DRM software because it conflicts the open philosophy.
Using other format circumvents this issue entirely. Yes it would be "practically" the same, but "philosophically" it would be not, and that really matters for me.

Also I 100% agree with you on the open standard format. that's a good point.

[wlindley]

The existing pak and save-file formats are complex, cruft-laden, and obscure.  Worse, save files tend to break with relatively minor changes to the program source unless you are aware of all the issues.

Wouldn't it be better to have well-documented pak and save-file formats that could be easily read, and created, by tools you could write in any language?  Imagine a terrain editor or world creation tool written in your favorite language.  Or imaging being able to write (or have someone else write) any kind of tool that could analyze or modify paks or saved games.  How about a Blender plugin that writes pakfiles directly?

This would require:

   
• Replacing the reader and writer code to comprehend the new formats.  All the old 'cruft' dealing with a zillion different ancient formats would go away, being instead a call to some existing library we could find that knows how to read/write compressed json or whatever style we select.
• Keeping the old pak and savefile reader code (from makeobj and from the existing primary game) and, once the data are read in, writing with the same new code from above.  This would let you convert old paks and old save games.

Thoughts?

[Mariculous]

Wouldn't it be better to have well-documented pak and save-file formats that could be easily read, and created, by tools you could write in any language?

My personal opinion about this:
Whilst human readable saves are desirable, there are more important factors.
The "perfect" savegame should be minimal in size, save/load fast as lightning, provide some error-correction mechanism as well as being human readable.
This is a tradeoff. In my opinion human-readability is the least important factor.
Do you have any specific serialisation library in mind that results in small save sizes whilst being human-readable (after decompression)?

Paksets are a different story.
There's not a big memory impact. Pak128-britain-ex has 850 MB of sources resulting in 650MB of obj files.
A small issue here is loading time due to many small files. Zipping these might fix the issue.

I am not sure about the impact of parsing dats and preprocessing the images when loading the pakset from source. This might potentially slow down loading times quite much.

[wlindley]

Save files almost certainly should be binary... just something better-documented, and more resilient to version changes -- particularly adding new items to a data structure.

I have wanted to add a meaningful "date of object creation" to every type of building in the game (houses, factories, stations) but my changes have been rejected because "that would change the meaning of the existing 'time of object creation' value" (the difference between a countdown timer, standard game ticks-from-game-start, and a year-month value).  Other folks have wanted to add other object data but run into the same issue.

If instead the save file were flexible enough to allow tagged values, then not only could we make development simpler, but by skipping "same as default" values when saving, the save files might achieve size savings... and by specifying that default values should be used for objects when loading, it might be even made possible to load newer saved games on older compiled binaries.

[Mariculous]

I fail to see how this could work in practice.
The tag itself consumes memory. If we want to be able to skip a value, we have to store a type tag with each value otherwise we cannot determine if the next expected value was skipped or not.
I doubt that skipping a few values can compensate the additional memory of the tags.
The only way to find out is to implement this and then compare it against the old format.

Still there is a way to achieve more flexibility moving the savegame structure "documentation" away from the code to a (possibly human-readable JSON) savegame structure file that is generated with each savegame version of simutrans.

[prissi]

The xml_bzip savegame format tags each value by its type (to keep the size only 1.5 times larger). It detects wrong formats of parameters while loading in a savegame. Still, broken savegames are rare nowadays and mostly due to program errors forming a bad save. That would be the same trouble with any other format. And a broken file by hardware errors is detected by the compression alogrithm, but that would be dead no matter what encoding.

The pak format is actually very flexible with a versioning built in. Simutrans happily loads paks from 2002, 19 years ago. Try to throw such old files to some other modern programs ...