News:

Simutrans.com Portal
Our Simutrans site. You can find everything about Simutrans from here.

Simutrans flatpak

Started by Roboron, February 10, 2022, 02:22:27 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Roboron

I've just pushed a repository with an initial flatpak manifest for Simutrans https://github.com/simutrans/flatpak

This is working already, and can be built following the instructions (some information may be missing, i.e. installing the sdk). What's more, the sandboxing works pretty well, as Simutrans is only able to access networking, audio, and graphics. It does not even have access to the home directory.

But I only got it working after applying many workarounds, so I am not convinced I should submit this in this state. I'll expose the problems I encountered and the patches (or hacks) I had to apply.

1. Surprisingly the default runtime has most of our libraries, and only bzip2, miniupnpc and fluidsynth are not there. Fluidsynth however is available as shared module so we don't need to take care of it either. Only bzip2 and miniupnpc needs configuration on our part. I already set up them.

2. Because we can't access the system, I had to include in the manifest an external soundfont.

3. I encountered problem while building the latest release source with revision. CMake does not have an option to set the revision manually, and Make didn't work properly because git tries hard to find a repository where there is none, and output garbage as result (or even worse, it picks this flatpak repository commit). I had to add "echo "#define REVISION 10421" >> revision.h" between configuration and build to make it work. This needs fixing.

4. The source released for 123.0.1 also lacked the ".desktop" file, and this forced me to include this file in the repository manually. Please do include this file for the next release (@prissi).

5. Having to define install directories one by one was a bit unpleasing, the next CMake improvement (after manual revision number) should be implementation of https://cmake.org/cmake/help/latest/module/GNUInstallDirs.html

6. While testing simutrans in the sandboxed environment also found a bug in the pak_installer.sh, where paksets with no "simutrans" directory were not installed in the correct location. See https://github.com/simutrans/flatpak/blob/main/get_pak.patch for a quick patch

7. Finally, let's talk about the sandboxing, because it was the most challenging part to get it working. This is the typical structure of a linux simutrans installation:

/bin/simutrans -> binary
/usr/share/games/simutrans -> data (usually patched by distributions)
~/simutrans -> user savegames and other stuff (hardcoded in simutrans code, sometimes changed by distributions to ~/.simutrans)

And this is the flatpak equivalent:

/app/bin/simutrans
/app/share/games/simutrans -> ¡¡Read only!!
~/.var/app/com.simutrans.Simutrans/data/simutrans  -> Only directory with write access ($XDG_DATA_HOME)

And this is the challenge: Simutrans wants to read AND install files (paksets) on /app/share/games/simutrans, but simutrans can NOT install paksets because it is read-only.

Simutrans does not have currently a way to load and install paksets from non-install (data) directory, which would be the appropriate solution (and not only in flatpak, but in linux distributions installations too, since before the pakset installer one needed root access to install paksets). Therefore I had to add a "hack" to the code to make it work

// Make simutrans copy the installation when opened (workaround)
system("\cp -r /app/share/games/simutrans/ $XDG_DATA_HOME 2>/dev/null")


This will copy the base files to ~/.var/app/com.simutrans.Simutrans/data/simutrans when simutrans is opened, allowing me to set that directory in the code as the actual data directory (even if it is just a mirror). Of course this is not the correct approach, it is a quick workaround to continue with the flatpak adventure, alongside with setting singleuser_install = 1 to avoid having to patch the user directory too (since it is inaccessible).

8. Related to this last point, we really need to adopt the XDG Base Directory Specification, if only for the user data directory. I know this point has been controversial in the past, but let approach it differently. This is what XDG Base Directory Specification says about  $XDG_DATA_HOME:

Quote$XDG_DATA_HOME defines the base directory relative to which user-specific data files should be stored. If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used.

Take in mind that the default is for $XDG_DATA_HOME to be unset, it will be only set if the user has set it. So far Simutrans does not save anything in a hidden directory because otherwise users would not find it easily (or this is the reasoning). But if a user has set $XDG_DATA_HOME he probably knows where to look for (hidden) files. So we should respect this user decision and only if $XDG_DATA_HOME is not set, use $HOME/simutrans then (instead of $HOME/.local/share, so we keep the best of both worlds). This will allow for flatpak and some distributions to require one less patch, as flatpak sets $XDG_DATA_HOME to the aforementioned directory.


Of course, I will work on some of these things next, but specially on the last two some feedback is welcome. You can also take a look at the metada.xml file to see if there's something missing, and try building the flatpak if you can.

prissi

#1
I think flatpak is even less portable than the simutrans binary linked statically with minupnpc and bz2. Neither Suse nor Debian nor Redhat support them out of the box. So my personal feeling is that the effort is more rewarding on other things.

However, you raised valid points about the directory structure. This is a difficult (and very long) story and very OS dependent. For instance, Android wants them in its directory, and with MacOS, I am not sure ... And for the program directory, the use of an external window installer. (That decision of windows to not allow programs in Programs to write to their own directory was just braindead, after they allowed installation into AppDataLocal ... )

Back to Simutrans: Internally it would be little effort to have more than one path for looking for paks and addons, like the font search shows. I think the best way would be to allow as many locations as possible by default, probably sorting with preferences

prissi

I think my last message was too negative. In order to get broad exposure, I think it is needed to get Simutrans into the stores. I am working on the Windows store and Android Play store, and you have nicely taken care of the Steam store (which indeed brought many new users). Beyond the stores for the stable, there are still the repos of the distributions; with the slow paced updates of Simutrans stable releases, the version is often not too old either. However, the exposure varies. for instance in Ubuntu Simutrans is only found with apt install but not with the graphical tool.

With this background, I wrote my thoughts on flatpak. It would be another thing to maintain. Would it really broaden the exposure much? Or could we use the time better using other ways to "bring Simutrans to the masses"?

Roboron

I did not do this because I thought it would bring Simutrans to the masses (for that, my work on Steam is much more important). I did this out of curiosity because I wanted to look into flatpak distribution. However, it can bring some extra exposure (definitely not as much as publishing to Android of course, but I don't have an Android to help with such task), since Simutrans will appear in most linux "software stores" (including GNOME Software Center and KDE Discover, which are not distro specific) that already make use of flatpak to install applications when available. Whether or not such stores also includes system packages also is a matter of distro politics (with Ubuntu only using its "snaps" nonsense which I expect will be dropped in the future as it did not gain any traction outside Ubuntu) . While Flatpak is still in maturity phase, I can already see some distributions making the jump to "flatpak-only" software centers.

elementary OS is moving to Flatpak and it's not a joke (article doesn't have date, but this already happened)
=> https://ubunlog.com/en/elementary-you-go-to-flatpak-and-it%27s-not-a-joke/

I can not predict the future to say if this will become the standard of app distribution, but a tendency there is. You said that Red Hat does not support them out of the box, but you are wrong. The have been rolling "Fedora Flatpaks" for its own packages since Fedora 34.

Anyway, the issues I discovered during this process were not flatpak specific for the most part, it just were more evident because of the constrictions I had to face. Let's talk about MacOS again (unfortunately). I'm pretty sure that our current approach is incorrect. The app bundle should be self-sufficient, including inside the bundle files Simutrans needs to run (the data), so it can be installed into Applications whit a simple drag-and-drop (which currently can't because it needs the top directory) . But the app bundle is also read-only, so Simutrans can't install paksets inside. Does this sound familiar? Yes, it is the same problem Flatpak has. The translocation was then our particular workaround to have data and paksets all in one place, but it is not the appropriate solution, which would be solved the same way that for flatpak: installing and reading paksets from a directory we actually have access to (the user Library directory in this case I guess). Actually this was suggested in the past, see https://forum.simutrans.com/index.php/topic,9609.msg90117.html#msg90117

About maintainability, if I end up submitting this (maybe for the next release after the issues have been addressed), you can count on me to maintain it. For the most part it should just be a matter of updating the link to the new sources, much easier than Steam...

Sirius

Hey there, openSUSE user here.
flatpak is in the main repo.
Any openSUSE user can install it as simple as sudo zypper in flatpak
Have fun :)

Didn't check the simutrans flatpak yet.

isidoro

@Roboron: Ubuntu "snap" also has a very serious issue: updates of programs distributed with this tool are mandatory.  So, the freedom of what's installed or not in your computer is taken from the owner of the computer to the software developer.

When this issue was raised to Canonical, the corporation said that it is a policy of the company and won't be changed no matter what the users of Ubuntu will ask.

I had a problem with an update forced by snap that broke the program I used during pandemic to broadcast my classes and had no other chance than to block the Canonical snap server in my router.

So I guess "flatpak" is the right way to go if one wants a portable binary distribution without the problems of static compilation (size of the programs, real shared libraries than can be updated for all programs, etc.)

prissi

Ok, let's split flatpak and directory disucssion.

Roboron

#7
Well well, after the changes to the directory structure and pointing to the SVN r10630 (current head), there are no patches needed any-more for the flatpak build 🎉

Which means in other words that this is ready to be published on flathub when the next release is done.

Note: Currently manually specifying the revision number for building in the yaml because it won't pick it automagically.

Quote from: isidoro on February 11, 2022, 12:21:09 AM@Roboron: Ubuntu "snap" also has a very serious issue: updates of programs distributed with this tool are mandatory.  So, the freedom of what's installed or not in your computer is taken from the owner of the computer to the software developer.

Oof. That's... worse than I thought.

R1dO

Even though this thread is about flatpack and not snap, i feel the following quote needs some extra information cause it could lead to some wrong expectations.

Quote from: isidoro on February 11, 2022, 12:21:09 AM@Roboron: Ubuntu "snap" also has a very serious issue: updates of programs distributed with this tool are mandatory.  So, the freedom of what's installed or not in your computer is taken from the owner of the computer to the software developer.

The whole idea of using snaps is that it takes the burden of updating software away from the user. It is not taking away any freedom by itself (snap is usually optional on linux systems and it is even possible to fix a specific version, see later on in this post).
Should a developer choose they want to provide multiple versions it is even supported by snap. An example of how that works from the user perspective can be found on the following Ask Ubuntu thread.

And if the developer does not want to provide multiple versions: "snap" has limited support for running older versions, albeit it can only go back one version. This is described in the Snapcraft IO documentation under the section "Revert to a previously used snap".
If there is an issue here I would argue that it is because several months after reverting a user would most likely forget ever using that feature. Since snap honors a user's wish (you want to use *this* specific version, fine ... i will not bother you about updates again) `snap refresh` will still tell the app is up-to date even if there are (multiple) newer versions available.

Of course the quote is true if the developer (assuming they only provide a snap version of an application) chooses not to publish multiple versions to the centralized repository snap gets it's programs from (snapcraft.io?). But in that case it is no worse than a developer which chooses to only provide a download link to the latest version of a product on their website. Not something you can blame on snap.

Hope this clarifies a bit