News:

Use the "Forum Search"
It may help you to find anything in the forum ;).

CMake build support

Started by ceeac, August 05, 2018, 08:50:34 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

ceeac

This patch is a work-in-progress attempt to provide support for building Simutrans with CMake.

Currently this patch only works on Linux, not on Windows or macOS.
What should work:

  • Support for clang and gcc compilers
  • Building Simutrans
  • Building makeobj
  • Building nettool
  • Using CCache to speed up compilation (if available)
  • Automatically selecting a suitable graphics backend based on installed libraries (can be overridden manually)
  • Freetype support
  • MiniUPNP support
  • Automatically pulling revision information from git or svn
  • Profiling information support (-DPROFILE)
There is also a github branch available: https://github.com/ceeac/simutrans/tree/cmake

Instructions for building (with the patch applied):

mkdir build
cd build
cmake ..
make


Instructions for building from git:

git clone https://github.com/ceeac/simutrans.git simutrans
cd simutrans
git checkout cmake
mkdir build
cd build
cmake ..
make


Please test and provide feedback.  :)

Ters

We're having trouble keeping the current Makefile system working on all platforms, and keeping the Makefile and Visual Studio projects synchronized. Yet another build system in parallel is not what we need. If the build system's dependencies are rather small and can generate the Visual Studio project from some cross-platform model, it sounds promising as a complete replacement for what we got.

Personally, I've always had trouble getting cmake to work on mingw.

prissi

There is an autoconf working for mingw and linux (and possibly even MacOS, if someone would give feedback). I have no experience with cmake, and I am not sure how easy this is to install. Currently the MSVC is maintained seperately from the rest, and if thing could get streamlines, it would be really nice. But my past experience made me very cautious on portability promises.

I think Dwachs uses clang, and I think no changes are needed to the makefile. I could also use our makefile for the nightly crosscompilation, so it seems quite robust and tolerates even old-stable debian (what my server runs).

Cmake would need to fulfill all this. But then it could be of great help. I may test it given time later in August.

Mariculous

#3
*digging deep*
Cmake made its way into extended.
Is there any chance to get this integrated into standard as well?
Many IDEs nowdays can use CMake as a project file, so does mine. It can generate buildfiles for many build system, including but not limited to makefiles and vxcproj.

Mariculous

#4
I just backported the CMake support from extended.
See this PR https://github.com/aburch/simutrans/pull/15

Thanks to ceeac for the great job.

prissi

#5
Somehow Cmake is broken. It cannot generate builds for MSYS2 Makefiles, or I am too stupid for this. It does not even generate a MSVC project, as it fails to find zlib (installed by VCPKG).

If anyone can write clear reproducable instructions on how to generate a mingw Makefile or a MSVC project file (or whatever) with CMake, I might consider it. But as it stands now, it just does not work.

EDIT2: It seems to miss also Xaudio support on windows, which does not need the large fluidsynth libraries and still can play multiple sounds at once. Also lots of references to extended in there. Standard does not use git (it is a read only mirror of the SVN). Amended those, but as said I could not test it, since all three cmake (the MSYS2, the CMake gui and the one in MSVC failed to process this patch).

Mariculous

For reference as An-dz requested a patch instead of a PR, there is the patch related to this PR.
https://patch-diff.githubusercontent.com/raw/aburch/simutrans/pull/15.patch

I am not a CMake expert nor do I build simutrans on windows but I might have a look later on.

Roboron

I can't compile using cmake when applying the last patch from prissi.

Without the changes to simdebug.h, compilation will fail:

brueckenbauer.cc:668:2: error: 'DBG_MESSAGE' was not declared in this scope

Compilation will also fail on water_vehicle.cc without the additional includes. So those changes on the patch by Freahk were necessary. I've re-added them on the patch attached, and also removed a last reference to git versioning.

I'm also interested on getting this working for Standard. Let's recap what's missing:

1. :done: ? Strip Extended-related features.
2. Document how to generate MinGW Makefiles from CMake.
3. CMake should be able to locate libraries installed by VCPKG.
4. CMake should be able to locate natively available Xaudio on Windows.

2. How to generate MinGW Makefiles from CMake.

Looking at how Simutrans Extended does it, an additional cmake toolchain configuration file is needed to build on MinGW.

=> https://github.com/jamespetts/simutrans-extended/blob/master/.github/toolchain_mingw.cmake

I've tried it on my mingw linux installation (note that you need to adjust the TARGET_ENVIRONMENT variable to fit your MinGW installation). It worked, until the linking phase, when it failed with undefined references caused by miniupnp (and fluidsynth, but that's not necessary anyway). I don't know if this is reproducible on other mingw installations and in such a case if anyone from Extended know how to solve it. Please confirm this.

Anyway, I tried compiling without those and it worked.

cmake -DCMAKE_TOOLCHAIN_FILE=toolchain_mingw.cmake -DSIMUTRANS_USE_FLUIDSYNTH_MIDI=0 -DSIMUTRANS_USE_UPNP=0 ..
make


I don't have more time now, but I'll leave hints about the other points.

3. CMake should be able to locate libraries installed by VCPKG.

This seems to be what we are looking for:

=> https://vcpkg.readthedocs.io/en/latest/examples/installing-and-using-packages/#cmake

4. CMake should be able to locate natively available Xaudio on Windows.

First it needs to be added to SimutransDependencies.cmake. If adding findpackage(Xaudio) or similar is not enough, we probably have to manually point CMake to the library location (on a new file cmake/FindXaudio.cmake).

prissi

#8
cmake does not work in a plain vanilla installation in Windows 10 for me. I tried open cmake files in MSVC, using cmake on commandline, or using cmake GUI or cmake that comes with msys2. I tried on two computers. The mingw target in cmake is only for crosscompiling it seems.

MSVC studio does not allow any settings for cmake, and it finds the libaries during compilation even when not defined in the project files. And still it cannot use cmake.

EDIT:

I finally get cmake to do something. However, the commands given in the docs are wrong.

On MSYS2

cmake . -B aaa
[...]
System is unknown to cmake, create:
Platform/MINGW32_NT-10.0-19042 to use this system, please post your config file on discourse.cmake.org so it can be added to cmake
[...]

This version number however changes with each update to the system. As it is now, linking fails since it builds a 64 bit SDL2 even if no SDL is there and on MSYS2-i686. So the cmake file seems very broken to me and not detect the system correctly.

MSVC
Cmake just build x64 targets for MSVC. So this is why it seems to fails with my installation, which just have 32 bit libaries.

So the cmake files seems to need much much more work. This seems much more complicated than autoconf, which is agnostic on bit size or if cross compiling or not (it find that out itself as it should).

Also it seems there is no Xcode mac target?

And why does it force clang instead using system default. That would take care of crosscompilation automatically.

I have to admid that at the moment the advantages of cmake seems low to me.

EDIT2:
I think I worded that too hard. At the moment, there is some help needed from a person more experienced with cMake to have some files that work out of the box on more than one architecture or setup without assuming bit size etc.

Roboron

Ok let's go one issue at a time.

Building on MSYS with cmake

First, you may want to uninstall the cmake version that comes with MSYS, as this doesn't work and will be called instead of mingw's cmake.

pacman -R cmake

Second, make sure you have installed cmake for your mingw architecture

pacman -S mingw-w64-{x86_64,i686}-cmake

Now when you call cmake the right cmake binary will be called (you can also call it explictly with /mingw64/bin/cmake if you don't want to uninstall the MSYS one). And you should be able to build simutrans with:

cmake -G "MSYS Makefiles" ..

=> https://cmake.org/cmake/help/latest/generator/MSYS%20Makefiles.html

I've tried it and it works, so please try it.

On the attached patch I've added (forced) static linking when building with mingw (for some reason Extended lack this?). Previously mentioned linking issues were fixed.

prissi


cmake -G "MSYS Makefiles" -B aaa

or so is the correct command it seems to build. Ok, thank it works at least.

But it builds SDL2 and not Xaudio GDI, and builds makeobj and nettol, which is not needed very time. Also how to make debug/release builds, server builds and set all the other flags in the code now?

Also a lot of strange warning flags seems activated, giving many useless errors (like when the format string is not explicitely given).

Also the building seems to fail to generate the resouces. See the detailed setting for release and for this build:

And I still cannot use this cmake with MSVC studio, even in the 2019 latest version. It just offers x64-debug as target, no choice for SDL2 or not, not the other exe.

So there is certainly progress, but still not there yet.

Mariculous

#11
See cmake -L[A][H] for a list of available variables.
The optional A will also list so called "advanced" configuration variables.
The optional H will add a short description of these variables to the output.
To set these variables, use the -D<var>=<value>

for example, to build simutrans headless, call cmake -DSIMUTRANS_BACKEND=none

Roboron

Let's continue with MSYS compilation, I think we are close.

QuoteBut it builds SDL2 and not Xaudio GDI

If SDL2 is found, then it builds with SDL2, yes. However, you can specify gdi backend with -DSIMUTRANS_BACKEND=gdi option. For usingh Xaudio2 with GDI, the attached patch add support for that.

Quotebuilds makeobj and nettol, which is not needed very time

I've found that if you only want to compile one target you can do so running cmake instead of make like this:

cmake --build . --target simutrans

QuoteAlso how to make debug/release builds, server builds and set all the other flags in the code now?

The type of build can also be configured when calling cmake. To sum up, using MSYS to make a Release build with GDI backend and only compiling the simutrans executable:

cmake -G "MSYS Makefiles" -DCMAKE_BUILD_TYPE=Release -DSIMUTRANS_BACKEND=gdi ..
cmake --build . --target simutrans


Similarly other backends (sdl2, sdl2mixer) and builds (Release, Debug) can be given.

QuoteAlso the building seems to fail to generate the resouces.

What do you mean with resouces¿

Mariculous

cmake --build . --target simutrans
You can, and I'd recommend this as it will (or at least should) work with any supported build system but you don't have to.
You can simply use
make simutrans
instead.

prissi

Thanks. The resources is a file simres.rc containing copyright, version, and icons, and the manifest, which states high resolution aware

Roboron

Ah, yes. It was never included in the source list. Now it is.

This patch also removes the config/ files as that is not related to cmake. I have also included some minor fixes I needed for cross-compilation.

QuoteYou can, and I'd recommend this as it will (or at least should) work with any supported build system but you don't have to.

Ah, that's great to know. Thank you.

Roboron

Quote from: prissi on April 23, 2021, 01:39:59 AMAnd I still cannot use this cmake with MSVC studio, even in the 2019 latest version. It just offers x64-debug as target

It seems that you need to specify the target architecture when generating the MSVC project, cmake can't generate both at the same time. Quoting ceeac on the Extended thread:

Quotecmake.exe .. -G "Visual Studio 16 2019" -A x64 -DCMAKE_TOOLCHAIN_FILE=./vcpkg/scripts/buildsystems/vcpkg.cmake

In this call you need to replace "-A x64" with "-A Win32" to generate a 32 bit MSVC project. I've tried and succedeed.

Does that work for you? If so, I think that's all. I can work next on documenting all of this on the wiki (and maybe later updating the README).

Attached patch has no new changes, I only re-added some files that were not added on the previous one for some reason.

prissi

Ok, MSVC delibrately hid the options. I could load the cmake file, and compile simutrans then after some clicking in some cmake options.. But I cannot debug it, since there is no window to enter the debug options (like start in D:/Simjtrans with -use_workdir ..." ALso I have no idea what binary (GDI or SDL2) was actually built. Somehow cmake let me feel like a Mac user, (or living in Japan): Everything great until you want something non-standard and then everything falls apart ...



Roboron

Quote from: prissi on May 11, 2021, 12:52:18 PMBut I cannot debug it, since there is no window to enter the debug options (like start in D:/Simjtrans with -use_workdir ...

Have you considered opening a shell in the executable directory and call it from there? There may be a way to do it directly from MSVC, but I don't know barely anything about MSVC.

Quote from: prissi on May 11, 2021, 12:52:18 PMALso I have no idea what binary (GDI or SDL2) was actually built.

Well, you should know it since you called cmake, and you can call it with -DSIMUTRANS_BACKEND=gdi to specify the backend. If you didn't specify any backend, SDL2 will be choosed if SDL2 is found, otherwise GDI.

I also noticed you didn't even need to open MSVC to compile. Calling cmake with the previously mentioned cmake command:

cmake --build . --target simutrans --config Release

Will build simutrans on the shell using the MSVC compiler. Sweet!

prissi

For building on the commandline, one does not need cmake. The only advantage would be if cmake can build on the shell and inside MSVC, i.e. that one only has to support one building system instead two (as it is now). For building on the command line, it is hard to beat the autotools, since they run on anything that has bash and a c-compiler ...

(But then maybe cmake versus autotools is like svn versus git, both have their good and bad points, and both can do the job and it is down to personal preference)

Having said that, I still would like to avoid to support two building systems or getting Android running. If cmake can help with that, it would be great.

(Also I a pretty sure there is somewhere an option to set debug commands also when using cmake with MSVC.)

Mariculous

#20
Quote from: prissi on May 11, 2021, 12:52:18 PMsince there is no window to enter the debug options (like start in D:/Simjtrans with -use_workdir ..."
I don't know anything about MSVC, but generally, it's a run/launch option, not a compile option, so I'd expect even in MSVC this has nothing to do with how the program was compiled (Cmake or not) but only to how it is launched, so there should be a MSVC setting hidden somewhere in the depth of the UI.

Edit: I just found this one.
Launch.vs.json and "args": ["-use_workdir"] might be what you are looking for, but again, I have no idea of MSVC.
https://docs.microsoft.com/de-de/cpp/build/configure-cmake-debugging-sessions?view=msvc-160#reference-keys-in-cmakesettingsjson

Roboron

Quote from: prissi on May 11, 2021, 02:09:08 PMHaving said that, I still would like to avoid to support two building systems or getting Android running. If cmake can help with that, it would be great.

According to the documentation about using C++ code with Android, yes, CMake will be needed to build an Android version of Simutrans.

QuoteAndroid Studio's default build tool to compile native libraries is CMake. Android Studio also supports ndk-build due to the large number of existing projects that use the build toolkit. However, if you are creating a new native library, you should use CMake.

=> https://developer.android.com/ndk/guides

Probably some minor changes to the CMakeLists.txt will be needed for Android, but let's focus on having CMake ready for current supported systems before going on that hole.

prissi

Some Progress: During compile it fails for nettool. It seems nettool ignores the x86 setting and is compiled as 64bit app by default?!?

But we are getting closer. The libraries are not static linked (which should be the default, at least for windows and likely for MacOS too) And because there is not dynamic version of zstd installed on my system starting simutrans fails (but linking works, looks like a bug in vcpkg to still have the linking stub there). Still static libaries should be the preferred way. (That way a release only contains the needed code instead everything and the binary works on any computer regardsless of the OS version.)

Also the message level is 0 by default instead 3. I try some things for force it to three when not given, but not much success with it.

Byt the way: Has cMake building tested for MacOS?

Mariculous

Quote from: prissi on May 12, 2021, 01:39:29 AMByt the way: Has cMake building tested for MacOS?
Extended builds without any issues on MacOS using Cmake.
At least that's a feedback I got from the pak192.comic community.

prissi

#24
It does not work with MSVC. Nettool never finds zstd.h while all the other are built.

Moreover, after and error or a warning, the program is not rebuilt. delete the out folder and start with cache generation again. Most annoying.

Debugging with a different startup directory is very difficult. One has to click in the clock in the projet view window the solution explorer icon select cmake solutions, selectct simutrans and then there one can edit a json with startup options as below
https://stackoverflow.com/questions/41864259/how-to-set-working-directory-for-visual-studio-2017-rc-cmake-project

However, it did not work for me. Is there no one using MSVC with cMake?

EDIT:
I could finally generate a project file using the commandline which compiled and linked simutrans (but not nettool, that is definitely broken, does not find zlib.h!" I looked into the cMakefile but I am not sure how to fix this.)

Furthermore, the cmake generation fails when not specifying the vcpgk.cmake full path, despite being in the MSVC developer prompt (where it otherwise compiles normally without spoecify anything.). And this is needed each time I test a SDL2 ot normal build and so on. Honestly, this is very uncorfortable as developer and wastes a lot of time and resets all customisation each time as well. (Not to mention that on each of my developer computers the github location is different due to different user names and primary partition size.)

I found a lot of wrong space/tabs in the files, and some other stuff like static linking. Also the endianness support is gone, which might be required for some architectures on Linux.

EDIT2:
So until the inbuilt cMake support from MSVC is fixed, the MSVC proj needed to be still supported. Unfortunately this removes any gain of switching to cmake, since it would require redoing all nightly builds etc. Also I can fix Makefiles (and thus autoconf) but I fail a cMake (yet).

That all would change if MSVC cmake support become useable for development and not only for single use compiling. Or there is an android port which requires cmake, and thus a third build system to support. So this is considered, I would say, but the gain is not very high compared to the troubles it brings short term for changing all build systems, deployment etc.

ceeac

Quote from: prissi on May 13, 2021, 01:18:52 AM(but not nettool, that is definitely broken, does not find zlib.h!"
The error was also present in trunk, but masked by the fact that zlib is required for the main program. Should be fixed now (r9772).

prissi

Thanks. I will try with cMake later this week, hopefully.

Roboron

Quote from: prissi on May 13, 2021, 01:18:52 AMFurthermore, the cmake generation fails when not specifying the vcpgk.cmake full path

It says in the vcpkg documentation you have to set it in MSVC:

=> https://github.com/Microsoft/vcpkg#vcpkg-with-visual-studio-cmake-projects

After that it also says we can set a default vcpkg directory it in the CMakeLists.txt, so I did (it is set to the build directory). So if you bootstrap vcpkg in the build directory (as recommended), you won't need to call CMake with -DCMAKE_TOOLCHAIN_FILE.

I have also try to improve the static libs. I've found that calling CMake with -DVCPKG_TARGET_TRIPLET=x64-windows-static linked against the static libraries built by vcpkg, so we can set this to usefor static libs by default.

if (WIN32)
# Windows uses vcpkg (installed to the build dir) to link against static libraries by default
set(CMAKE_TOOLCHAIN_FILE ${CMAKE_BINARY_DIR}/vcpkg/scripts/buildsystems/vcpkg.cmake CACHE STRING "Vcpkg toolchain file")
if(NOT ("${CMAKE_GENERATOR_PLATFORM}" STREQUAL "x64")) # 32 bit Windows
set(VCPKG_TARGET_TRIPLET x86-windows-static CACHE STRING "Vcpkg target libraries")
else( NOT ("${CMAKE_GENERATOR_PLATFORM}" STREQUAL "x86") ) # 64 bit Windows
set(VCPKG_TARGET_TRIPLET x64-windows-static CACHE STRING "Vcpkg target libraries")
endif()
endif ()


For linking dynamically one would need now to call cmake with -DVCPKG_TARGET_TRIPLET=x64-windows instead (that is, the situation is the reverse now, but I guess that's ok since static linking is preferred). Maybe doing this an option would be better for the user, feel free to improve.

Quote from: prissi on May 13, 2021, 01:18:52 AMAlso the endianness support is gone, which might be required for some architectures on Linux.

There's a test for endianness that set the SIM_BIG_ENDIAN flag. Isn't that what you are refering to?

if (SIMUTRANS_BIG_ENDIAN)
target_compile_definitions(simutrans PRIVATE SIM_BIG_ENDIAN=1)
endif ()


Attached patch include the vcpkg static libs code and some files (nettool and makeobj CMakeList.txt) that were not in your previous patch for some reason.

prissi

#28
Thanks, I will test it.

Can you give also a list of options, please. Since there is no template any more, there is absolutely no way to find out what flags on may want to set and what not. cmake -LAH never works for me, and is of no use if used with MSVC, since it cannot be accessed there.

EDIT:
At least miniunpnc does not link with this, seems to be still linking to the DLL. And it does not find zstd, even though it is installed the same way then freetype etc.

EDIT 2:
I cannot use cmake even any more with MSYS2. I had to remove the whole if (WIN32) part

EDIT 3:
freetype does not link libbrotli and thus I cannot build MSYS2. Seems like a problem with freetype?

EDIT 4:
I think it does not make sense to set the vcpkg triplet. I have anyway to edit json files to build and debug on MSVC, so specifying this is no further issue. It can be even set graphically inside MSVC. (However, the standard user will have no idea that this is there.)

EDIT 5:
I could finally build and debug something with MSVC, so there may be a light at the horizon. It requires editing hidden json file and clicking on obscure images never needed otherwise, but it is possible to use the cmake file now in MSVC. I have to test this on my other computer, to seem how reproducible this is.
The find miniu... needs at least this to build.

if (MSVC)
    set_property(TARGET MiniUPNP::MiniUPNP PROPERTY INTERFACE_LINK_LIBRARIES "iphlpapi.lib")
endif (MSVC)
endif (MiniUPNP_FOUND)


Mariculous

#29
Quote from: Freahk on April 23, 2021, 12:21:31 PMSee cmake -L[A][H] for a list of available variables.
The optional A will also list so called "advanced" configuration variables.
The optional H will add a short description of these variables to the output.
To set these variables, use the -D<var>=<value>

for example, to build simutrans headless, call cmake -DSIMUTRANS_BACKEND=none

This should work, I'll post the output here.
However, how does it fail? What's the reported error?
Usually you want to run CMake in a subdirectory as I did below.
If the error message is something like
QuoteCMake Error at CMakeLists.txt:10 (message):
  Building Simutrans in-source is not supported.  Please delete
  /path/to/simutrans/source/CMakeCache.txt and configure in a
  different (preferrably empty) directory.
then you should delete /path/to/simutrans/source/CMakeCache.txt, create a build subdirectory, for example build-standard and run Cmake -LAH .. from that directory.

I didn't run te latest state of this, so a few variables might have changed in the meantime..

The short list, which is roughly the default.config equivalent.
QuoteDome@linux-9uud:~/projects/simutrans-extended/build-standard> cmake -L ..
-- Could NOT find CCache (missing: CCache_EXECUTABLE)
-- Configuring Simutrans-Extended (commit 4e94f3f on standard-cmake-support) ...
-- Configuring done
-- Generating done
-- Build files have been written to: /home/Dome/projects/simutrans-extended/build-standard
-- Cache values
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
CMAKE_INSTALL_PREFIX:PATH=/usr/local
SIMUTRANS_BACKEND:STRING=sdl2
SIMUTRANS_BUILD_32BIT:BOOL=OFF
SIMUTRANS_DEBUG_SIMRAND:BOOL=OFF
SIMUTRANS_ENABLE_IPV6:BOOL=ON
SIMUTRANS_ENABLE_PROFILING:BOOL=OFF
SIMUTRANS_ENABLE_RANDOMNESS:BOOL=ON
SIMUTRANS_MSG_LEVEL:STRING=0
SIMUTRANS_MULTI_THREAD:BOOL=ON
SIMUTRANS_USE_FLUIDSYNTH_MIDI:BOOL=ON
SIMUTRANS_USE_FREETYPE:BOOL=OFF
SIMUTRANS_USE_SYSLOG:BOOL=OFF
SIMUTRANS_USE_UPNP:BOOL=ON
SIMUTRANS_USE_ZSTD:BOOL=ON
SIMUTRANS_VALGRIND_SUPPORT:BOOL=OFF
SIMUTRANS_WITH_REVISION:BOOL=ON


And here is the full list with all advanced variables
QuoteDome@linux-9uud:~/projects/simutrans-extended/build-standard> cmake -LAH ..
-- Could NOT find CCache (missing: CCache_EXECUTABLE)
-- Configuring Simutrans-Extended (commit 4e94f3f on standard-cmake-support) ...
-- Configuring done
-- Generating done
-- Build files have been written to: /home/Dome/projects/simutrans-extended/build-standard
-- Cache values
// Path to a file.
BZIP2_INCLUDE_DIR:PATH=/usr/include

// Path to a library.
BZIP2_LIBRARY_DEBUG:FILEPATH=BZIP2_LIBRARY_DEBUG-NOTFOUND

// Path to a library.
BZIP2_LIBRARY_RELEASE:FILEPATH=/usr/lib64/libbz2.so

// Path to a program.
CCache_EXECUTABLE:FILEPATH=CCache_EXECUTABLE-NOTFOUND

// Path to a program.
CMAKE_ADDR2LINE:FILEPATH=/usr/bin/addr2line

// Path to a program.
CMAKE_AR:FILEPATH=/usr/bin/ar

// Build type. Valid values are Debug Release MinSizeRel RelWithDebInfo
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo

// Enable/Disable color output during build.
CMAKE_COLOR_MAKEFILE:BOOL=ON

// CXX compiler
CMAKE_CXX_COMPILER:FILEPATH=/usr/bin/c++

// A wrapper around 'ar' adding the appropriate '--plugin' option for the GCC compiler
CMAKE_CXX_COMPILER_AR:FILEPATH=/usr/bin/gcc-ar-7

// A wrapper around 'ranlib' adding the appropriate '--plugin' option for the GCC compiler
CMAKE_CXX_COMPILER_RANLIB:FILEPATH=/usr/bin/gcc-ranlib-7

// Flags used by the CXX compiler during all build types.
CMAKE_CXX_FLAGS:STRING=

// Flags used by the CXX compiler during DEBUG builds.
CMAKE_CXX_FLAGS_DEBUG:STRING=-g

// Flags used by the CXX compiler during MINSIZEREL builds.
CMAKE_CXX_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG

// Flags used by the CXX compiler during RELEASE builds.
CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG

// Flags used by the CXX compiler during RELWITHDEBINFO builds.
CMAKE_CXX_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG

// C compiler
CMAKE_C_COMPILER:FILEPATH=/usr/bin/cc

// A wrapper around 'ar' adding the appropriate '--plugin' option for the GCC compiler
CMAKE_C_COMPILER_AR:FILEPATH=/usr/bin/gcc-ar-7

// A wrapper around 'ranlib' adding the appropriate '--plugin' option for the GCC compiler
CMAKE_C_COMPILER_RANLIB:FILEPATH=/usr/bin/gcc-ranlib-7

// Flags used by the C compiler during all build types.
CMAKE_C_FLAGS:STRING=

// Flags used by the C compiler during DEBUG builds.
CMAKE_C_FLAGS_DEBUG:STRING=-g

// Flags used by the C compiler during MINSIZEREL builds.
CMAKE_C_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG

// Flags used by the C compiler during RELEASE builds.
CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG

// Flags used by the C compiler during RELWITHDEBINFO builds.
CMAKE_C_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG

// Path to a program.
CMAKE_DLLTOOL:FILEPATH=CMAKE_DLLTOOL-NOTFOUND

// Flags used by the linker during all build types.
CMAKE_EXE_LINKER_FLAGS:STRING=

// Flags used by the linker during DEBUG builds.
CMAKE_EXE_LINKER_FLAGS_DEBUG:STRING=

// Flags used by the linker during MINSIZEREL builds.
CMAKE_EXE_LINKER_FLAGS_MINSIZEREL:STRING=

// Flags used by the linker during RELEASE builds.
CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING=

// Flags used by the linker during RELWITHDEBINFO builds.
CMAKE_EXE_LINKER_FLAGS_RELWITHDEBINFO:STRING=

// Enable/Disable output of compile commands during generation.
CMAKE_EXPORT_COMPILE_COMMANDS:BOOL=

// Install path prefix, prepended onto install directories.
CMAKE_INSTALL_PREFIX:PATH=/usr/local

// Path to a program.
CMAKE_LINKER:FILEPATH=/usr/bin/ld

// Path to a program.
CMAKE_MAKE_PROGRAM:FILEPATH=/usr/bin/gmake

// Flags used by the linker during the creation of modules during all build types.
CMAKE_MODULE_LINKER_FLAGS:STRING=

// Flags used by the linker during the creation of modules during DEBUG builds.
CMAKE_MODULE_LINKER_FLAGS_DEBUG:STRING=

// Flags used by the linker during the creation of modules during MINSIZEREL builds.
CMAKE_MODULE_LINKER_FLAGS_MINSIZEREL:STRING=

// Flags used by the linker during the creation of modules during RELEASE builds.
CMAKE_MODULE_LINKER_FLAGS_RELEASE:STRING=

// Flags used by the linker during the creation of modules during RELWITHDEBINFO builds.
CMAKE_MODULE_LINKER_FLAGS_RELWITHDEBINFO:STRING=

// Path to a program.
CMAKE_NM:FILEPATH=/usr/bin/nm

// Path to a program.
CMAKE_OBJCOPY:FILEPATH=/usr/bin/objcopy

// Path to a program.
CMAKE_OBJDUMP:FILEPATH=/usr/bin/objdump

// Path to a program.
CMAKE_RANLIB:FILEPATH=/usr/bin/ranlib

// Path to a program.
CMAKE_READELF:FILEPATH=/usr/bin/readelf

// Flags used by the linker during the creation of shared libraries during all build types.
CMAKE_SHARED_LINKER_FLAGS:STRING=

// Flags used by the linker during the creation of shared libraries during DEBUG builds.
CMAKE_SHARED_LINKER_FLAGS_DEBUG:STRING=

// Flags used by the linker during the creation of shared libraries during MINSIZEREL builds.
CMAKE_SHARED_LINKER_FLAGS_MINSIZEREL:STRING=

// Flags used by the linker during the creation of shared libraries during RELEASE builds.
CMAKE_SHARED_LINKER_FLAGS_RELEASE:STRING=

// Flags used by the linker during the creation of shared libraries during RELWITHDEBINFO builds.
CMAKE_SHARED_LINKER_FLAGS_RELWITHDEBINFO:STRING=

// If set, runtime paths are not added when installing shared libraries, but are added when building.
CMAKE_SKIP_INSTALL_RPATH:BOOL=NO

// If set, runtime paths are not added when using shared libraries.
CMAKE_SKIP_RPATH:BOOL=NO

// Flags used by the linker during the creation of static libraries during all build types.
CMAKE_STATIC_LINKER_FLAGS:STRING=

// Flags used by the linker during the creation of static libraries during DEBUG builds.
CMAKE_STATIC_LINKER_FLAGS_DEBUG:STRING=

// Flags used by the linker during the creation of static libraries during MINSIZEREL builds.
CMAKE_STATIC_LINKER_FLAGS_MINSIZEREL:STRING=

// Flags used by the linker during the creation of static libraries during RELEASE builds.
CMAKE_STATIC_LINKER_FLAGS_RELEASE:STRING=

// Flags used by the linker during the creation of static libraries during RELWITHDEBINFO builds.
CMAKE_STATIC_LINKER_FLAGS_RELWITHDEBINFO:STRING=

// Path to a program.
CMAKE_STRIP:FILEPATH=/usr/bin/strip

// If this value is on, makefiles will be generated without the .SILENT directive, and all commands will be echoed to the console during the make.  This is useful for debugging only. With Visual Studio IDE projects all commands are done without /nologo.
CMAKE_VERBOSE_MAKEFILE:BOOL=FALSE

// Enable to build RPM source packages
CPACK_SOURCE_RPM:BOOL=OFF

// Enable to build TBZ2 source packages
CPACK_SOURCE_TBZ2:BOOL=ON

// Enable to build TGZ source packages
CPACK_SOURCE_TGZ:BOOL=ON

// Enable to build TXZ source packages
CPACK_SOURCE_TXZ:BOOL=ON

// Enable to build TZ source packages
CPACK_SOURCE_TZ:BOOL=ON

// Enable to build ZIP source packages
CPACK_SOURCE_ZIP:BOOL=OFF

// Path to a file.
FREETYPE_INCLUDE_DIR_freetype2:PATH=/usr/include/freetype2

// Path to a file.
FREETYPE_INCLUDE_DIR_ft2build:PATH=/usr/include/freetype2

// Path to a library.
FREETYPE_LIBRARY_DEBUG:FILEPATH=FREETYPE_LIBRARY_DEBUG-NOTFOUND

// Path to a library.
FREETYPE_LIBRARY_RELEASE:FILEPATH=/usr/lib64/libfreetype.so

// Path to a file.
FluidSynth_INCLUDE_DIR:PATH=/usr/include

// Path to a library.
FluidSynth_LIBRARY:FILEPATH=/usr/lib64/libfluidsynth.so

// Git command line client
GIT_EXECUTABLE:FILEPATH=/usr/bin/git

// Path to a file.
MiniUPNP_INCLUDE_DIR:PATH=/usr/include/miniupnpc

// Path to a library.
MiniUPNP_LIBRARY:FILEPATH=/usr/lib64/libminiupnpc.so

// Path to a library.
MiniUPNP_STATIC_LIBRARY:FILEPATH=MiniUPNP_STATIC_LIBRARY-NOTFOUND

// Path to a library.
PNG_LIBRARY_DEBUG:FILEPATH=PNG_LIBRARY_DEBUG-NOTFOUND

// Path to a library.
PNG_LIBRARY_RELEASE:FILEPATH=/usr/lib64/libpng.so

// Path to a file.
PNG_PNG_INCLUDE_DIR:PATH=/usr/include

// Where the SDL2 headers can be found
SDL2_INCLUDE_DIR:PATH=/usr/include/SDL2

// Where the SDL2 Library can be found
SDL2_LIBRARY:FILEPATH=/usr/lib64/libSDL2.so

// Where the SDL2_mixer headers can be found
SDL2_MIXER_INCLUDE_DIR:PATH=/usr/include/SDL2

// Where the SDL2_mixer Library can be found
SDL2_MIXER_LIBRARY:FILEPATH=/usr/lib64/libSDL2_mixer.so

// Disable search SDL2_mixer Library in default path
SDL2_MIXER_NO_DEFAULT_PATH:BOOL=OFF

// Custom SDL2_mixer Library path
SDL2_MIXER_PATH:STRING=

// Disable search SDL2 Library in default path
SDL2_NO_DEFAULT_PATH:BOOL=OFF

// Custom SDL2 Library path
SDL2_PATH:STRING=

// Graphics backend
SIMUTRANS_BACKEND:STRING=sdl2

// Build 32 or 64 bit executable
SIMUTRANS_BUILD_32BIT:BOOL=OFF

// Debug the random number generator
SIMUTRANS_DEBUG_SIMRAND:BOOL=OFF

// Enable connections using IPv6 in addition to IPv4
SIMUTRANS_ENABLE_IPV6:BOOL=ON

// Enable profiling code
SIMUTRANS_ENABLE_PROFILING:BOOL=OFF

// Disable to make get_random_seed() always return the same number
SIMUTRANS_ENABLE_RANDOMNESS:BOOL=ON

// Message verbosity level
SIMUTRANS_MSG_LEVEL:STRING=0

// Use multiple threads for drawing
SIMUTRANS_MULTI_THREAD:BOOL=ON

// Enable FluidSynth for MIDI playback
SIMUTRANS_USE_FLUIDSYNTH_MIDI:BOOL=ON

// Enable TrueType font support using freetype library
SIMUTRANS_USE_FREETYPE:BOOL=OFF

// Enable logging to syslog
SIMUTRANS_USE_SYSLOG:BOOL=OFF

// Use MiniUPNP for easier server setup
SIMUTRANS_USE_UPNP:BOOL=ON

// Enable support for zstd save file compression (larger save files than bzip2, but faster)
SIMUTRANS_USE_ZSTD:BOOL=ON

// Add support for valgrind "memcheck" tool
SIMUTRANS_VALGRIND_SUPPORT:BOOL=OFF

// Build executable with git commit information
SIMUTRANS_WITH_REVISION:BOOL=ON

// Path to a file.
ZLIB_INCLUDE_DIR:PATH=/usr/include

// Path to a library.
ZLIB_LIBRARY_DEBUG:FILEPATH=ZLIB_LIBRARY_DEBUG-NOTFOUND

// Path to a library.
ZLIB_LIBRARY_RELEASE:FILEPATH=/usr/lib64/libz.so

// Path to a file.
ZSTD_INCLUDE_DIR:PATH=/usr/include

// Path to a library.
ZSTD_LIBRARY:FILEPATH=/usr/lib64/libzstd.so

prissi

#30
mingw just cannot built a working executable. After removing msys64 totally and reinstalling, I was able to build, but it failed at linking; essentially all stuff to harfbuzz (from freetype) was missing. (The latter is anywhere a 9 MB unneccessary monster.) MVSC requires editing the .json configuration files to add the correct triplet for static linking.

Compared to a config file, this listing of parameters is of limited use. For instance, there are no valid parameters given for backend or value range for debug parameters (or are there other command line switches). Also the optimised profiling build option seems to have vanished.

One can probably end up with a single build system in the end; however, the windows build process (which was hailed a the biggest avantage of cmake) is not so straightforward. And debugging the cmake Makefiles to find out why harfbuzz is not included, is not very easy. Especially comparing to "./configure make" on almost any other system.

EDIT:
No success with msys2 though. But here are the file which would build at least with a correct revision.

Roboron

#31
Quote from: prissi on June 01, 2021, 03:19:42 AMI cannot use cmake even any more with MSYS2. I had to remove the whole if (WIN32) part

I don't have such a problem but we can also replace WIN32 with MSVC (this checks for the compiler, not the OS).

Quote from: prissi on June 01, 2021, 12:21:00 PMI was able to build, but it failed at linking; essentially all stuff to harfbuzz (from freetype) was missing.

I see. When adding the necessary linking flags for MinGW, I didn't check for every combination of libraries. I was able to build without problems because another library (fluidsynth) was linking to harfbuzz, but it indeed fail if I try to compile with FreeType only. This should fix it; try again.

Quoteif (SIMUTRANS_USE_FREETYPE)
   target_compile_definitions(simutrans PRIVATE USE_FREETYPE=1)
   target_include_directories(simutrans PRIVATE ${FREETYPE_INCLUDE_DIRS})
   target_link_libraries(simutrans PRIVATE ${FREETYPE_LIBRARIES})
   if (MINGW)
       # MinGW needs additional link flags
       target_link_libraries(simutrans PRIVATE  -lpng16 -lharfbuzz -lrpcrt4 -lgraphite2 -lglib-2.0 -lbrotlidec -lbrotlicommon -lfreetype)
   endif (MINGW)
endif (SIMUTRANS_USE_FREETYPE)

EDIT. Using MSVC to check for the compiler does not work on my system. But searching for Visual Studio in CMAKE_GENERATOR seems to do the trick

if (${CMAKE_GENERATOR} MATCHES "Visual Studio*") ...

prissi

I have a serious problem with cmake. While it should be platform-independent, yet I have to tell with which libraries to link SDL2 or freetype? This may work for one year or two, and then break again.

This is not good for something that long around as simutrans. Why not using pkg-config? The installed library should know best with what options it was compiled! Instead some SDL2 module from some time ago was used for SDL2. I highly doubt this will work when compiling for arm ...

Also strange why the msys build freetype would ask for libharfbuzz, since I built (and linked) mine without for the classic make. Maybe it tries to make a 64 build on a 32 bit shell? Unfortunately I failed to see the actual commandline passed to the compiler to clear this up.

Anyway, cmake would be very broken, if it only support cmake. In the internet I found "PkgConfig" which according to the logfiles produces the right includes/libs/etc. But then it says some error has been produced and no makefile is generated. But the log file has no error.

if (SIMUTRANS_USE_FREETYPE)
find_package(PkgConfig REQUIRED)
pkg_check_modules(Freetype REQUIRED freetype2)
target_compile_options(simutrans PUBLIC ${FREETYPE_CFLAGS_OTHER})
target_include_directories(simutrans PUBLIC ${FREETYPE_INCLUDE_DIRS})
target_link_libraries(simutrans PRIVATE ${FREETYPE_LIBRARIES})
target_compile_definitions(simutrans PRIVATE USE_FREETYPE=1)
endif (SIMUTRANS_USE_FREETYPE)

In the internet I found

Roboron

This worked for me:

if (SIMUTRANS_USE_FREETYPE)
target_compile_definitions(simutrans PRIVATE USE_FREETYPE=1)
target_include_directories(simutrans PRIVATE ${FREETYPE_INCLUDE_DIRS})
if(MINGW)
find_package(PkgConfig REQUIRED)
pkg_check_modules(FREETYPE REQUIRED freetype2)
target_link_libraries(simutrans PRIVATE ${FREETYPE_STATIC_LDFLAGS} )
else ()
target_link_libraries(simutrans PRIVATE ${FREETYPE_LIBRARIES})
endif (MINGW)
endif (SIMUTRANS_USE_FREETYPE)


It is indeed an improvement to be able to use pkg-config inside cmake. Note that this still needs some improvements (we should check for PkgConfig (and the modules) in SimutransDependencies.cmake, and should be applied to other libraries as well. But test it before going forward :-)

prissi

#34
How does the neccessary freetype libaries are found in linux (or whatever your build system uses?) without a configure script?

AFTERTHOUGHT:
Also there is a long findsdl2 script from somewhere (origin unknown) under a different li\cense than simutrans artistic license.. And yet we have to give all libaries by hand for linking Mingw. This is broken by design.

I the net I read that cmake should be great for windows. If you are not using windows, do not bother, use autotools. Well, I am using windows and spent a weel trying to get a perfectly well working environment to compile und windows. What is the point. It seems less work to have a handcrafted makefile for each architekture than using the convoluted cmake syntax, which just hides the fact that there is a script for each architekture.

I am really fed up. For me cmake generates tons of problems and it solves exactly nothing. Unless there comes an architecture which requires cmake, there is no point for me trying to get this broken piece of **** engineering "cmake" to work. Sorry for the harsh words. I will take a break here, this is too frustrating. I have so little time in the next weeks, I will rather spend them constructively than trying project maintenance.

Roboron

Quote from: prissi on June 02, 2021, 11:45:54 AMHow does the neccessary freetype libaries are found in linux (or whatever your build system uses?) without a configure script?

The FindLibrary.cmake files does this. They are called by find_package in SimutransDependencies.cmake. Most of those files (Miniupnpc, ZSTD...) were created by ceeac in the original contribution, some others (like Freetype)  are imported from cmake defaults (it should be in /usr/share/cmake).

But the problem here is not that cmake doesn't find, Freetype or others (it does), but that it doesn't set the linker flag to static.

prissi

#36
It does not set any libaries, also for linking SDL2 the libaries are set by hand.

EDIT:
I had a look at the cmakelist.txt of other projects, and usually they do not include these many specific module. and specific linker flags. I wonder if we did something wrong.

Roboron

#37
Quote from: prissi on June 02, 2021, 12:03:59 PM
It does not set any libaries

Yes, i does, look at the comment at the top of each file:

Quote# This module defines
#  MiniUPNP_FOUND, if miniupnp library and headers have been found
#  MiniUPNP_LIBRARY, the miniupnp variant
#  MiniUPNP_INCLUDE_DIR, where to find miniupnpc.h and family)
#  MiniUPNP_VERSION, the API version of MiniUPNP

Moreover, looking at MiniUPNP specifically, it seems that it also set the MiniUPNP_STATIC_LIBRARIES variable (but it is usually not done, and definitely no on the default FindFreetype.cmake file, I have taken a look before).

find_library(MiniUPNP_STATIC_LIBRARY libminiupnpc.a
HINTS $ENV{MINIUPNP_STATIC_LIBRARY}
)
set(MiniUPNP_STATIC_LIBRARIES ${MiniUPNP_STATIC_LIBRARY})


But just looking at it I'm not sure this is enough to link statically, I'll have to test.

Quote from: prissi on June 02, 2021, 12:03:59 PMalso for linking SDL2 the libaries are set by hand.

Yes, that needs to change.

Quote from: prissi on June 02, 2021, 12:03:59 PMI had a look at the cmakelist.txt of other projects, and usually they do not include these many specific module. and specific linker flags. I wonder if we did something wrong.

I wonder if they use mingw static linking. Statically linking with MSVC + vcpkg didn't require extra flags, and maybe most projects use that workflow nowadays.

prissi

Other projects seem to use just (as we do too)

if(MINGW)
# Let executables run outside MinGW environment
# Force searching static libs
set(CMAKE_FIND_LIBRARY_SUFFIXES ".a" ON)

# Force static linking
link_libraries(-static -static-libgcc -static-libstdc++)
endif()


No matter if static or dynamic linking, the libaries must be defined at link time, even DLLs or the linking fails. VCPKG has cmakelist for all libaries, so the configure of the dependencies is delegated to them.

Roboron

#39
I did some investigation, looking at the top projects that use CMake. Examining the CMake files of OpenRCT2 I've found a pretty similar scheme to what I previously proposed. I'll show some relevant lines I found on OpenRCT2/src/openrct2/CMakeLists.txt (only showing libpng for simplicity)

# Third party libraries
if (MSVC)
    find_package(png 1.6 REQUIRED)
else ()
    PKG_CHECK_MODULES(PNG IMPORTED_TARGET libpng>=1.6)
endif ()

if (STATIC)
    target_link_libraries(${PROJECT_NAME} ${PNG_STATIC_LIBRARIES})
else ()
    if (NOT MSVC)
        target_link_libraries(${PROJECT_NAME} PkgConfig::PNG)
    else ()
        target_link_libraries(${PROJECT_NAME} ${PNG_LIBRARIES})
    endif ()
endif ()


So the process is pretty much like that:

1. Are we building for MSVC -> find_package (what we currentli use)
2. If not -> PKG_CHECK_MODULES (in other words, pkg-config)
3. If STATIC is set, they also use pkg-config defined variables like we did for Freetype

We don't have the need to do #2 as everything is working fine with find_package on every system, but the solution of static linking seems to be #3, at least according to this project.

Did you find any other example?

prissi

After more or less removing everything unnecessary until I got to build all variants under Msys2 this is the state. Tested with GDI, freetype, ZSTD, SDL2 and fluidsynth. There was also a lot of wrong and unneeded option cause for example the visual buffer update display.

MSVC is still no cooperative in actually debugging, it is fixed on the compile path. At least mingw builds fine, also static. The question is, if it builds for you and on Mac ...

Roboron

QuoteCMake Error at CMakeLists.txt:20 (include):
  include could not find requested file:

    simutrans-revision

1. Fixed and renamed to SimutransRevision.cmake for consistency.

2. include(SimutransCompileOptions) was imported AFTER some options were already evaluated (like SIMUTRANS_USE_FREETYPE), so it was compiling without those options defined. I moved it to the top (next to backend selection) in attached patch.

3. Those last lines lines were wrong. First because Freetype_FOUND (as used later) would not be the same as FREETYPE_FOUND, and also it was duplicated probably because you copy-pasted ;-)
if (MSVC)
find_package(Freetype)
find_package(FluidSynth)
else ()
pkg_check_modules(FREETYPE IMPORTED_TARGET freetype2)
pkg_check_modules(FREETYPE IMPORTED_TARGET fluidsynth)
endif ()


For the same reason this needed to change (the same for FluidSynth):

if (SIMUTRANS_USE_FREETYPE)
  target_include_directories(simutrans PRIVATE ${FREETYPE_INCLUDE_DIRS})
  if (MINGW)
    target_link_libraries(simutrans PRIVATE ${FREETYPE_STATIC_LIBRARIES})
  else ()
    target_link_libraries(simutrans PRIVATE ${FREETYPE_LIBRARIES})
  endif (MINGW)
  target_compile_definitions(simutrans PRIVATE USE_FREETYPE=1)
endif (SIMUTRANS_USE_FREETYPE)


After I changed those two things (included in the patch)  it successfully compiled and linked in my main machine (Linux x64).

4. Linking statically SDL2 on MinGW failed. The same solution applied to Freetype and FluidSynth was necessary to build. Added.

5. There's a problem when linking statically the fluidsynth library, but it is not ours. The static library doesn't include the static dependencies, and therefore pkg-config only returns -lfluidsynth when searching for them. I'll report it to the devs, but until them, we will need to list the libraries ourselves just like it was before.

6. Please keep the optional CCache. It improves the time of (re)compilation by a lot. Instead o waiting for minutes it only takes a few seconds to recompile. I re-added it.

7. I've created a new file (SimutransSourceList.cmake) containing the list of sources, that is included in CMakeLists.txt. I think this improves the readability of the main file without 300 lines dedicated to it.

I think the debug options still need some polishing. Particularly the debug level is unset by default, which causes warning while building for Release. But I'll leave it for today, I made enough progress.

prissi

Thank you for your work, I will test it. In the past static linking for SDL2 produced huge non-working binaries (same for pthread) so I think I did not realise that it was not statically linked since there was a SDL2.dll in the folder anyway.

I think after some more work on the cmake files, it may indeed become a working alternative. At least I feel comfortable enough to do something with cmake files. I still think their system is quite broken, and mostly either geared towards building everything from source or making libaries and the main use of programs is rather an afterthough. Also I still have to get it work (for debugging) on MSVC.

I am not sure whether it need to rebuilt everything after each checkout to get the revision right. That is ok for the nightly but for programming it would be a nuissance. Also the revision is built with a script on MSVC inside the project files, so it is not there yet.

Roboron

Now trying to compile on MinGW (x64) on MSYS, it failed at the end:

C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/simutrans.dir/objects.a(simrandom.cc.obj): in function `log2(unsigned int)':
C:/msys64/home/Rober/trunk/utils/simrandom.cc:310: multiple definition of `log2(unsigned int)'; CMakeFiles/simutrans.dir/objects.a(schedule.cc.obj):C:/msys64/home/Rober/trunk/utils/int_math.h:30: first defined here
collect2.exe: error: ld returned 1 exit status


It is true that there is a duplicated function, see:

https://github.com/aburch/simutrans/blob/master/utils/int_math.h#L29-L42
https://github.com/aburch/simutrans/blob/master/utils/simrandom.cc#L309-L325

But this error did not show up before, so I wonder if I did something wrong. EDIT: This error only happened when building for Debug, but not Release. (???)

Also about your last commit (probably related to this) "change order of defines to keep gcc happy" https://github.com/aburch/simutrans/commit/fe55c7ab1e373e267bbc0f0ba42c13600fb2e084
GCC is still not happy, it warnings that _XOPEN_SOURCE is not defined when evaluated. Have you considered this instead?

#if !defined(__APPLE__)  &&  ( !defined(_XOPEN_SOURCE) || _XOPEN_SOURCE < 600 )

Quote from: prissi on June 03, 2021, 11:26:58 PMI am not sure whether it need to rebuilt everything after each checkout to get the revision right.

I'm pretty sure one would need to re-generate the build files, since the revision is obtained and cached during this process. I've made some investigation, and yes, there are ways to get the revision also when building, although they are not so simple. I'll leave this for reference:

=> https://stackoverflow.com/questions/3780667/use-cmake-to-get-build-time-subversion-revision

Wether or not is worth to include such things may depend on how you think CMkae should be used: if as a replacement for the current build system or only as an alternative. There's still work to do for the first possibility, so we can left this for later in any case.

Quote from: prissi on June 03, 2021, 11:26:58 PMAlso the revision is built with a script on MSVC inside the project files, so it is not there yet.

Do you mean the current project files? There's no problem setting the revision with cmake on MSVC, as long as svn is found (for me it worked after attached patch) and there's no script inside the generated project files that I see, so I don't see why we need such a script.

Added in this patch only 3 small changes:

* SIMUTRANS_WC_REVISION was defined but never used: in other words, the REVISION compile optionwas not set, it disappeared at some opint. Re-added.
* Now setting DEBUG to 0 for Release and MinSizeRel builds
* Also required minimum fluidsynth version 2.1.0 - to prevent older linux distributions from compiling without SDL2 audio driver support.

prissi

I have a working configuration for MSVC project files, direct builds and Mingw builds. I will submit it into the SVN, so we can continue from there. This required a lot of changes beyond a few lines (especially making sure of the VCPKG triplets etc).

There is still the annoying console window popping up for releases and the revision number is not contained in the window title in MSVC builds.

Andarix

Quote from: prissi on June 05, 2021, 02:44:14 PM
...
There is still the annoying console window popping up for releases and the revision number is not contained in the window title in MSVC builds.

for this used revision.h

command line script windows


git svn log --oneline --limit=1 > status.txt

set /p string=<status.txt

echo #define REVISION %string:~1,5% > revision.h



not work on git action (git version not work command 'git svn log', to old version)

Roboron

#46
Thanks, it will be much easier to apply, review and create patchs now.

Quote from: prissi on June 05, 2021, 02:44:14 PMThere is still the annoying console window popping up for releases

This seems to work (building for release = no console, else console). You can change it and make an option instead if you want.

#
# target & sources
#
if (CMAKE_BUILD_TYPE STREQUAL "Release" OR CMAKE_BUILD_TYPE STREQUAL "MinSizeRel")
add_executable(simutrans WIN32 MACOSX_BUNDLE)
else ()
add_executable(simutrans)
endif ()
include(SimutransSourceList)


Notice one can add MACOSX_BUNDLE because (I read on the documentation) those variables are only taken in account depending on the OS we are building on, so it has no effect on WIN32. But I have yet to test if it gerenerates the bundle (probably some configuration is needed). I'll work on it later, remove it if you want.

Quote from: prissi on June 05, 2021, 02:44:14 PMthe revision number is not contained in the window title in MSVC builds.

Removing the if case for Visual Studio here worked for me:

string(FIND ${CMAKE_GENERATOR} "Visual Studio" VS )
if (${VS} EQUAL 0)
# using a script for MSVC project files
file(WRITE ${CMAKE_SOURCE_DIR}/revision.h "#define REVISION \n")
add_custom_command(TARGET simutrans PRE_BUILD WORKING_DIRECTORY ${CMAKE_SOURCE_DIR} COMMAND cscript.exe /Nologo revision.jse COMMENT "Find SVN revision")

else ()
# using custom target svnhear that is always built to update the revision.h file using cmake script
add_custom_target(svnheader ALL DEPENDS svn_header)
add_custom_command(OUTPUT svn_header ${CMAKE_SOURCE_DIR}/revision.h COMMAND ${CMAKE_COMMAND} -DSOURCE_DIR=${CMAKE_SOURCE_DIR} -P ${CMAKE_MODULE_PATH}/SimutransRevision.cmake)
set_source_files_properties(${CMAKE_SOURCE_DIR}/revisiom.h PROPERTIES GENERATED TRUE HEADER_FILE_ONLY TRUE)
add_dependencies(simutrans svnheader)
target_compile_definitions(simutrans PRIVATE REVISION_FROM_FILE)

endif ()


In other words, the else() code seems to be valid also for MSVC builds. Why try to force a different method? Is it not working for you? (One thing I noticed is that it is generating the revision.h three times, or so is telling me the MinGW console when building... but I don't know why).

I added those changes in this patch, and included the last two from my last message.

prissi

#47
I tried this (adding WIN32) and it did not work or got not added. It is really hard to debug. I finally had to modify the source to change the subsystem.

The generator for MSVC is actually "Ninja *", "Visual Studio *" generates classical project files. The cannot assum cmake, hence the old script code.

However, generating the MSVC project files is the worst thing for a clicky windows hero:

mkdir out
cd out
cmake .. -A Win32 -DCMAKE_TOOLCHAIN_FILE=C:\Users\xxx\Documents\GitHub\vcpkg\scripts\buildsystems\vcpkg.cmake

even though I removed the need for triplet. Type one letter wrong in the CMAKE part and it does not generate an error message but just does not generate a makefile. This is absolutely bad design, so no user without reading half the internet will get this ri\ght. Furthermore, if you use the wrong MSVC command line like x64 or Win32 on the respective other, it will also fail. Same if you misspell the triplet (like x64-window-static) it will generate a file that builds but not links with cryptic error messages.

Also missing a way to force fresh generation without deleting the entire out folder each time would be nice. Deleting the cache alone generate spurious errors.

EDIT:
your revision code on git just returns nightly instead a proper revision number. Cange the git code to work also with github.

Also fixed the double build of the revision.

Roboron

#48
1. When defining a toolchain file, but not the vcpkg one (in my case, for cross-compilation), it failed because vcpkg triplet can't be found.

Quotecmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=~/toolchain-mingw32.cmake ..
CMake Error at cmake/SimutransVcpkgTriplet.cmake:44 (message):
  Please specify VCPKG triplet!
Call Stack (most recent call first):
  CMakeLists.txt:18 (include)


-- Configuring incomplete, errors occurred!

Fixed by including SimutransVcpkgTriplet only when on MSVC.


if (MSVC)
include(SimutransVcpkgTriplet)
endif ()


2. Since you removed the FindZSTD.cmake, it is necessary to search it with pkgconf when not on MSVC, added.

Quote from: prissi on June 06, 2021, 12:38:37 AMI tried this (adding WIN32) and it did not work or got not added. It is really hard to debug. I finally had to modify the source to change the subsystem.

3. Your condition was not working, but even if it were, it would throw an error because add_executable was called twice (hence the change to SimutransSourceList). Use message warnings to debug and check if the conditions are working. You can also output anything to cache. The generated CMakeCache.txt usually has all the info you need to debug (for example that's where I found fluidsynth wasn't exporting the static libraries). Just like your previous one, this condition always fail for me (not on MSVC):

if ($<CONFIG:Release> OR $<CONFIG:MinSizeRel>)
message(WARNING "building release")
add_executable(simutrans WIN32 MACOSX_BUNDLE)
else ()
message(WARNING "NOT building release")
add_executable(simutrans)
endif ()


Quotecmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=~/toolchain-mingw32.cmake ..
CMake Warning at CMakeLists.txt:92 (message):
  NOT building release

If this condition is needed for MSVC, maybe keep both?

if ( ($<CONFIG:Release> OR $<CONFIG:MinSizeRel>) OR (CMAKE_BUILD_TYPE STREQUAL "Release" OR CMAKE_BUILD_TYPE STREQUAL "MinSizeRel") )

4. The SimutransRevision.cmake is not working properly, it is generating junk

Quote from: revision.h#define REVISION Ruta:

Note that there are a variable called SIMUTRANS_WC_REVISION and another called SIMUTRANS_GIT_WC_REVISION, and both are used, I think the problem is there. I tried to rename them in my last patch, and that was working. Please re-check!

QuoteThe generator for MSVC is actually "Ninja *", "Visual Studio *" generates classical project files. The cannot assum cmake, hence the old script code.

Oh, that explain things...

QuoteThis is absolutely bad design, so no user without reading half the internet will get this ri\ght

I think it's just a matter of documenting the process definedly and clearly. I don't expect a user will try to build Simutrans without previously checking the build instructions, and that is were all those details will need to be explained step by step.

Quote from: prissi on June 06, 2021, 12:38:37 AMAlso missing a way to force fresh generation without deleting the entire out folder each time would be nice. Deleting the cache alone generate spurious errors.

The CMakeCache.txt alone isn't enough, you also need to remove CMakeFiles directory.

prissi

Quote from: Roboron on June 06, 2021, 11:07:15 AM
If this condition is needed for MSVC, maybe keep both?
According to documentation, STREQUAL Release does not compare true for RELEASE while the other construct does.

Quote
4. The SimutransRevision.cmake is not working properly, it is generating junk
It works fine on the aburch from github and the svn. Your code returned had "git.exe git ..which did not work for sure, and after removing the double git always retuned  "Nightly" instead of a revision umber. Because this is the only tag in aburch. What is the actual printout from "git log -1 -pretty" from you repo then?

It should look like
commit dd4efdf6a17078fc927619f5e1310747045be92d
Author: Markus Pristovsek <prissi@physik.tu-berlin.de>
Date:   Sun Jun 6 03:16:48 2021 +0000

    Several fixes to cmake builds

    git-svn-id: svn://tron.homeunix.org/simutrans/simutrans/trunk@9867 8aca7d54-2c30-db11-9de9-000461428c89


Quote
I think it's just a matter of documenting the process definedly and clearly. I don't expect a user will try to build Simutrans without previously checking the build instructions, and that is were all those details will need to be explained step by step.
I think windows users better stay with the built MSVC cmkae, ad bad as it is. Still it would need three pages of screenshots.

Quote
The CMakeCache.txt alone isn't enough, you also need to remove CMakeFiles directory.
That why even with MSVC I had a mingw terminal open for "rm -rf out"


Roboron

#50
Quote from: prissi on June 06, 2021, 11:48:51 AMAccording to documentation, STREQUAL Release does not compare true for RELEASE while the other construct does.

Can you reference it? Because the behaviour I'm seeing is the opposite: the STREQUAL comparation does work, while the $<CONFIG:Release> does not, it is never true for Release (just check my previous output).

Quote from: prissi on June 06, 2021, 11:48:51 AMIt works fine on the aburch from github and the svn.

That's not the problem, prissi. I'm not using git, but svn. The command works fine. The problem has to do with the use of variables, see:

if ( NOT SIMUTRANS_WC_REVISION AND Subversion_FOUND )
execute_process(WORKING_DIRECTORY ${SOURCE_DIR} COMMAND ${SVN_EXECUTABLE} svn info RESULT_VARIABLE res_var OUTPUT_VARIABLE SIMUTRANS_GIT_WC_REVISION)
if ( ${res_var} EQUAL 0 )
        message( "svn ok:" ${SIMUTRANS_GIT_WC_REVISION})
string( REGEX REPLACE "\n" " " TEMP1 ${SIMUTRANS_GIT_WC_REVISION})
string( REGEX REPLACE "^.*Revision: " "" TEMP2 ${TEMP1})
string( REGEX REPLACE " .*$" "" SIMUTRANS_WC_REVISION ${TEMP2})
endif()
endif ()
if (SIMUTRANS_WC_REVISION)
# write a file with the SVNVERSION define
file(WRITE revision.h.txt "#define REVISION ${SIMUTRANS_WC_REVISION}\n")
...


1. First if checks for SIMUTRANS_WC_REVISION
2. Output the result to SIMUTRANS_GIT_WC_REVISION. What?
3. Show on console SIMUTRANS_GIT_WC_REVISION. It shows the correct revision, of course. But...
4. Write junk to SIMUTRANS_WC_REVISION ("Ruta" in thsi case).
5. Check again for SIMUTRANS_WC_REVISION (it is defined, but is is not valid)
6. Finally write the invalid content on SIMUTRANS_WC_REVISION to the file...

This is what is failing.

prissi

#51
I was just reusing variables from the copying code. The code f\works fine

execute_process(WORKING_DIRECTORY ${SOURCE_DIR} COMMAND ${SVN_EXECUTABLE} svn info RESULT_VARIABLE res_var OUTPUT_VARIABLE SIMUTRANS_RAW_WC_REVISION)
if ( ${res_var} EQUAL 0 )
message( "svn ok:" ${SIMUTRANS_RAW_WC_REVISION})
string( REGEX REPLACE "\\n" " " TEMP1 ${SIMUTRANS_RAW_WC_REVISION})
string( REGEX REPLACE "^.*Revision: " "" TEMP2 ${TEMP1})
string( REGEX REPLACE " .*$" "" SIMUTRANS_WC_REVISION ${TEMP2})
endif()

Ok, maybe a better name for the first output. Let's call it SIMUTRANS_RAW_WC_REVISION.
Replace all "\n" (overwise regex fails for unknowns reasons) and put results in TEMP1. Remove everything until "...Revision: " and put results in TEMP2. Remove the trailing junk and put result in SIMUTRANS_WC_REVISION .

Maybe you have a different separtor. What is the output of snv:ok? Maybe one need to replace in the first step "[\n\r\t]"?

In my case it is
Working Copy Root Path: /home/Markus/simutrans
URL: svn://servers.simutrans.org/simutrans/trunk
Relative URL: ^/simutrans/trunk
Repository Root: svn://servers.simutrans.org
Repository UUID: 8aca7d54-2c30-db11-9de9-000461428c89
Revision: 9867
Node Kind: directory
Schedule: normal
Last Changed Author: prissi
Last Changed Rev: 9867
Last Changed Date: 2021-06-06 12:16:48 +0900 (Sun, 06 Jun 2021)


-- Compiling Simutrans revision 9867 ...


EDIT:
Or could it be that the output in spain is not revision?

EDIT2:
I tried the "svn info --show-item revision" command instead. Maybe this work for any language. I had to include a second svn in the command, for unknown reasons ...

Roboron

Quote from: prissi on June 06, 2021, 12:51:57 PMOr could it be that the output in spain is not revision?

Yes it is! You nailed it. The new solution is working prefectly, thank you.

There's also a git command that returns the revision number directly (git rev-list --count HEAD), but sadly this can't be used as the revision from git is different from the svn :-(

if (CMAKE_BUILD_TYPE EQUAL "DEBUG")

This doesn't work for me, it always evaluate to false :-/

CMAKE_BUILD_TYPE vs $<CONFIG:>, I think I now understand the difference.

- CMAKE_BUILD_TYPE is set ONLY for single-generators (Unix makefiles). It is NOT set for multi-generators (Visual Studio). So this is not what we should use.
- $<CONFIG:Release> is valid for both, BUT IT CAN NOT BE EVALUATED ON IF STATEMENTS!!!. That's why it wasn't working!

Instead, they are evaluated "on the fly" on target properties. For example, for our case:

add_executable(simutrans)
set_target_properties(simutrans PROPERTIES WIN32_EXECUTABLE $<CONFIG:Release>)


Is what we were looking for! Please test with this.

EDIT: More info about "generator expressions" https://cmake.org/cmake/help/latest/manual/cmake-generator-expressions.7.html#manual:cmake-generator-expressions(7)

TurfIt

Why is the exisiting working revision build system being broken?

===> HOSTCXX dataobj/environment.cc
In file included from dataobj/environment.cc:10:
dataobj/../simversion.h:10:10: fatal error: revision.h: No such file or directory
   10 | #include "revision.h"
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [common.mk:51: /r/build/dataobj/environment.o] Error 1


r9868 simversion.h:
--#if defined(REVISION_FROM_FILE)  &&  !defined(REVISION)
// include external generated revision file
#include "revision.h"
--#endif




prissi

Later actually windres should choke on the same position. Should be fixed in r9873

Roboron

I worked on the Mac support, and it went so well it is definitely an improvement over the current system. This patch contains the following:

* Enable the build on Mac (it was just a matter of manually setting the homebrew lib directory for linking).
* Generate a Mac App Bundle. This is done from cmake itself, it doesn't use the current script.
* Add linked libraries to the bundle (when running install). This is the major improvement over the current system, since this is done "automagically" with the linked libraries, instead of having to manually add them one-by-one like we do now (without option to exclude/include only what you need).
* Minor fixes for compilation warnings.
* Added EXCLUDE_FROM_ALL to makeobj and nettools so they are not compiled unless specified otherwise. This is because, even if you set the target build to simutrans only, they are built anyway when running install.

Note that the function used for including dynamic linked libraries into the Mac bundle (fixup_bundle) may also be used for including dynamic libs on other systems (or so I have read).

install(CODE "
include(BundleUtilities)
fixup_bundle(\"${CMAKE_BINARY_DIR}/simutrans/simutrans.app\" \"\" \"\")
")


But since that was not my target this time I didn't dig further. However, this could be beneficial when packaging the Steam (linux) version, since last time I had to do it manually. I'll probably revisit it later.

To build on Mac just do the usual
cmake ..
sudo cmake --build . --target install


(sudo is necessary to link some homebrew libraries, but not all)

--target install will install simutrans on /usr/local (or /usr/share I don't remember) by default, but it is necessary to link the libraries to the bundle. Maybe changing the default installation directory to the build directory is a good idea, so the user can copy it whenever it wants - Simutrans is not usually installed system-wide anyway.

prissi

Does the bundle works also on ACs without homebrew installed? This was the reason why it was not used in the current build system on github.

Roboron

Quote from: prissi on June 13, 2021, 06:02:55 AM
Does the bundle works also on ACs without homebrew installed? This was the reason why it was not used in the current build system on github.

Yes, that's the point of including the libraries into the bundle on the install phase. I don't have a second machine to test, but I tried uninstalling SDL2 from homebrew, then running the game, and everything worked as expected.

prissi

I submitted something that might work for MacOS and Windows, I hope.

Could you give a a script for MacOS similar to the current github script to prepare for MacOS compilation, so I can check the .github  yml for MacOS

Roboron

I don't have much experience with the github build system, but hopefully this works.

Ended up changing the install directory (for mac only) because it's easier to zip if it is in the build directory.

Roboron

Quote from: GitHub MacOS NightlyIn file included from /Users/runner/work/simutrans/simutrans/dataobj/loadsave.cc:28:
/Users/runner/work/simutrans/simutrans/dataobj/../io/rdwr/zstd_file_rdwr_stream.h:12:10: fatal error: 'zstd.h' file not found
#include <zstd.h>
         ^~~~~~~~
1 error generated.

Maybe it will be fixed if we include zstd directories. I've not had such problem myself, so I can't personally test it.

ceeac

One can also use the PkgCofig imported target for this according to the documentation. I have used this solution in r9890 along with some fixes to compile on Linux/SDL2+Freetype. IMO the apporach with the imported target is much cleaner because one does not need to keep track of multiple variables.

Roboron

That's better.  But there's not imported target (or so I see) for static libraries, so for MinGW we still need to use target_include_directories (at least, for Freetype, others doesn't seem to be needed for whatever reason (?)).

The Mac GitHub action is still failing for SDL2. With this new approach we should remove the elseif(APPLE) conditions.

prissi

#63
The problem is the include path for cmake projects on MacOS: This is <SDL.h> not <SDL2/SDL.h>. Can be fixed with ifdefs though. I am working right now on my github fork to test it, so be a little patient. Also the installing was not working.

I hope from github forward and backward I have not messed up the changes to CMakeList.txt concerning libaries. I am too tired tonight, please have a look.

Roboron

I've seen you have included pak64. I wonder if this really makes sense, now that with you can select a pakset at game start if you have no one.

Other than that, very well done. Finally we can move onto another things :-)

prissi

Does the file download work? I heard the program directory is write protected (like in windows) and hence cannot be modified on run time? But any documentation I found is highly useless, so would you please try installation on the fly. (I.e. building without pak and starting).

Roboron

#66
Yes, Yes it works. I tried it before, that's why I suggested it. I had no problem downloading a pakset at start after building without one.

EDIT: I have tried opening the app bundle downloaded from GitHub, and three things happened

1. It warns about the developer being not trusted. You can still open it by trying to open it a second time.
2. The app bundle closes immediately after opening it. I remember this is something that was brought up before by THLeader https://forum.simutrans.com/index.php/topic,19444.msg192083.html#msg192083
3. If I open it from CLI (opening the executable /simutrans.app/Contents/MacOS/simutrans) you were indeed right, it is not able to download any pakset.

But the pakset problem #3 is probably related to problem #2 - since changing the app bundle to my build directory fix both of them. There's something we are overlooking about Apple MacOS and permissions, but I don't know what it is. And I honestly don't have much motivation to fix it, since this is not a cmake-specific problem, nor it is a problem for me for the Steam release (because I call /simutrans.app/Contents/MacOS/simutrans to avoid it). If anyone wants to continue work on this better to move the discussion to another thread.

Roboron

I've added documentation about compiling with CMake to the wiki. Only thing left is how to do it from MSVC GUI (I don't know) https://simutrans-germany.com/wiki/wiki/en_CompilingSimutrans#Compiling_with_CMake

If you want I can work on a new version of the README including this information.

Also a little patch to add an option to generate a Steam build is attached (currently it is only used for unmapping the F12 key, but it will be useful when implementing Steam-related code in the future).

prissi

By the way, does the nightly build now runs on Mac? The last comment was not clear to me.

Roboron

#69
Quote from: prissi on October 09, 2021, 01:36:03 PMBy the way, does the nightly build now runs on Mac? The last comment was not clear to me.

It runs, but you have to call the executable inside the bundle. Some issue (I think related to permissions) is preventing the bundle from being launched. This is only reproducible when I download the nightly, if I compile the bundle myself, I can open it just right. That's why I think it may be related to permissions, but I don't know enough about Mac to fix it.

EDIT: I start to remember, I think it has something to do with every resource Simutrans needs (that is, texts, music, and so on), needs to be INSIDE the bundle.

EDIT2: At my notes I found something about working directories. When launching the app bundle the working directory is randomnized. That would explain why Simutrans does not open. See https://stackoverflow.com/questions/42600770/chdir-to-the-location-of-the-app-bundle-in-c-macos-application

prissi

It seems there is a signing option for MacOS in cmake. https://cmake.org/cmake/help/latest/cpack_gen/bundle.html#cpack_gen:CPack%20Bundle%20Generator

Do you have a "team id" we could use?

Roboron

#71
I don't. Not even an Apple account.

I tried again downloading a nightly from GitHub and this time I looked at system logs. When opening the app bundle built from GitHub, the following message log appeared:

QuoteSecTranslocateCreateSecureDirectoryForURL: created /private/var/folders/dc/p5l3qqtn5wbb_snq36008jzc0000gn/T/AppTranslocation/C4A1F2B7-DFEB-414E-8616-93B74D55868D/d/simutrans.app

This message didn't appear when opening my own built app bundle. According to that, the theory of the randomized working directory seems to be right. But what is causing this?

Quotexattr simutrans.app

Executing this gives information about the attributes associated with the bundle. It seems that for downloaded app bundles, Apple automatically add the "com.apple.quarantine" attribute to the bundle, which causes the randomized working directory behaviour. I know because:

Quotesudo xattr -r -d com.apple.quarantine simutrans.app

Removes the attribute and makes simutrans work as expected...

So... what are our options? Pick one:
1) Stop distributing Simutrans via downloads for Apple machines and use an "approved distribution method" to avoid quarantine (basically, App Store, ugh). EDIT: Steam does not add the quarantine attribute.
2) Modify Simutrans so it can work even when "quarantined" (I previously said everything may need to be inside the app bundle, this was the reason after all).
3) Do nothing and just inform our users that they will need to disable the quarantine manually (the lazy option, but we don't have many mac users...).

prissi

Here is a patch which could solve this by unsetting the translocation flag is set. I took it from here: https://www.synack.com/blog/untranslocating-apps/ and the code from here https://github.com/objective-see/TaskExplorer

But if fails linking, and getting this right with github is a pain. Maybe you can tell me what CMake needs to linkt his ...

Otherwise, how to put everything into the bundle? Can this be done with Cmake?

Roboron

#73
After fixing some typos and adding the "-framework cocoa" link flag, most undefined references are gone, except one.

There's a undefined reference for "RestartIfTranslocated()" from _main in simsys2.cc

I don't understand why... may be something to be with Objective-C... translocation.m being compiled as a C object instead of CXX (.mm)?

EDIT1: Yeah, I was in the right direction, Adding extern "C" before including the library was necessary. Now linking. Will upload to a github fork later to test if the untranslocation works.

EDIT2: It works! I can see Simutrans opening, then closing, then opening again. The game launches and the quarantine attribute is removed!

prissi

Great. Now we have a working macos version and an Android version for the next release. By the way, when you download the bundle from github, could you also install paks?

Roboron

Yes I can install paks (I tested with pak64). The only issue still present is #1 (the user needs to override security settings to open the application, presumably because the bundle is not signed).

prissi

Ok, since not having a Mac: Is this just a "Do you trust this App" kind of popup like in windows?

Roboron

Kind of, but the user needs to do right click -> Open for the "allow" pop-up to appear, double clicking is not enough.

Double click


Right click -> Open

prissi

That could be maybe partly solved with a certificate. But also on windows, there is the do you trust etc. when downloading the nightly first.

TurfIt

Quote from: prissi on October 11, 2021, 12:24:33 AM
Great. Now we have a working macos version and an Android version for the next release.

No, now "we" have a macos version that cannot be distributed - convey a modified covered work.

278 OSX/translocation.m
@@ -0,0 +1,278 @@
// from https://github.com/objective-see/TaskExplorer.git see https://www.synack.com/blog/untranslocating-apps/
// The original code is under GPL V3, most of the code is taken from Consts.h and Utilities.m

No way to fulfill the terms of The Artistic License 1.0 and GPLv3 at the same time...  and any nightly builds since r10144 are in violation.

prissi

Well, I do not care ... But I remove the text and just put the link to the website with the original code without any license text. (So this is freeware then.) Because we cannot change the license retroactively since not all contributors can agree. (Not everyone is alive, even, as there was a patch from Dirk, I think).

Roboron

Not that I want to start a relicensing effort but note that back in the 2000's Mozilla relicensed Firefox to (L)GPL without all the contributors agreeing:

Quote from: https://blogs.fsfe.org/ciaran/?p=58...it is not necessary to get permission from 100% of the copyright holders. It would suffice if there was permission from the copyright holders of 95% of the source code and no objections from the holders of the other 5%. This, I'm told, is how Mozilla was able to relicense to the GPL in 2003 despite years of community contributions.

Note also that small contributions may not be subject to copyright, but in any case, they are also easier to replace. The big contributors is where the challenge is.

P.D.: And also you could just simply ask for permission to objective-see, since they hold the rights of such code they can grant you permission to use it under AL... Just saying.

prissi

Just five words more on license before I will not say any more on that topic:
- With relicensing at will by ignoring contributors, one can saz xyz has nothing constructive, so let's decide to go closed source ... Who can decide to ignore someone?
- Thus, the GPL3 is hindering progress in open source, because either relicensing (and if it is that easy one can go closed source after it as well, yes!). Or what you learn by looking at it you must forget when no doing GPL3, so no one care if it is open source?!? (I think this goes diametral against the spirit of the open source foundation ...)

And finally for hobby projects there is no "simutrans owner" who could be sued. Maybe it would be illegal, but since there is no juridical person that committed the crime, nor anything or anybody at disadvantage afterwards, I would see it hard to actually have a crime here ...

Roboron

Back to mac, when investigating this issue I found a piece of duplicated code https://github.com/aburch/simutrans/blob/master/simmain.cc#L647-#L667

prissi


Roboron

This patch fix compilation linking for mingw 32-bit builds.

prissi

Thank, you could submit these trivialities yourself.

Roboron

Quote from: prissi on January 05, 2022, 10:57:55 AM
Thank, you could submit these trivialities yourself.

I was thinking the same, but I have not been given access to the SVN as of yet. So if you give me access to the SVN I will stop bothering you with these trivialities :_)

Roboron

It is my intention to complete the install target, which is currently half-baked for windows. This patch:

1. Use the "simutrans" directory for output of the binary in MSVC instead of Debug/Release directories
2. Execute "nsismake" when on MinGW to compile the pakset downloader (this is also available when cross-compiling)
3. Execute "nsismake.exe" with the full install path when on MSVC. Will only work if installed in the default location (Program Files (x86)), but that was the best I was able to do.

I've tested for MSVC in the CLI doing "cmake --build . --config Release --target install", and so far everything seems to work. The text download script is a bit weird, but it does the job. I don't know if everything will work in Visual Studio itself.

In other news: While doing this I uploaded the necessary NSIS plugins to the AUR for my own ease of use when cross-compiling.

prissi

I could not test this

$ patch -p0 -i ../cmake-install.diff
patch unexpectedly ends in middle of line
patch: **** Only garbage was found in the patch input.

I think there is some problem with special characters. So I had to patch by hand. Since force to build the different flavours manually, I fear I will personally continue to use the distribute.sh script and for MSVC build the project files, since CMake support in MSVC is like everything supported by MS from outside: Useable, but just.

MSVC will not build nsis, but this is not really a problem. However, even release builds add an pdb. Well, manually deletion is needed then, no problem.

Roboron

#90
Quote from: prissi on January 12, 2022, 06:27:10 AMSince force to build the different flavours manually

What do you mean with this?

Quote from: prissi on January 12, 2022, 06:27:10 AM
MSVC will not build nsis, but this is not really a problem. However, even release builds add an pdb. Well, manually deletion is needed then, no problem.

I've just tested with Microsoft Visual Studio Community 2019 Versión 16.9.4, and for me nsis is called, the pakset downloader is compiled and no .pdb files are generated when building on Release (they are for Debug). Basically everything works the same than calling CMake from the CLI. What I did so far:

1. Open the trunk directory with Visual Studio. CMake cache generation starts automatically.
2. Generation failed because vcpkg discoverability. Well I found the root cause (the MSVC variable was being used before it being defined). Attached the fix in this patch.
3. Project -> CMake Cache -> Delete Cache and Project -> Generate Cache to re-generate the cache, this time it succeded.
4. Compile -> Install simutrans.

After compiling the Visual Studio output was exactly the same than the one when running with --target install on powershell. So your are probably doing something wrong.

This is my CMake settings.json (it's everything as it was by default, I only changed Debug and x86-Debug for Release and x86-Release)


{
  "configurations": [
    {
      "name": "x64-Release",
      "generator": "Ninja",
      "configurationType": "Release",
      "inheritEnvironments": [ "msvc_x64_x64" ],
      "buildRoot": "${projectDir}\\out\\build\\${name}",
      "installRoot": "${projectDir}\\out\\install\\${name}",
      "cmakeCommandArgs": "",
      "buildCommandArgs": "",
      "ctestCommandArgs": ""
    }
  ]
}


EDIT: Sorry, I added an additional patch (cmake-output.diff) to fix linking with ld that broke when I moved the output code.

prissi

#91
Thank you for testing.

I use mingw for the binaries for distribution, not MSVC, and have to compile SDL2 and gdi for 32 and 64 bit. Takes about half an evening on my machine, so I run it as a script, which is way more convenient with mingw.

Also my MSVC added a pdb for release builds. Not sure why.

For MSVC, VCPKG may need to set the correct vcpkg target name manually, because especially for x86 this does not to work sometimes.

Also you vcpgk path seems strange, but if it works for you ...

Also I do not understand this one:

if (CMAKE_GENERATOR MATCHES "Visual Studio.*" OR CMAKE_GENERATOR MATCHES "Ninja" AND WIN32)

Why does it work for commandline? It should fails as well without a triplet, or it could not get the correct libaries. That error message seems very much needed also for cammandline.

Roboron

Quote from: prissi on January 13, 2022, 12:09:02 AMAlso you vcpgk path seems strange, but if it works for you ...

I swear I saw that path recommended in the documentation. But I can't find it now. I might be imagining things... But anyway, setting the vcpkg target when calling CMake for configuration is the usual way to proceed.

Quote from: prissi on January 13, 2022, 12:09:02 AMI run it as a script, which is way more convenient with mingw.

Who says you can't use it in a script? Instead of

autoconf
./configure
make
./distribute.sh


You do

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release -G "MSYS Makefiles" -DSIMUTRANS_BACKEND=gdi ..
cmake --build . --target install


And that's really. If you want to build another executable with another set of options (i.e. sdl2), just call cmake again with such options. I did use MinGW and CMake for the Steam binaries. But since it lacked the nsis step I had to add the pakset installer manually :-(. Hence my motivation to complete this build process.

Now that I consider it "complete", I will work on explaining this information in the README. Maybe also set up a GitHub repository later for it to compile releases instead of using my machine.

QuoteAlso my MSVC added a pdb for release builds. Not sure why.

Well, then you can add back the line that deleted it, I guess. But be careful because it had a missing quote somewhere, so it never worked.

Quote from: prissi on January 13, 2022, 12:09:02 AM
Also I do not understand this one:
Código: [Seleccionar]

if (CMAKE_GENERATOR MATCHES "Visual Studio.*" OR CMAKE_GENERATOR MATCHES "Ninja" AND WIN32)

Why does it work for commandline? It should fails as well without a triplet, or it could not get the correct libaries. That error message seems very much needed also for cammandline.

When calling CMake on the command line you usually have to specify the generator (at least on windows):

cmake.exe .. -G "Visual Studio 16 2019" -A x64.

So this will trigger the condition. Basically that condition is the replacement of the MSVC one, because Visual Studio uses Ninja generator instead (or at least, in my default installation it does). That at least was what I come up with, if you have a better idea...

Roboron

r10509 introduced GNUInstallDirs to CMake for installation on linux systems. To control the installation location set the CMAKE_INSTALL_PREFIX variable (it defaults to /usr/local/share apparently). If you want to use "share/games/simutrans" instead of "share/simutrans" as the data dir (i.e. Debian maintainers) I also added the USE_GAMES_DATADIR option for that, so it can be set relative to the CMAKE_INSTALL_DATADIR (so the .desktop and .svg files are installed in the correct location whatever you choose).

This revision also moved all the CMake installation stuff to cmake/SimutransInstall.cmake, as it was starting to clutter the CMakeLists.txt a little.

R1dO

First of all. Thanks for your work on the CMake support, makes compiling a lot easier on my end.

I noted that the install target put the binary (e.g. simutrans) in the bin folder under the CMAKE_INSTALL_PREFIX path (as is common for linux software).
Unfortunately that means simutrans will error out upon start (message something akin to fonts not found). I believe this has to do with simutrans expecting the binary to be in the same folder as the datadir.

There are 2 solutions for this problem (other than rewriting logic for simutrans datapath dependencies):
- Start the binary with the argument `-set_workdir ~/.local/share/simutrans` (i defined the CMAKE_INSTALL_PREFIX  to be `~/.local`)
- Adapt cmake to install the binary in the datafolder and provide a symlink to that file in the ~/.local/bin folder (again, path based on my INSTALL_PREFIX)

If interested in the second option you can try the attached patch. Please keep in mind that patch files are kinda new territory for me (normally i do pull requests for git), so there might be some patching errors there.

prissi

The directory system will be changed in the near future to allow more search options (include a distribution default one) for the data dir, the installer dir and the user dirs.

Roboron

See https://forum.simutrans.com/index.php/topic,21419.msg199065.html for more information.

Actually, your post made me think that we should probably search for relative path (../share{/games}/simutrans) instead of defining a full path for distribution installs. If you have any more comments about this please use the linked thread.

R1dO

Thank you both for the reactions and pointer to that tread. On first glance it seems like a smart approach.

If I feel i could bring some worthwhile pointers to the discussion i would use that thread.