The International Simutrans Forum

 

Author Topic: CMake build support  (Read 3125 times)

0 Members and 1 Guest are viewing this topic.

Offline ceeac

  • Devotee
  • *
  • Posts: 243
CMake build support
« on: August 05, 2018, 08:50:34 AM »
This patch is a work-in-progress attempt to provide support for building Simutrans with CMake.

Currently this patch only works on Linux, not on Windows or macOS.
What should work:
  • Support for clang and gcc compilers
  • Building Simutrans
  • Building makeobj
  • Building nettool
  • Using CCache to speed up compilation (if available)
  • Automatically selecting a suitable graphics backend based on installed libraries (can be overridden manually)
  • Freetype support
  • MiniUPNP support
  • Automatically pulling revision information from git or svn
  • Profiling information support (-DPROFILE)
There is also a github branch available: https://github.com/ceeac/simutrans/tree/cmake

Instructions for building (with the patch applied):
Code: [Select]
mkdir build
cd build
cmake ..
make

Instructions for building from git:
Code: [Select]
git clone https://github.com/ceeac/simutrans.git simutrans
cd simutrans
git checkout cmake
mkdir build
cd build
cmake ..
make

Please test and provide feedback.  :)

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 5691
  • Languages: EN, NO
Re: CMake build support
« Reply #1 on: August 05, 2018, 11:04:15 AM »
We're having trouble keeping the current Makefile system working on all platforms, and keeping the Makefile and Visual Studio projects synchronized. Yet another build system in parallel is not what we need. If the build system's dependencies are rather small and can generate the Visual Studio project from some cross-platform model, it sounds promising as a complete replacement for what we got.

Personally, I've always had trouble getting cmake to work on mingw.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10572
  • Languages: De,EN,JP
Re: CMake build support
« Reply #2 on: August 05, 2018, 10:57:55 PM »
There is an autoconf working for mingw and linux (and possibly even MacOS, if someone would give feedback). I have no experience with cmake, and I am not sure how easy this is to install. Currently the MSVC is maintained seperately from the rest, and if thing could get streamlines, it would be really nice. But my past experience made me very cautious on portability promises.

I think Dwachs uses clang, and I think no changes are needed to the makefile. I could also use our makefile for the nightly crosscompilation, so it seems quite robust and tolerates even old-stable debian (what my server runs).

Cmake would need to fulfill all this. But then it could be of great help. I may test it given time later in August.

Offline Freahk

  • Devotee
  • *
  • Posts: 1473
  • Languages: DE, EN
Re: CMake build support
« Reply #3 on: April 14, 2021, 10:43:16 AM »
*digging deep*
Cmake made its way into extended.
Is there any chance to get this integrated into standard as well?
Many IDEs nowdays can use CMake as a project file, so does mine. It can generate buildfiles for many build system, including but not limited to makefiles and vxcproj.
« Last Edit: April 14, 2021, 07:57:54 PM by Freahk »

Offline Freahk

  • Devotee
  • *
  • Posts: 1473
  • Languages: DE, EN
Re: CMake build support
« Reply #4 on: April 14, 2021, 07:33:45 PM »
I just backported the CMake support from extended.
See this PR https://github.com/aburch/simutrans/pull/15

Thanks to ceeac for the great job.
« Last Edit: April 14, 2021, 08:00:18 PM by Freahk »

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10572
  • Languages: De,EN,JP
Re: CMake build support
« Reply #5 on: April 15, 2021, 02:44:54 AM »
Somehow Cmake is broken. It cannot generate builds for MSYS2 Makefiles, or I am too stupid for this. It does not even generate a MSVC project, as it fails to find zlib (installed by VCPKG).

If anyone can write clear reproducable instructions on how to generate a mingw Makefile or a MSVC project file (or whatever) with CMake, I might consider it. But as it stands now, it just does not work.

EDIT2: It seems to miss also Xaudio support on windows, which does not need the large fluidsynth libraries and still can play multiple sounds at once. Also lots of references to extended in there. Standard does not use git (it is a read only mirror of the SVN). Amended those, but as said I could not test it, since all three cmake (the MSYS2, the CMake gui and the one in MSVC failed to process this patch).
« Last Edit: April 15, 2021, 05:28:25 AM by prissi »

Offline Freahk

  • Devotee
  • *
  • Posts: 1473
  • Languages: DE, EN
Re: CMake build support
« Reply #6 on: April 15, 2021, 11:29:59 AM »
For reference as An-dz requested a patch instead of a PR, there is the patch related to this PR.
https://patch-diff.githubusercontent.com/raw/aburch/simutrans/pull/15.patch

I am not a CMake expert nor do I build simutrans on windows but I might have a look later on.

Offline Roboron

  • Devotee
  • *
  • Posts: 261
    • Las Galácticas Aventuras de Komoyo Diga
  • Languages: ES, EN
Re: CMake build support
« Reply #7 on: April 15, 2021, 05:08:41 PM »
I can't compile using cmake when applying the last patch from prissi.

Without the changes to simdebug.h, compilation will fail:

Code: [Select]
brueckenbauer.cc:668:2: error: ‘DBG_MESSAGE’ was not declared in this scope
Compilation will also fail on water_vehicle.cc without the additional includes. So those changes on the patch by Freahk were necessary. I've re-added them on the patch attached, and also removed a last reference to git versioning.

I'm also interested on getting this working for Standard. Let's recap what's missing:

1. :done: ? Strip Extended-related features.
2. Document how to generate MinGW Makefiles from CMake.
3. CMake should be able to locate libraries installed by VCPKG.
4. CMake should be able to locate natively available Xaudio on Windows.

2. How to generate MinGW Makefiles from CMake.

Looking at how Simutrans Extended does it, an additional cmake toolchain configuration file is needed to build on MinGW.

=> https://github.com/jamespetts/simutrans-extended/blob/master/.github/toolchain_mingw.cmake

I've tried it on my mingw linux installation (note that you need to adjust the TARGET_ENVIRONMENT variable to fit your MinGW installation). It worked, until the linking phase, when it failed with undefined references caused by miniupnp (and fluidsynth, but that's not necessary anyway). I don't know if this is reproducible on other mingw installations and in such a case if anyone from Extended know how to solve it. Please confirm this.

Anyway, I tried compiling without those and it worked.

Code: [Select]
cmake -DCMAKE_TOOLCHAIN_FILE=toolchain_mingw.cmake -DSIMUTRANS_USE_FLUIDSYNTH_MIDI=0 -DSIMUTRANS_USE_UPNP=0 ..
make

I don't have more time now, but I'll leave hints about the other points.

3. CMake should be able to locate libraries installed by VCPKG.

This seems to be what we are looking for:

=> https://vcpkg.readthedocs.io/en/latest/examples/installing-and-using-packages/#cmake

4. CMake should be able to locate natively available Xaudio on Windows.

First it needs to be added to SimutransDependencies.cmake. If adding findpackage(Xaudio) or similar is not enough, we probably have to manually point CMake to the library location (on a new file cmake/FindXaudio.cmake).

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10572
  • Languages: De,EN,JP
Re: CMake build support
« Reply #8 on: April 16, 2021, 12:31:46 AM »
cmake does not work in a plain vanilla installation in Windows 10 for me. I tried open cmake files in MSVC, using cmake on commandline, or using cmake GUI or cmake that comes with msys2. I tried on two computers. The mingw target in cmake is only for crosscompiling it seems.

MSVC studio does not allow any settings for cmake, and it finds the libaries during compilation even when not defined in the project files. And still it cannot use cmake.

EDIT:

I finally get cmake to do something. However, the commands given in the docs are wrong.

On MSYS2
Code: [Select]
cmake . -B aaa
[...]
System is unknown to cmake, create:
Platform/MINGW32_NT-10.0-19042 to use this system, please post your config file on discourse.cmake.org so it can be added to cmake
[...]
This version number however changes with each update to the system. As it is now, linking fails since it builds a 64 bit SDL2 even if no SDL is there and on MSYS2-i686. So the cmake file seems very broken to me and not detect the system correctly.

MSVC
Cmake just build x64 targets for MSVC. So this is why it seems to fails with my installation, which just have 32 bit libaries.

So the cmake files seems to need much much more work. This seems much more complicated than autoconf, which is agnostic on bit size or if cross compiling or not (it find that out itself as it should).

Also it seems there is no Xcode mac target?

And why does it force clang instead using system default. That would take care of crosscompilation automatically.

I have to admid that at the moment the advantages of cmake seems low to me.

EDIT2:
I think I worded that too hard. At the moment, there is some help needed from a person more experienced with cMake to have some files that work out of the box on more than one architecture or setup without assuming bit size etc.
« Last Edit: April 17, 2021, 07:05:53 AM by prissi »

Offline Roboron

  • Devotee
  • *
  • Posts: 261
    • Las Galácticas Aventuras de Komoyo Diga
  • Languages: ES, EN
Re: CMake build support
« Reply #9 on: April 22, 2021, 11:56:12 PM »
Ok let's go one issue at a time.

Building on MSYS with cmake

First, you may want to uninstall the cmake version that comes with MSYS, as this doesn't work and will be called instead of mingw's cmake.

Code: [Select]
pacman -R cmake
Second, make sure you have installed cmake for your mingw architecture

Code: [Select]
pacman -S mingw-w64-{x86_64,i686}-cmake
Now when you call cmake the right cmake binary will be called (you can also call it explictly with /mingw64/bin/cmake if you don't want to uninstall the MSYS one). And you should be able to build simutrans with:

Code: [Select]
cmake -G "MSYS Makefiles" ..
=> https://cmake.org/cmake/help/latest/generator/MSYS%20Makefiles.html

I've tried it and it works, so please try it.

On the attached patch I've added (forced) static linking when building with mingw (for some reason Extended lack this?). Previously mentioned linking issues were fixed.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10572
  • Languages: De,EN,JP
Re: CMake build support
« Reply #10 on: April 23, 2021, 01:39:59 AM »
Code: [Select]
cmake -G "MSYS Makefiles" -B aaa
or so is the correct command it seems to build. Ok, thank it works at least.

But it builds SDL2 and not Xaudio GDI, and builds makeobj and nettol, which is not needed very time. Also how to make debug/release builds, server builds and set all the other flags in the code now?

Also a lot of strange warning flags seems activated, giving many useless errors (like when the format string is not explicitely given).

Also the building seems to fail to generate the resouces. See the detailed setting for release and for this build:

And I still cannot use this cmake with MSVC studio, even in the 2019 latest version. It just offers x64-debug as target, no choice for SDL2 or not, not the other exe.

So there is certainly progress, but still not there yet.

Offline Freahk

  • Devotee
  • *
  • Posts: 1473
  • Languages: DE, EN
Re: CMake build support
« Reply #11 on: April 23, 2021, 12:21:31 PM »
See cmake -L[A][H] for a list of available variables.
The optional A will also list so called "advanced" configuration variables.
The optional H will add a short description of these variables to the output.
To set these variables, use the -D<var>=<value>

for example, to build simutrans headless, call cmake -DSIMUTRANS_BACKEND=none
« Last Edit: April 23, 2021, 03:44:56 PM by Freahk »

Offline Roboron

  • Devotee
  • *
  • Posts: 261
    • Las Galácticas Aventuras de Komoyo Diga
  • Languages: ES, EN
Re: CMake build support
« Reply #12 on: April 23, 2021, 05:09:33 PM »
Let's continue with MSYS compilation, I think we are close.

Quote
But it builds SDL2 and not Xaudio GDI

If SDL2 is found, then it builds with SDL2, yes. However, you can specify gdi backend with -DSIMUTRANS_BACKEND=gdi option. For usingh Xaudio2 with GDI, the attached patch add support for that.

Quote
builds makeobj and nettol, which is not needed very time

I've found that if you only want to compile one target you can do so running cmake instead of make like this:

Code: [Select]
cmake --build . --target simutrans
Quote
Also how to make debug/release builds, server builds and set all the other flags in the code now?

The type of build can also be configured when calling cmake. To sum up, using MSYS to make a Release build with GDI backend and only compiling the simutrans executable:

Code: [Select]
cmake -G "MSYS Makefiles" -DCMAKE_BUILD_TYPE=Release -DSIMUTRANS_BACKEND=gdi ..
cmake --build . --target simutrans

Similarly other backends (sdl2, sdl2mixer) and builds (Release, Debug) can be given.

Quote
Also the building seems to fail to generate the resouces.

What do you mean with resouces¿

Offline Freahk

  • Devotee
  • *
  • Posts: 1473
  • Languages: DE, EN
Re: CMake build support
« Reply #13 on: April 23, 2021, 05:28:29 PM »
Code: [Select]
cmake --build . --target simutransYou can, and I'd recommend this as it will (or at least should) work with any supported build system but you don't have to.
You can simply use
Code: [Select]
make simutransinstead.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10572
  • Languages: De,EN,JP
Re: CMake build support
« Reply #14 on: April 24, 2021, 12:38:21 PM »
Thanks. The resources is a file simres.rc containing copyright, version, and icons, and the manifest, which states high resolution aware

Offline Roboron

  • Devotee
  • *
  • Posts: 261
    • Las Galácticas Aventuras de Komoyo Diga
  • Languages: ES, EN
Re: CMake build support
« Reply #15 on: April 25, 2021, 12:35:54 AM »
Ah, yes. It was never included in the source list. Now it is.

This patch also removes the config/ files as that is not related to cmake. I have also included some minor fixes I needed for cross-compilation.

Quote
You can, and I'd recommend this as it will (or at least should) work with any supported build system but you don't have to.

Ah, that's great to know. Thank you.

Offline Roboron

  • Devotee
  • *
  • Posts: 261
    • Las Galácticas Aventuras de Komoyo Diga
  • Languages: ES, EN
Re: CMake build support
« Reply #16 on: May 11, 2021, 12:14:16 PM »
And I still cannot use this cmake with MSVC studio, even in the 2019 latest version. It just offers x64-debug as target

It seems that you need to specify the target architecture when generating the MSVC project, cmake can't generate both at the same time. Quoting ceeac on the Extended thread:

Quote
cmake.exe .. -G "Visual Studio 16 2019" -A x64 -DCMAKE_TOOLCHAIN_FILE=./vcpkg/scripts/buildsystems/vcpkg.cmake

In this call you need to replace "-A x64" with "-A Win32" to generate a 32 bit MSVC project. I've tried and succedeed.

Does that work for you? If so, I think that's all. I can work next on documenting all of this on the wiki (and maybe later updating the README).

Attached patch has no new changes, I only re-added some files that were not added on the previous one for some reason.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10572
  • Languages: De,EN,JP
Re: CMake build support
« Reply #17 on: May 11, 2021, 12:52:18 PM »
Ok, MSVC delibrately hid the options. I could load the cmake file, and compile simutrans then after some clicking in some cmake options.. But I cannot debug it, since there is no window to enter the debug options (like start in D:/Simjtrans with -use_workdir ..." ALso I have no idea what binary (GDI or SDL2) was actually built. Somehow cmake let me feel like a Mac user, (or living in Japan): Everything great until you want something non-standard and then everything falls apart ...



Offline Roboron

  • Devotee
  • *
  • Posts: 261
    • Las Galácticas Aventuras de Komoyo Diga
  • Languages: ES, EN
Re: CMake build support
« Reply #18 on: May 11, 2021, 01:50:38 PM »
But I cannot debug it, since there is no window to enter the debug options (like start in D:/Simjtrans with -use_workdir ...

Have you considered opening a shell in the executable directory and call it from there? There may be a way to do it directly from MSVC, but I don't know barely anything about MSVC.

ALso I have no idea what binary (GDI or SDL2) was actually built.

Well, you should know it since you called cmake, and you can call it with -DSIMUTRANS_BACKEND=gdi to specify the backend. If you didn't specify any backend, SDL2 will be choosed if SDL2 is found, otherwise GDI.

I also noticed you didn't even need to open MSVC to compile. Calling cmake with the previously mentioned cmake command:

Code: [Select]
cmake --build . --target simutrans --config Release
Will build simutrans on the shell using the MSVC compiler. Sweet!

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10572
  • Languages: De,EN,JP
Re: CMake build support
« Reply #19 on: May 11, 2021, 02:09:08 PM »
For building on the commandline, one does not need cmake. The only advantage would be if cmake can build on the shell and inside MSVC, i.e. that one only has to support one building system instead two (as it is now). For building on the command line, it is hard to beat the autotools, since they run on anything that has bash and a c-compiler ...

(But then maybe cmake versus autotools is like svn versus git, both have their good and bad points, and both can do the job and it is down to personal preference)

Having said that, I still would like to avoid to support two building systems or getting Android running. If cmake can help with that, it would be great.

(Also I a pretty sure there is somewhere an option to set debug commands also when using cmake with MSVC.)

Offline Freahk

  • Devotee
  • *
  • Posts: 1473
  • Languages: DE, EN
Re: CMake build support
« Reply #20 on: May 11, 2021, 02:15:29 PM »
since there is no window to enter the debug options (like start in D:/Simjtrans with -use_workdir ..."
I don't know anything about MSVC, but generally, it's a run/launch option, not a compile option, so I'd expect even in MSVC this has nothing to do with how the program was compiled (Cmake or not) but only to how it is launched, so there should be a MSVC setting hidden somewhere in the depth of the UI.

Edit: I just found this one.
Launch.vs.json and "args": ["-use_workdir"] might be what you are looking for, but again, I have no idea of MSVC.
https://docs.microsoft.com/de-de/cpp/build/configure-cmake-debugging-sessions?view=msvc-160#reference-keys-in-cmakesettingsjson
« Last Edit: May 11, 2021, 02:30:19 PM by Freahk »

Offline Roboron

  • Devotee
  • *
  • Posts: 261
    • Las Galácticas Aventuras de Komoyo Diga
  • Languages: ES, EN
Re: CMake build support
« Reply #21 on: May 11, 2021, 03:56:31 PM »
Having said that, I still would like to avoid to support two building systems or getting Android running. If cmake can help with that, it would be great.

According to the documentation about using C++ code with Android, yes, CMake will be needed to build an Android version of Simutrans.

Quote
Android Studio's default build tool to compile native libraries is CMake. Android Studio also supports ndk-build due to the large number of existing projects that use the build toolkit. However, if you are creating a new native library, you should use CMake.

=> https://developer.android.com/ndk/guides

Probably some minor changes to the CMakeLists.txt will be needed for Android, but let's focus on having CMake ready for current supported systems before going on that hole.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10572
  • Languages: De,EN,JP
Re: CMake build support
« Reply #22 on: May 12, 2021, 01:39:29 AM »
Some Progress: During compile it fails for nettool. It seems nettool ignores the x86 setting and is compiled as 64bit app by default?!?

But we are getting closer. The libraries are not static linked (which should be the default, at least for windows and likely for MacOS too) And because there is not dynamic version of zstd installed on my system starting simutrans fails (but linking works, looks like a bug in vcpkg to still have the linking stub there). Still static libaries should be the preferred way. (That way a release only contains the needed code instead everything and the binary works on any computer regardsless of the OS version.)

Also the message level is 0 by default instead 3. I try some things for force it to three when not given, but not much success with it.

Byt the way: Has cMake building tested for MacOS?

Offline Freahk

  • Devotee
  • *
  • Posts: 1473
  • Languages: DE, EN
Re: CMake build support
« Reply #23 on: May 12, 2021, 09:34:18 AM »
Byt the way: Has cMake building tested for MacOS?
Extended builds without any issues on MacOS using Cmake.
At least that's a feedback I got from the pak192.comic community.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10572
  • Languages: De,EN,JP
Re: CMake build support
« Reply #24 on: May 13, 2021, 01:18:52 AM »
It does not work with MSVC. Nettool never finds zstd.h while all the other are built.

Moreover, after and error or a warning, the program is not rebuilt. delete the out folder and start with cache generation again. Most annoying.

Debugging with a different startup directory is very difficult. One has to click in the clock in the projet view window the solution explorer icon select cmake solutions, selectct simutrans and then there one can edit a json with startup options as below
https://stackoverflow.com/questions/41864259/how-to-set-working-directory-for-visual-studio-2017-rc-cmake-project

However, it did not work for me. Is there no one using MSVC with cMake?

EDIT:
I could finally generate a project file using the commandline which compiled and linked simutrans (but not nettool, that is definitely broken, does not find zlib.h!" I looked into the cMakefile but I am not sure how to fix this.)

Furthermore, the cmake generation fails when not specifying the vcpgk.cmake full path, despite being in the MSVC developer prompt (where it otherwise compiles normally without spoecify anything.). And this is needed each time I test a SDL2 ot normal build and so on. Honestly, this is very uncorfortable as developer and wastes a lot of time and resets all customisation each time as well. (Not to mention that on each of my developer computers the github location is different due to different user names and primary partition size.)

I found a lot of wrong space/tabs in the files, and some other stuff like static linking. Also the endianness support is gone, which might be required for some architectures on Linux.

EDIT2:
So until the inbuilt cMake support from MSVC is fixed, the MSVC proj needed to be still supported. Unfortunately this removes any gain of switching to cmake, since it would require redoing all nightly builds etc. Also I can fix Makefiles (and thus autoconf) but I fail a cMake (yet).

That all would change if MSVC cmake support become useable for development and not only for single use compiling. Or there is an android port which requires cmake, and thus a third build system to support. So this is considered, I would say, but the gain is not very high compared to the troubles it brings short term for changing all build systems, deployment etc.
« Last Edit: May 13, 2021, 03:19:49 AM by prissi »