Server compilation issues after merging nightly pull requests

[jamespetts]

Having merged the latest nightly build, the automatic builds of the command line server builds have stopped working, failing with the following errors:

Code Select Expand

===> LD  build/server/simutrans-extended
/usr/bin/ld: build/server/io/raw_image_png.o: in function `raw_image_t::read_png_data(_IO_FILE*)':
raw_image_png.cc:(.text+0x33): undefined reference to `png_create_read_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x49): undefined reference to `png_create_info_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x6d): undefined reference to `png_set_longjmp_fn'
/usr/bin/ld: raw_image_png.cc:(.text+0x90): undefined reference to `png_init_io'
/usr/bin/ld: raw_image_png.cc:(.text+0x9f): undefined reference to `png_read_info'
/usr/bin/ld: raw_image_png.cc:(.text+0xcc): undefined reference to `png_get_IHDR'
/usr/bin/ld: raw_image_png.cc:(.text+0x106): undefined reference to `png_destroy_read_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x156): undefined reference to `png_set_strip_16'
/usr/bin/ld: raw_image_png.cc:(.text+0x160): undefined reference to `png_set_packing'
/usr/bin/ld: raw_image_png.cc:(.text+0x18e): undefined reference to `png_read_update_info'
/usr/bin/ld: raw_image_png.cc:(.text+0x19d): undefined reference to `png_get_rowbytes'
/usr/bin/ld: raw_image_png.cc:(.text+0x204): undefined reference to `png_read_image'
/usr/bin/ld: raw_image_png.cc:(.text+0x24b): undefined reference to `png_destroy_read_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x25a): undefined reference to `png_set_expand'
/usr/bin/ld: raw_image_png.cc:(.text+0x26e): undefined reference to `png_get_valid'
/usr/bin/ld: raw_image_png.cc:(.text+0x381): undefined reference to `png_read_end'
/usr/bin/ld: raw_image_png.cc:(.text+0x392): undefined reference to `png_destroy_read_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x3f5): undefined reference to `png_set_filler'
/usr/bin/ld: raw_image_png.cc:(.text+0x431): undefined reference to `png_set_filler'
/usr/bin/ld: raw_image_png.cc:(.text+0x470): undefined reference to `png_set_strip_alpha'
/usr/bin/ld: raw_image_png.cc:(.text+0x4aa): undefined reference to `png_destroy_read_struct'
/usr/bin/ld: build/server/io/raw_image_png.o: in function `raw_image_t::write_png(char const*) const':
raw_image_png.cc:(.text+0x612): undefined reference to `png_create_write_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x628): undefined reference to `png_create_info_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x64c): undefined reference to `png_set_longjmp_fn'
/usr/bin/ld: raw_image_png.cc:(.text+0x66f): undefined reference to `png_init_io'
/usr/bin/ld: raw_image_png.cc:(.text+0x6b0): undefined reference to `png_set_IHDR'
/usr/bin/ld: raw_image_png.cc:(.text+0x6c3): undefined reference to `png_write_info'
/usr/bin/ld: raw_image_png.cc:(.text+0x70b): undefined reference to `png_write_row'
/usr/bin/ld: raw_image_png.cc:(.text+0x71f): undefined reference to `png_write_end'
/usr/bin/ld: raw_image_png.cc:(.text+0x72e): undefined reference to `png_destroy_write_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x787): undefined reference to `png_destroy_write_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x7b7): undefined reference to `png_destroy_write_struct'
/usr/bin/ld: raw_image_png.cc:(.text+0x7d4): undefined reference to `png_destroy_write_struct'
collect2: error: ld returned 1 exit status
make: *** [common.mk:27: build/server/simutrans-extended] Error 1

This appears to be because raw_image_png.cc, one of the new files incorporated in the merge, is wanting access to various PNG functions, which are not set to compile with the command line server build.

Can I check whether anyone has tried compiling the command line server build? Should raw_image_png.cc be excluded from the command line server build?
Also - the mingw64 build is also now failing:

Code Select Expand

env CFG=mingw64 make -j6
expr: syntax error: unexpected argument '1'
Please install dpkg-dev to use pkg-config when cross-building
===> CXX sys/clipboard_s2.cc
x86_64-w64-mingw32-g++ -std=gnu++11 -DPNG_STATIC -DZLIB_STATIC -static -DNOMINMAX -DWIN32_LEAN_AND_MEAN -DWINVER=0x0501 -D_WIN32_IE=0x0500 -DHAS_64_BIT_SYSTEM -O3 -DNDEBUG -DUSE_FREETYPE  -DMULTI_THREAD -DREVISION="05025e6" -Wall -W -Wcast-qual -Wpointer-arith -Wcast-align -D_REENTRANT -DUSE_C -DALT_SDL_DIR -fno-delete-null-pointer-checks -fno-strict-aliasing -std=c++11 -I/root/freetype-2.10.4/include -I/root/SDL2-2.0.7/include -I/usr/share/mxe/mxe/usr/x86_64-unknown-linux-gnu/include/ -I/root/bzip64/bzip2-1.0.6/ -I/root/zstd/zstd/lib -I/usr/include/freetype2 -I/root/miniupnpc/miniupnpc-2.1.20201016 -DUSE_ZSTD -I/usr/include/SDL2 -D_REENTRANT -DCOLOUR_DEPTH=16 -c -MMD -o build/mingw64/sys/clipboard_s2.o sys/clipboard_s2.cc
Please install dpkg-dev to use pkg-config when cross-building
===> CXX bauer/brueckenbauer.cc
sys/clipboard_s2.cc:14:10: fatal error: SDL2/SDL.h: No such file or directory
   14 | #include <SDL2/SDL.h>
      |          ^~~~~~~~~~~~
compilation terminated.
Please install dpkg-dev to use pkg-config when cross-building
make: *** [common.mk:56: build/mingw64/sys/clipboard_s2.o] Error 1
make: *** Waiting for unfinished jobs....
x86_64-w64-mingw32-g++ -std=gnu++11 -DPNG_STATIC -DZLIB_STATIC -static -DNOMINMAX -DWIN32_LEAN_AND_MEAN -DWINVER=0x0501 -D_WIN32_IE=0x0500 -DHAS_64_BIT_SYSTEM -O3 -DNDEBUG -DUSE_FREETYPE  -DMULTI_THREAD -DREVISION="05025e6" -Wall -W -Wcast-qual -Wpointer-arith -Wcast-align -D_REENTRANT -DUSE_C -DALT_SDL_DIR -fno-delete-null-pointer-checks -fno-strict-aliasing -std=c++11 -I/root/freetype-2.10.4/include -I/root/SDL2-2.0.7/include -I/usr/share/mxe/mxe/usr/x86_64-unknown-linux-gnu/include/ -I/root/bzip64/bzip2-1.0.6/ -I/root/zstd/zstd/lib -I/usr/include/freetype2 -I/root/miniupnpc/miniupnpc-2.1.20201016 -DUSE_ZSTD -I/usr/include/SDL2 -D_REENTRANT -DCOLOUR_DEPTH=16 -c -MMD -o build/mingw64/bauer/brueckenbauer.o bauer/brueckenbauer.cc
bauer/brueckenbauer.cc: In static member function 'static bool bridge_builder_t::is_blocked(koord3d, ribi_t::ribi, player_t*, const char*&)':
bauer/brueckenbauer.cc:300:10: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
  300 |     || w && (((w->get_desc()->get_waytype() != road_wt
      |        ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  301 |      && w->get_desc()->get_waytype() != tram_wt
      |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  302 |      && w->get_desc()->get_waytype() != water_wt)
      |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  303 |
      |
  304 |      || (w->get_owner() != player && !public_service)
      |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  305 |      || (w->is_public_right_of_way() && (!w->is_disused() || welt->get_city(gr2->get_pos().get_2d()))))))
      |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bauer/brueckenbauer.cc: In static member function 'static void bridge_builder_t::build_bridge(player_t*, koord3d, koord3d, koord, sint8, const bridge_desc_t*, const way_desc_t*, overtaking_mode_t)':
bauer/brueckenbauer.cc:896:40: warning: enumeral and non-enumeral type in conditional expression [-Wextra]
  896 |    const slope_t::type hang = start_gr ? start_gr->get_weg_hang() :  slope_t::flat;
      |                               ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bauer/brueckenbauer.cc:949:34: warning: enumeral and non-enumeral type in conditional expression [-Wextra]
  949 |    const slope_t::type hang = gr ? gr->get_weg_hang() :  slope_t::flat;
      |                               ~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bauer/brueckenbauer.cc:1037:35: warning: enumeral and non-enumeral type in conditional expression [-Wextra]
1037 |     const slope_t::type hang = gr ? gr->get_weg_hang() :  slope_t::flat;
      |                                ~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bauer/brueckenbauer.cc: In static member function 'static void bridge_builder_t::build_ramp(player_t*, koord3d, ribi_t::ribi, slope_t::type, const bridge_desc_t*, const way_desc_t*, overtaking_mode_t, bool)':
bauer/brueckenbauer.cc:1157:33: warning: enumeral and non-enumeral type in conditional expression [-Wextra]
1157 |   const slope_t::type hang = gr ? gr->get_weg_hang() : slope_t::flat;


I am not sure why this is, and it is not clear what the link is to these changes, but this was working until very recently.

Edit: I have managed to solve the latter problem - but I now get a different compile error with mingw64, this time directly linked to the raw image system:

Code Select Expand

===> CXX io/raw_image_ppm.cc
In file included from io/raw_image_png.cc:9:
/usr/x86_64-w64-mingw32/include/png.h:361:13: fatal error: pnglibconf.h: No such file or directory
  361 | #   include "pnglibconf.h"
      |             ^~~~~~~~~~~~~~
compilation terminated.
Please install dpkg-dev to use pkg-config when cross-building
make: *** [common.mk:56: build/mingw64/io/raw_image_png.o] Error 1
make: *** Waiting for unfinished jobs....
x86_64-w64-mingw32-g++ -std=gnu++11 -DPNG_STATIC -DZLIB_STATIC -static -DNOMINMAX -DWIN32_LEAN_AND_MEAN -DWINVER=0x0501 -D_WIN32_IE=0x0500 -DHAS_64_BIT_SYSTEM -O3 -DNDEBUG -DUSE_FREETYPE  -DMULTI_THREAD -DREVISION="05025e6" -Wall -W -Wcast-qual -Wpointer-arith -Wcast-align -D_REENTRANT -DUSE_C -DALT_SDL_DIR -fno-delete-null-pointer-checks -fno-strict-aliasing -std=c++11 -I/root/freetype-2.10.4/include -I/root/SDL2-2.0.7/include -I/usr/share/mxe/mxe/usr/x86_64-unknown-linux-gnu/include/ -I/root/bzip64/bzip2-1.0.6/ -I/root/zstd/zstd/lib -I/usr/include/freetype2 -I/root/miniupnpc/miniupnpc-2.1.20201016 -DUSE_ZSTD -I/usr/include/SDL2 -D_REENTRANT -DCOLOUR_DEPTH=16 -c -MMD -o build/mingw64/io/raw_image_ppm.o io/raw_image_ppm.cc


[Roboron]

I opened another PR which hopefully fix it https://github.com/jamespetts/simutrans-extended/pull/469

As mentioned in the other thread, libpng is now necessary to build and link the main program, too. But it shouldn't be for the server builds, so I excluded the raw_image files from the server build while adding libpng.lib to the dependencies in other builds.

In the process I discovered that you reverted not properly a previous change related to sound, which was also making the server build fail. Reverted now totally.

I could only test that compilation is successfull in the "Debug (non graphical server)". The other configuration "Release (command line server)" is asking me for a toolchain I don't know how to install.

Additional observation: I noticed that server builds are being compiled with USE_FREETYPE. Is that necessary?

[jamespetts]

Thank you for that. I have now merged your fix and this does indeed fix the server build.

However, the MinGW cross-compile builds still do not work. The trouble is that I had never set up a cross-compile libpng for them, and I had serious trouble in attempting to do so in the past, which is why the makeobj cross compile nightly builds do not work. What I need to do is work out how to cross compile the libpng library binaries, I think.

Edit: I have built what I believe are cross-compile binaries for libpng in a specified directory, but still get the following error when attempting to cross-compile Simutrans-Extended:

Code Select Expand

-o build/mingw64/Simutrans-Extended.exe
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lpng
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lpng
collect2: error: ld returned 1 exit status


I have specified the location of the compiled library files in LDFLAGS with the -I tag.

[Roboron]

The OpenTTD cross-compilation wiki page has a section about building libpng.

Code Select Expand

wget https://sourceforge.net/projects/libpng/files/libpng16/1.6.37/libpng-1.6.37.tar.gz/download -O- | tar xfz -
cd l*png*
./configure \
    --host=x86_64-w64-mingw32 \
    --prefix=/usr/local/x86_64-w64-mingw32 \
    CPPFLAGS="-I/usr/local/x86_64-w64-mingw32/include" \
    LDFLAGS="-L/usr/local/x86_64-w64-mingw32/lib"
make
sudo make install
cd ..

You can try that. I do cross-compile from Arch, and the GitHub CI from Fedora, but I have no experience cross-compiling on Ubuntu which I believe is what you use for cross-compilation.

[Mariculous]

Apart from that, you might consider using the existing nightly builds from github, so you don't have to maintain any additional build environment at all on BB (and you save those CPU cycles)
All you had to do in that case is to download the latest build from github and extract it to your simutrans installation.
Github itself cannot support the nightly updater, but you might extract these on your server in a way so nightly updater can use it.

[jamespetts]

Roboron - thank you: I will have a look at that.

Sirius - can I check whether the Windows 64-bit nightly builds are working fully on Github now? Has anyone tested their performance against the builds from the server?

[jamespetts]

I have now attempted to build libpng - as set out in the OpenTTD guide, but I am getting this error:

Code Select Expand


/usr/bin/x86_64-w64-mingw32-ld: skipping incompatible /usr/local/i686-w64-mingw32/lib/libpng.a when searching for -lpng
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lpng
collect2: error: ld returned 1 exit status
make: *** [common.mk:27: build/mingw64/Simutrans-Extended.exe] Error 1


[Roboron]

You installed the i686-w64-mingw32 version instead fo the x86_64-w64-mingw32 version.

Probably you took the instructions from the wiki instead of my comment where I already updated the path for you (I should have warned you anyway).

[jamespetts]

Thank you for that. I have changed over to the x86_64-w64-mingw32, but unfortunately, I am now just getting the original error again:

Code Select Expand


/usr/bin/x86_64-w64-mingw32-ld: cannot find -lpng
collect2: error: ld returned 1 exit status
make: *** [common.mk:27: build/mingw64/Simutrans-Extended.exe] Error 1


Edit: Ahh, I believe that I have fixed this - error in config file. Re-running nightly builds now.

Edit 2: This seems to have worked - and a happy side effect is that the Windows cross-compile makeobj builds are working again. Thank you for your assistance.

[Mariculous]

Sirius - can I check whether the Windows 64-bit nightly builds are working fully on Github now? Has anyone tested their performance against the builds from the server?

Win 64 bit minGW build are working quite well for a long time. Only MSVC SDL2 builds were broken.
Good news: Both are working now.

About the performance, I didn't benchmark these but the github build is using just the same compile options as any other Release CMake build in default setup.
If you are using any specific optimisations on BB, you might share these so they can find their way into the nightlies on github.

Edit: as for now, the full package doesn't seem to be working. Should now be fixed now. I'll se the results tomorrow (nightly should run in roughly 1 hour automatically)

[Roboron]

Thank you for your assistance.

Not needed, it was my action who caused this problem so it was my responsibility to help fix it.

a happy side effect is that the Windows cross-compile makeobj builds are working again

Another is that you are one step closer to provide cross-compiled builds with FreeType support (since libpng is a dependency). However, as Sirius suggested, the nightly builds are already set up for doing that, so you could save time and effort and provide those instead (or at the very least the MacOS nightly, that the BB server is unable to provide on its own).

@Sirius, but it seems that the first nightly build for jamespetts repository didn't even succeed in the check-updates step. Is it because the tag need to be created first? You should know more https://github.com/jamespetts/simutrans-extended/runs/4848164298?check_suite_focus=true
EDIT: Also your deploy failed, but for other reasons (wrong paths and failed downloads of artifacts).

[Mariculous]

Yes, the script expects the Nightly tag to exist. I should better tolerate a nonexisting nightly tag, simply assuming the nightly not being up-to-date in that case.

The nightly deployment doesn't compile anything on its own.
It will simply deploy binaries of the last successful commit to master ("deploy-bin" step) as well as to package these binaries with the latest version of pak128-britain-ex (the "update-paks"/"deploy-package" step in the action tree)
pak192.comic is on my roadmap btw.

The update-paks/deploy-package logic is closely tied to nightly deployment of the pakset:
update-paks will trigger nightly deployment of the paksets and wait for it to finish, then download the pakset to include it in the package.
The other way round, the pakset will always use makeobj from the latest successful commit to master to build the pak.

EDIT: Also your deploy failed, but for other reasons (wrong paths and failed downloads of artifacts).

Yes, this issue had been introduced by one of the CMake commits. I simply wasn't notified about this before because by default the upload-artifact action silently ignores missing files.
I have fixed a bunch of paths last night and set this to be an error, but obviously missed some paths.
Fixed now.

Preparing another commit to better get this integrated into repos other than simutrans/simutrans-extended with simutrans/simutrans-pak128.britain-ex

[Roboron]

Preparing another commit to better get this integrated into repos other than simutrans/simutrans-extended with simutrans/simutrans-pak128.britain-ex

Doesn't seem to be working :-(

[Mariculous]

Indeed.
Seems like workflow call paths cannot be relative. Migrating to the new reusable workflow feature instead.