Latest nightlies require GLIBC_2.29

[Vladki]

Due to bridgewater being upgraded to ubuntu 20.04, the stuff compiled there cannot be run on any older distro:

./simutrans-extended: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by ./simutrans-extended)
./simutrans-extended: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./simutrans-extended)


This is due to math functions:

$ objdump -T simutrans-extended|grep GLIBC_2.2[^.]
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.29  log
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.29  pow
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.27  expf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.29  log2
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.27  powf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.27  logf

I'm a bit confused, as I thought that all floating point math should be done using the custom float32e8_t type, and thus all fucntions should be implemented by that. So I have looked on the sources.

1. I did not find any use of logf(), expf() or exp()... Only regexp() :-)
2. I found several uses of pow(), log() and log2() that uses arguments of type (double),

Looking at many uses of pow(), they are pow (2, x)  which could be replaced by  1 << x
Also it looks like pow() and log() are often used for choosing colors... And some log() functions are not logarihtms, byt error logging

Anyway, is it desirable to check all occurences of pow(), log() and log2() and try to get rid of them, or at least replace them with pow() and log() operating on float32e8_t ?
I willing to try...

[jamespetts]

The float32e8_t is used only where code needs to be kept in synchronisation between network client and server. In other cases (e.g. the GUI or map generation), it is better to use native floating point code as it is much faster. It is also better not to diverge too much from Standard in these respects.

I am not sure that I understand why a specific version of glibc should be required; is anyone who knows more about the linking issues know able to throw more light on this?

[Vladki]

If I compile locally with GLIBC_2.24, and check with objdump I get: GLIBC_2.2.5 for log*, exp* and pow* functions. Only a few fucntions require 2.3.x and overall maximum is GLIBC_2.14  memcpy.
My guess is that the fucntions por log, exp and pow have been changed in 2.27 and 2.29, and thus if you compile simutrans with GLIBC 2.27 or newer, you cannot run it with older Glibc.
I also thought if it may be drawn in by the new compression libray (zstd ?), but I do not expect compression to use floating point calculations.

BTW what version of ubuntu did you use before the upgrade to 20.04 ?  Ubuntu 18.04 uses glibc 2.27 so this should have affected me even before, but it didn't.
So far I found out that the easiest way is to build on system with older glibc

[jamespetts]

Until the upgrade, I used 16.04, which is why you would not have been affected. For the Stephenson-Seimens server, you might simply find it easier to build your own nightlies rather than downloading the executables from the Bridgewater-Brunel server.

[Vladki]

I can compile myself, but others may not be able to do so.... Current nightlies are too fresh even for Debian stable (glibc 2.28). And regarding stephenson-siemens, it seems that nobody is playing any more. The game is in the future, so no new vehicles are coming (unless added to the pakset), and the world is probably connected as much as possible. So there is not much space for new development. Maybe if I add some more industries ... I think it was fun but it is time to start a new game.

[jamespetts]

I suspect that it probably is time to start a new game. I am afraid that I do not understand enough about the build and linking process to understand how to make a version that will work with older versions of Linux - the idea of removing all the pow(), log(), etc., is not sustainable for the reasons given above.

If anyone is able to assist with this, it would be appreciated.

[Roboron]

When deploying the latest Simutrans 122 to Steam, I also had to face a similar problem. I am using a Debian (stable) VM to compile Simutrans, because my main system is rolling release and that would generate an incompatible build for linux distributions with older libraries - exactly your case with glibc. Unfortunately, Debian stable also doesn't have an up-to-date enough version of zstd. In that moment my only ideas were using a newer distro, or just ditching zstd. In the end, I went with the second option, as I thought that compatibility was more desirable.

However, others (not so simple) solutions have come to my mind since then.

1) Don't update the system but instead try to compile a custom zstd (not sure if that would be posible with Debian's current libraries).
2) Use my current distribution and try to solve compatibility issues by linking statically glibc (and others). This may increase significally binary size, but I have not tested.
3) In your case, I suppose you can't simply swap operating systems on your server. But you could run the build on a docker container, provided you choose a linux distribution with an old enough glibc and a new enough zstd
4) Alternatively to 3, GitHub actions may be a solution to this.

[prissi]

In github actions you can choose the distribution: ubuntu-16.04 is available.

[jamespetts]

In github actions you can choose the distribution: ubuntu-16.04 is available.

Interesting. If anyone is able to assist in setting up Github Actions, this would be helpful.

[prissi]

ceeac had done this, I think:
https://forum.simutrans.com/index.php/topic,20233.0.html

Otherwise you can use the build environment from standard, essentially just everything in the folder .github

[jamespetts]

ceeac had done this, I think:
https://forum.simutrans.com/index.php/topic,20233.0.html

Otherwise you can use the build environment from standard, essentially just everything in the folder .github

I believe that this is currently not working for Extended for reasons that are not entirely clear to me - I get an error message e-mail from Github stating that the build has failed every time that I push a commit.

[freddyhayward]

I believe that this is currently not working for Extended for reasons that are not entirely clear to me - I get an error message e-mail from Github stating that the build has failed every time that I push a commit.

That's because of invalid line endings. Your IDE needs to be configured so that it doesn't produce them.

[prissi]

One could also run ./cleanup_code.sh as quick fix.

[jamespetts]

That's because of invalid line endings. Your IDE needs to be configured so that it doesn't produce them.

May I ask what specific attribute of the line endings makes them invalid for this purpose so that I can look at reconfiguring Visual Studio not to produce them?

[freddyhayward]

May I ask what specific attribute of the line endings makes them invalid for this purpose so that I can look at reconfiguring Visual Studio not to produce them?

I'm not sure of the precise terminology, but here are two pull requests that had previously solved the problem: https://github.com/jamespetts/simutrans-extended/pull/305/files https://github.com/jamespetts/simutrans-extended/pull/307/files

[jamespetts]

jamespetts

Those remove trailing whitespaces, which is not the same as the line ending type - are you sure that these are the same issue?

[Matthew]

I believe that this is currently not working for Extended for reasons that are not entirely clear to me - I get an error message e-mail from Github stating that the build has failed every time that I push a commit.

AAAH! This is an unfortunate misunderstanding. I looked into this a few weeks ago and now wish I had posted what I found. 

The only part of Github Actions that is failing internally is the code check. The actual compilation succeeds when the code check is passed.

This code check is performed by the cleanup_code.sh script, which searches for certain problems including trailing whitespaces and fixes them. On Github, the script exits with an error, which I believe is because it cannot fix the errors. I guess that this is because the Action does not have permission to modify the code, only to run it. So each individual part is doing its job, but Github has been asked to perform irreconciliable goals (protect the code and change the code).

I thought about suggesting removing the cleanup_code.sh, but it does serve a useful purpose. At least one of the contributors (I honestly forget who) has previously submitted PRs with hundreds of random trailing whitespaces (presumably inserted by his editor), so you get a diff which appears to change hundreds of files when the only real change is to one of them. And it's very easy to accidentally add trailing whitespace when you really are editing code.

Another way to solve the problem is for everybody to run `cleanup_code.sh` before submitting (using MinGW Bash for those on Windows), but it seems exactly the sort of automatic process that Github Actions would be perfect for.

ceeac had done this, I think:
https://forum.simutrans.com/index.php/topic,20233.0.html

Otherwise you can use the build environment from standard, essentially just everything in the folder .github

Extended's Github builds will probably fail at the moment because they are set to use `ubuntu-latest`, (= 18.04) and we have suddenly moved to only supporting compilation on 20.04 or later. But our builds already have a notification saying that Github will automatically update `ubuntu-latest` to 20.04; the migration began on 30 November so our respositories will be switched in the next few months without us taking any action. I guess this may also mean that for a while compilation may succeed for some users and not others, since Github have not said whether the migration schedule is on a per-user or per-repository basis.

Also, the setup may need to be changed with all the new dependencies. But they have all come across from Standard and the Standard build works, so in principle ceeac has already done most of the work for us.

[freddyhayward]

Those remove trailing whitespaces, which is not the same as the line ending type - are you sure that these are the same issue?

No I'm not sure, and it's possible that I have confused the terms. In any case, it should be sufficient to run ./cleanup_code before each commit as prissi suggests.

[Vladki]

But we already have nighlties for "ubuntu-latest = 20.04"  The point was to provide binaries for 18.04 or 16.04 if possible.

[Mariculous]

I also have to build simutrans from source on openSUSE Leap 15.2 (which is latest)

[Matthew]

But we already have nighlties for "ubuntu-latest = 20.04"  The point was to provide binaries for 18.04 or 16.04 if possible.

You are right; I got off-topic at some point.