Semi-automated tests

[ceeac]

For the past few months I have been working on semi-automated tests using the scripted scenario functionality. These tests do not cover the entire code base (far from it), but the existing tests should work, although the code is still a bit messy.

Source code and usage instructions are here: https://github.com/ceeac/simutrans-tests

Currently, the tests are only run when the test repository is updated, but I think it is also possible to set up a GitHub action to periodically check the svn and run the tests if there are new changes.

[Dwachs]

Incredible! I also did some test scenario in https://github.com/Dwachs/simutrans-scenarios/tree/master/pak/scenario/test_scripting, but yours seems to be much more complete! Wow!

Please commit that scenario-option patch.

[Mariculous]

Great!
Automated tests are clearly a good thing!
I did some experiements unit testing some of simutrans components using gtest.
It turned out to work well on most *_tpl whilst being extremely difficult on the other components.

[prissi]

Indeed, thank you very much. I second Dwachs, please submit the scenario option.

[Roboron]

Fantastic! Automated testing is one big step to improve Simutrans quality over time. Thank you!

[ceeac]

Please commit that scenario-option patch.

Done in r10104.

BTW, two tests currently are disabled due to engine bugs:
1. When building and removing a (multi-tile?) building on a sloped tile the grid height is not restored correctly
2. When building a powerline over a transformer connected to a factory, the maintenance is not updated correctly.

The bugs can be triggered by uncommenting the functions marked with FIXMEs in all_tests.nut. I have not yet investigated how to fix them, though.

[Yona-TYT]

Your transformer problem reminds me of an error obtaining the consumption values that I found years ago, the truth is I never check if this is solved.

https://forum.simutrans.com/index.php/topic,19272.0.html

[prissi]

The transformer bug has been fixed. The grid bug I fail to understand. The terrain after building removal is restored as it should be. Where is the error?

[ceeac]

The transformer bug has been fixed.

With the fix, ctrl-building over existing power lines increases maintenance, works fine without ctrl.

The terrain after building removal is restored as it should be. Where is the error?

See test_building_build_multi_tile_sloped:
When flattening a hill by building a multi-tile building over the whole hill, the grid height is still 1, but the tile slopes are not restored when the building is removed. Or maybe it makes more sense to reset the grid height to zero?
(The test itself passes, but enabling this test causes a different test to fail because of the bug.)

[prissi]

The hill should not be flattened, but the building should be at the highest point or fail. Investigating.

EDIT:
When manually building and removing a building, the height were always restored correctly, as it should be. I do not find an error. Not sure what the script tool does, maybe this is the problem.

The script does not run for me in german language. "Kohlegrube!=Coal mine", so it must be english. But then (after fixing the transformer bug for good), it proceed without error to the end.

EDIT2:
Finally, I see. This only happens if the entire hill is under the building, and the building is larger than the hill.

EDIT3:
r10018 should properly restore also the grid height (and r10017 the transformer issue).

[ceeac]

Thanks. There was still a small error which I think is fixed in r10019.

[ceeac]

Here is a patch that adds a GitHub actions workflow to run the tests for every nightly build. IMO this can be committed as-is for now, but at some point I want to fail the nightly builds when the tests fail. Thoughts?

[prissi]

I think it is a good idea to see the test. One gets anyway an email, if a test fails.

Why forcing clang 10? The release are built with whaever CC is set. If the test fails only for clang 10 but for the normal release works. SO why use clang and not whatever CC is set?

But you would need also to have the scenarios with that path, or?

[ceeac]

SO why use clang and not whatever CC is set?

Since the normal Ubuntu nightly builds are done using GCC, using clang makes sure that the code can be compiled using both compilers. Also, in my experience clang has slightly better support for the sanitizers (ASan/UBSan).

But you would need also to have the scenarios with that path, or?

My tests repository is cloned separately during the build and a symlink is created in simutrans/pak/scenario/ to use the tests. Sorry, I do not fully understand the question.

[prissi]

I though you wanted to have the tests also as part of the repo, which I am fine with.

[ceeac]

Actually not initially, but now I think having tests and code in the same repository is better, although the separation of code and tests could be improved by moving the source files (cc/h files) to a separate subdirectory. Not going to think about doing this any time soon, though.

Anyway, I have submitted the tests and the GitHub Action to run them in r10171.

[prissi]

Thank you.

[Dwachs]

Some error appears in the tests:

Code Select Expand


00:00:05 Script: Print: Running test 82/130...
00:00:05 Warning: gui_aligned_container_t::set_size: new size (0,30) smaller than min size (200,4)
00:00:05 ERROR: roadsign_t::~roadsign_t(): roadsign_t 0x604000180f90 was deleted but ground has no way of type 2!
00:00:05 For help with this error or to file a bug report please see the Simutrans forum:
00:00:05 https://forum.simutrans.com
00:00:05 Script: Print: Running test 83/130...

Roadsign gets deleted but there is no way under it, which is an error. I could not locate, which test is causing this. Also the test player gets bankrupt in between.

[Andarix]

automatet test load not language files

and it is started 3 times for me

https://github.com/Andarix/simutrans/actions

[Mariculous]

and it is started 3 times for me

This is due to the on: [push]
The workflow itself creates tags, which are considered pushes, so you have to restrict it.
For example, you might use

Code Select Expand


on:
  push:
    tags:
      - 'Nightly'


[ceeac]

Roadsign gets deleted but there is no way under it, which is an error. I could not locate, which test is causing this. Also the test player gets bankrupt in between.

https://forum.simutrans.com/index.php/topic,21103.0.html
Test is test_sign_signal_when_player_removed in test_sign.nut

To improve readability, one could print the name of the test next to its number. However, I could not find a way to convert the value of a squirrel function pointer to a string. :/ Maybe it is possible to do it the other way around and just use strings in all_tests.nut?

[Dwachs]

Maybe function.getinfos() could work ?

http://www.squirrel-lang.org/squirreldoc/reference/language/builtin_functions.html#function

[Andarix]

http://www.squirrel-lang.org/squirreldoc/reference/language/builtin_functions.html?highlight=string#compilestring

http://www.squirrel-lang.org/squirreldoc/search.html?q=toString&check_keywords=yes&area=default

[ceeac]

function.getinfos() works, thanks.

[Andarix]

The path to the installation of the pak64 does not seem to be correct.

....
2021-10-19T05:10:11.4060917Z adding pak64-122-0
2021-10-19T05:10:11.4061472Z
2021-10-19T05:10:11.4062333Z You will install now
2021-10-19T05:10:11.4062978Z 1) pak64-122-0
2021-10-19T05:10:11.4070053Z into directory '/home/runner/work/simutrans/simutrans/simutrans'
2021-10-19T05:10:11.4070830Z
2021-10-19T05:10:11.4071577Z -- Installing pak64-122-0 --
2021-10-19T05:10:11.4072877Z Downloading from 'http://downloads.sourceforge.net/project/simutrans/pak64/122-0/simupak64-122-0.zip'
2021-10-19T05:10:11.5677160Z
2021-10-19T05:10:11.7184596Z ######################################################################## 100.0%
2021-10-19T05:10:11.8550699Z                                                                            0.1%
2021-10-19T05:10:11.9617680Z #                                                                          2.3%
2021-10-19T05:10:12.0855128Z #############                                                             18.5%
2021-10-19T05:10:12.1863415Z ####################################                                      50.2%
2021-10-19T05:10:12.2863758Z ############################################                              62.3%
2021-10-19T05:10:12.3868379Z ######################################################                    75.5%
2021-10-19T05:10:12.4865778Z #############################################################             85.6%
2021-10-19T05:10:12.4871057Z ######################################################################    98.0%
2021-10-19T05:10:12.4871792Z ######################################################################## 100.0%
2021-10-19T05:10:12.6133752Z Extracting '/tmp/simupak64-122-0.zip' to '/home/runner/work/simutrans/simutrans/'...
2021-10-19T05:10:12.7727135Z Installation completed.
2021-10-19T05:10:12.7727491Z
2021-10-19T05:10:12.7736055Z ~/work/simutrans/simutrans
2021-10-19T05:10:12.7783548Z ##[group]Run mkdir -p ./simutrans/pak/scenario
2021-10-19T05:10:12.7784222Z mkdir -p ./simutrans/pak/scenario
2021-10-19T05:10:12.7784875Z ln -sT $GITHUB_WORKSPACE/tests ./simutrans/pak/scenario/automated-tests
2021-10-19T05:10:12.7832097Z shell: /usr/bin/bash -e {0}
2021-10-19T05:10:12.7832573Z ##[endgroup]
2021-10-19T05:10:12.7959798Z ##[group]Run mkdir -p ~/simutrans/
2021-10-19T05:10:12.7960322Z mkdir -p ~/simutrans/
2021-10-19T05:10:12.7960867Z echo "frames_per_second = 100"              >> ~/simutrans/simuconf.tab
2021-10-19T05:10:12.7961574Z echo "fast_forward_frames_per_second = 100" >> ~/simutrans/simuconf.tab
2021-10-19T05:10:12.7996763Z shell: /usr/bin/bash -e {0}
2021-10-19T05:10:12.7997132Z ##[endgroup]
2021-10-19T05:10:12.8111172Z ##[group]Run export ASAN_OPTIONS="print_stacktrace=1 abort_on_error=1 detect_leaks=0"
2021-10-19T05:10:12.8112154Z export ASAN_OPTIONS="print_stacktrace=1 abort_on_error=1 detect_leaks=0"
2021-10-19T05:10:12.8112994Z export UBSAN_OPTIONS="print_stacktrace=1 abort_on_error=1"
2021-10-19T05:10:12.8113750Z cp scripts/run-automated-tests.sh .
2021-10-19T05:10:12.8114491Z chmod +x run-automated-tests.sh
2021-10-19T05:10:12.8115207Z ulimit -St 600 # 10 minutes ought to be enough for anybody.
2021-10-19T05:10:12.8115909Z ./run-automated-tests.sh
2021-10-19T05:10:12.8156760Z shell: /usr/bin/bash -e {0}
2021-10-19T05:10:12.8157187Z ##[endgroup]
2021-10-19T05:10:12.8269389Z ~/work/simutrans/simutrans/simutrans ~/work/simutrans/simutrans
2021-10-19T05:10:12.8579338Z 00:00:00 Simutrans version 122.0.1 Nightly from Oct 19 2021 r10181
2021-10-19T05:10:12.8584773Z 00:00:00 Warning: loadsave_t::rd_open:   File 'settings.xml' does not exist or is not accessible
2021-10-19T05:10:12.8587791Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'pl'
2021-10-19T05:10:12.8590645Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'ro'
2021-10-19T05:10:12.8594197Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'lt'
2021-10-19T05:10:12.8597444Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'de'
2021-10-19T05:10:12.8600249Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'en'
2021-10-19T05:10:12.8603023Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'ja'
2021-10-19T05:10:12.8605830Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'id'
2021-10-19T05:10:12.8608634Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'ca'
2021-10-19T05:10:12.8612498Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'no'
2021-10-19T05:10:12.8616453Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'nl'
2021-10-19T05:10:12.8620801Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'zh'
2021-10-19T05:10:12.8624547Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'hu'
2021-10-19T05:10:12.8628834Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'dk'
2021-10-19T05:10:12.8633396Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'es'
2021-10-19T05:10:12.8638274Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'it'
2021-10-19T05:10:12.8642680Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'pt'
2021-10-19T05:10:12.8646840Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'hr'
2021-10-19T05:10:12.8650760Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'sv'
2021-10-19T05:10:12.8654042Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'ko'
2021-10-19T05:10:12.8657299Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'sk'
2021-10-19T05:10:12.8661036Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'fi'
2021-10-19T05:10:12.8664152Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'cn'
2021-10-19T05:10:12.8667180Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'gr'
2021-10-19T05:10:12.8670866Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'ru'
2021-10-19T05:10:12.8674831Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'uk'
2021-10-19T05:10:12.8678601Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'fr'
2021-10-19T05:10:12.8684741Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'et'
2021-10-19T05:10:12.8686057Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'ce'
2021-10-19T05:10:12.8742952Z 00:00:00 Warning: translator::load_files_from_folder():   no pak texts for language 'cz'
2021-10-19T05:10:12.8744369Z 00:00:00 FATAL ERROR: simu_main() - Unable to load any language files
2021-10-19T05:10:12.8745004Z 00:00:00 *** PLEASE INSTALL PROPER BASE FILES ***
2021-10-19T05:10:12.8745607Z 00:00:00
2021-10-19T05:10:12.8746216Z 00:00:00 either run ./get_lang_files.sh
2021-10-19T05:10:12.8747118Z 00:00:00
2021-10-19T05:10:12.8747595Z 00:00:00 or
2021-10-19T05:10:12.8748063Z 00:00:00
2021-10-19T05:10:12.8748812Z 00:00:00 download a complete simutrans archive and put the text/ folder here.
2021-10-19T05:10:12.8749874Z 00:00:00 Aborting program execution ...
2021-10-19T05:10:12.8750270Z 00:00:00
2021-10-19T05:10:12.8751172Z 00:00:00 For help with this error or to file a bug report please see the Simutrans forum at
2021-10-19T05:10:12.8752274Z 00:00:00 https://forum.simutrans.com
2021-10-19T05:10:12.8754398Z 00:00:00 dr_fatal_notify: ERROR: FATAL ERROR: simu_main() - Unable to load any language files
2021-10-19T05:10:12.8755242Z 00:00:00 *** PLEASE INSTALL PROPER BASE FILES ***
2021-10-19T05:10:12.8755619Z 00:00:00
2021-10-19T05:10:12.8755994Z 00:00:00 either run ./get_lang_files.sh
...

[ceeac]

No, the path is correct: The first simutrans/ subdirectory is the repository name, the second one is the name of the checkout (defaults to the name of the repository) and the third one is the data directory.

automatet test load not language files

Fixed in r10185.

[Andarix]

r10185 works

log