News:

Simutrans Forum Archive
A complete record of the old Simutrans Forum.

All online servers have crashed

Started by Matthew, June 27, 2021, 09:05:52 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Matthew

All online servers appeared to crash after the update this morning (27 June).

Here is the Bridgewater-Brunel log:
Simutrans version 120.4.1 Extended Nightly development build 14.15 from Jun 27 2021 #decc493
Message: simu_main(): Server started on port 13353
Warning: nwc_routesearch_t::reset: all static variables are reset
Warning: nwc_routesearch_t::reset: all static variables are reset
Message: network_init_server(): Attempting to bind listening sockets for: "46.32.231.222"

Message: network_init_server(): Added valid listen socket for address: "46.32.231.222"

Message: socket_list_t::add_server: add server socket[5]
Warning: nwc_routesearch_t::reset: all static variables are reset
FATAL ERROR: gui_theme_t::init_gui_themes - Wrong theme loaded
Aborting program execution ...

For help with this error or to file a bug report please see the Simutrans forum at
https://forum.simutrans.com


It's curious that the server is crashing on GUI code, since it doesn't have a GUI.

The client can load a Bridgewater-Brunel game in single-player mode. Before loading, it also logs a theme problem:
Warning: simwin.cc themes_init(): Can't read themes from themes.tab

I had a look at recent commits. There was nothing specific relating to themes. But I noticed this enormous commit has the message: Merge branch 'master' of https://github.com/Ranran-the-JuicyPork/simutrans-extended.git into merge-from-standard-june-2021
Maybe it was accidentally merged into the main branch rather than merge-from-standard-june-2021??
(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。

Ranran(Hibernating)

I noticed that the parameters added in r9793 are missing in some theme files, but I'm not sure if this is related to the server crash.
And there was a syntax error in one theme file. That is my mistake.  :-[
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

jamespetts

Thank you for the report. The merger was intentional in order to merge some of Ranran's latest work, but it seems that this fails with the command line server build.

I have incorporated Ranran's patch for this and run an emergency rebuild on the Bridgewater-Brunel server, but unfortunately, this has not worked, and I still get this error when starting the command line server build:


FATAL ERROR: gui_theme_t::init_gui_themes - Wrong theme loaded
Aborting program execution ...

For help with this error or to file a bug report please see the Simutrans forum at
https://forum.simutrans.com
dr_fatal_notify: ERROR: FATAL ERROR: gui_theme_t::init_gui_themes - Wrong theme loaded
Aborting program execution ...

For help with this error or to file a bug report please see the Simutrans forum at
https://forum.simutrans.com
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Ranran(Hibernating)

if(  skinverwaltung_t::divider == NULL  ) {
// usually there should be a default theme which would provided missing images even for outdated themes
dbg->fatal( "gui_theme_t::init_gui_themes", "Wrong theme loaded" );
}

The only place to print that error message is when the divider image cannot be loaded.
I have no idea why it doesn't load.
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

freddyhayward

Quote from: Ranran on June 27, 2021, 11:24:16 AM
if(  skinverwaltung_t::divider == NULL  ) {
// usually there should be a default theme which would provided missing images even for outdated themes
dbg->fatal( "gui_theme_t::init_gui_themes", "Wrong theme loaded" );
}

The only place to print that error message is when the divider image cannot be loaded.
I have no idea why it doesn't load.
In a headless server, themes and images are not loaded.

jamespetts

I have attempted a fix for this by excluding the themes code for a command line server build, but this also seems to fail in that the pakset now does not load:
Message: simu_main():   Reading electricity consumption configuration ...
Message: simu_main():   Reading menu configuration ...
ERROR: create_general_tool():   cannot satisfy request for general_tool[42]!
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
ERROR: create_general_tool():   cannot satisfy request for general_tool[43]!
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
Warning: create_simple_tool():  deprecated tool [27] requested
ERROR: create_dialog_tool():    cannot satisfy request for dialog_tool[34]!
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
Message: simu_main():   Reading object data from Pak128.Britain-Ex/...
FATAL ERROR: simu_main() - Failed to load pakset. Please re-download or select another pakset.
Aborting program execution ...

For help with this error or to file a bug report please see the Simutrans forum at
https://forum.simutrans.com
dr_fatal_notify: ERROR: FATAL ERROR: simu_main() - Failed to load pakset. Please re-download or select another pakset.
Aborting program execution ...

For help with this error or to file a bug report please see the Simutrans forum at
https://forum.simutrans.com

Aborted (core dumped)

Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Ranran(Hibernating)

Warning: simwin.cc themes_init(): Can't read themes from themes.tab
This message seems to be displayed until change the theme once. If you delete setting.xml, it will reappear.
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

jamespetts

Quote from: Ranran on June 27, 2021, 01:00:00 PM
Warning: simwin.cc themes_init(): Can't read themes from themes.tab
This message seems to be displayed until change the theme once. If you delete setting.xml, it will reappear.

I have re-tested this with a command line server build, which I have now set up to be compiled in Visual Studio. Unfortunately, deleting settings.xml does not prevent the fatal error. I have reverted my earlier attempted fix.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Ranran(Hibernating)

#8
Sadly, I've reverted and tested the commits up to May 27 4, but it seems that the cause is earlier.

Playing offline is fine, so I can't figure out what's affecting it... (´・ω・`)


EDIT:
Warning: simwin.cc themes_init(): Can't read themes from themes.tab
Warning: create_simple_tool(): deprecated tool [27] requested
ERROR: create_dialog_tool(): cannot satisfy request for dialog_tool[34]!

These errors are still present around February and may not be related to the server crash.
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

Ranran(Hibernating)

I noticed that the locked gadget branch was already merged into the nightly build.
It hasn't covered many themes yet, so it had to be thoroughly tested.
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

Ranran(Hibernating)

#10
Quote from: Ranran on June 27, 2021, 02:15:00 PMEDIT:
Warning: simwin.cc themes_init(): Can't read themes from themes.tab
Warning: create_simple_tool(): deprecated tool [27] requested
ERROR: create_dialog_tool(): cannot satisfy request for dialog_tool[34]!

These errors are still present around February and may not be related to the server crash.
These errors are not related to the server crash as they still occur after reverting the incorporating from the standard.


Message: simu_main():   Reading object data from Pak128.Britain-Ex/...
FATAL ERROR: simu_main() - Failed to load pakset. Please re-download or select another pakset.

This error does not occur in offline games, i.e. pakset loads correctly, but I don't know why this is occuring in server games.
Also, judging from the log, it seems that false is returned because the pak file or folder cannot be found in obj_reader_t::load ().
But why does it occur only in server games?

EDIT:
I suspect that either the change to simmain.cc or the changes that were made at the same time were the cause.
https://github.com/jamespetts/simutrans-extended/commit/1cfa6333f20721476695a71a0ab2f9004c26d008

EDIT2:
That error message was added in r9699 (EX).
Changes around it may also be related to this.

I'm not sure about the difference between offline games and network games, especially with regard to this symptomatology.
I am worried that I may not be able to find the problem area by searching in the dark clouds. I think it would be quicker to identify the problem by retaking the incorporating little by little again.
I apologize for the inconvenience.
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

Phystam

In my environment of the server game, the console shows this message and simultaneously crashes:

This message cannot be shown in GUI clients -- it is shown only in no-window game.

Ranran(Hibernating)

I presume that both are the same problem in that the network game server does not correctly recognize the required (pakset/theme) folders.


The client server can start properly. But another client's connection doesn't work.
The connected client is in a paused state and cannot be released.
Quote from: Ranran on June 27, 2021, 11:54:47 PMI am worried that I may not be able to find the problem area by searching in the dark clouds. I think it would be quicker to identify the problem by retaking the incorporating little by little again.
I apologize for the inconvenience.
So unfortunately it currently has a big problem with network games and I think we'll have to undo it and then retake the merging. (´・ω・`)
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

freddyhayward

Quote from: jamespetts on June 27, 2021, 10:57:24 AMThank you for the report. The merger was intentional in order to merge some of Ranran's latest work, but it seems that this fails with the command line server build.
What work are you referring to here?

prissi

Standard has that warning, but the server starts anyway (at debug level 2):

...
Warning: sound_desc_t::get_sound_id():  sound "steam.wav" no found
Warning: simwin.cc themes_init():       Can't read themes from themes.tab
Warning: karte_t::load: disconnecting all clients
...

jamespetts

Quote from: freddyhayward on June 28, 2021, 10:21:55 AM
What work are you referring to here?

I was referring to this work.

Ranran - are you able to revert changes up to the point when the bug is first introduced and push this onto a branch? Thank you for looking into this.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Ranran(Hibernating)

#16
Quote from: prissi on June 28, 2021, 12:32:20 PMStandard has that warning, but the server starts anyway (at debug level 2):


Warning: simwin.cc themes_init():       Can't read themes from themes.tab

This warning messege will be displayed if the specified theme(env_t::default_theme) is searched for in the specified folder and not found. For example, try to open modern.tab from the user folder. However, if it is not found, then the default theme, themes.tab, will be opened and the problem will not occur.
Initially, the wrong theme is specified, or it looks like this warning is always displayed to look for a null theme. Once you change the theme, this warning will disappear because env_t::default_theme remembers the correct theme.
If there is no themes.tab, a total of 2 warnings will be displayed. In that case, I don't think it can be booted the game.

Quote from: jamespetts on June 28, 2021, 06:24:17 PMRanran - are you able to revert changes up to the point when the bug is first introduced and push this onto a branch? Thank you for looking into this.
Pull request #401 once reverts mass merging from the standard.

Unfortunately, it will be incompatible with saved games made in the last two days of nightly builds. I haven't tested the compatibility of saves, so I'm not sure. However, it will be readable again after the merge is redone.

EDIT:
Pull request #402: has been reverted mass merging and then changed up to r9642 and some theme and GUI related changes.
It seems that the save conflict with these two days will be resolved to some extent.
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

prissi

Sorry, indeed standard was trying to often to load themes if never ever initilised with graphics. Fixed in r9919


Ranran(Hibernating)

prissi's change made me realize that one commit was missing .
I was thinking of retaking the patch, but before that, James, can you please check once if the changes in my master branch improve the symptom?
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

jamespetts

Quote from: Ranran on June 29, 2021, 09:07:22 AM
prissi's change made me realize that one commit was missing .
I was thinking of retaking the patch, but before that, James, can you please check once if the changes in my master branch improve the symptom?

Thank you for your work on this. I have merged your master branch. However, this does not seem to assist: running this in Visual Studio, I get the same error message of "wrong theme loaded" when trying to run a command line server build.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

freddyhayward

Quote from: Ranran on June 29, 2021, 09:07:22 AM
prissi's change made me realize that one commit was missing .
I was thinking of retaking the patch, but before that, James, can you please check once if the changes in my master branch improve the symptom?
I have a different error for your master branch, but your r9757 branch works fine.

freddyhayward

Quote from: jamespetts on June 29, 2021, 09:57:17 AM
Thank you for your work on this. I have merged your master branch. However, this does not seem to assist: running this in Visual Studio, I get the same error message of "wrong theme loaded" when trying to run a command line server build.
Could you please revert the offending changes until the problem is resolved? This sort of troubleshooting should be happening in a separate branch, not the master branch.

Ranran(Hibernating)

#22
Quote from: jamespetts on June 29, 2021, 09:57:17 AMThank you for your work on this. I have merged your master branch. However, this does not seem to assist: running this in Visual Studio, I get the same error message of "wrong theme loaded" when trying to run a command line server build.
Quote from: freddyhayward on June 29, 2021, 10:02:12 AMI have a different error for your master branch, but your r9757 branch works fine.
Thank you for your confirming. Now, I would like to go back and merge again.

Quote from: Ranran on June 28, 2021, 10:12:51 PMPull request #402: has been reverted mass merging and then changed up to r9642 and some theme and GUI related changes.
It seems that the save conflict with these two days will be resolved to some extent.
After reverting the merge of my master branch, please merge pull request #402.
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

jamespetts

Thank you for that - I have now incorporated the std-r9637, which appears to deal with the themes problem, although does crash the game on startup unless settings.xml be deleted. I am not sure whether this will be an issue with the server. I am running an emergency rebuild on the server now to allow the game to restart, assuming that there be no settings.xml issue there.

Players will have to delete their settings.xml files to get the game to work again. I have very little time at present, but will look into the other branches when I have more time.

Thank you for your work on this.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Ranran(Hibernating)

Quote from: jamespetts on June 29, 2021, 11:57:29 AMwhich appears to deal with the themes problem, although does crash the game on startup unless settings.xml be deleted. I am not sure whether this will be an issue with the server. I am running an emergency rebuild on the server now to allow the game to restart, assuming that there be no settings.xml issue there.
Pull request # 404 tries to work around the setting.xml issue, but this may also cause new setting.xml conflicts before and after this change.
(´・ω・`)シミュトランスのアップデート履歴(日本語) (※更新停止中)
bit.ly/3AuKHHP

jamespetts

Quote from: Ranran on June 29, 2021, 01:20:22 PM
Pull request # 404 tries to work around the setting.xml issue, but this may also cause new setting.xml conflicts before and after this change.

Thank you - incorporated. Hopefully, this will assist; I am afraid that I have not had time to test this fully.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Matthew

Thank you to everyone, and especially Ranran, who has worked on this problem.

Quote from: Ranran on June 27, 2021, 02:15:00 PM

These errors are still present around February and may not be related to the server crash.

It looks like my comment about themes started the investigation in the wrong direction - sorry for that. I did not play Simutrans-Extended between January and May, so this was the first time I'd seen the themes error in the log.


(Signature being tested) If you enjoy playing Simutrans, then you might also enjoy watching Japan Railway Journal
Available in English and simplified Chinese
如果您喜欢玩Simutrans的话,那么说不定就想看《日本铁路之旅》(英语也有简体中文字幕)。