deManglers test Simutrans-Experimental Server

[deMangler]

There is a simutrans experimental test server.


System: Minimal Debian 6.0 VPS with 256MB RAM / 256MB swap.
Some basic information about the current status of the server can be found here.


Game clients can connect to it at zeropoint.cc

It uses Pak128.Britain-Ex-0.8 and uses version here:- (version 9.12 clients should work)


Any and all feedback appreciated.
Thanks.
dM




I have tried to connect to it but I always have a pak mismatch (even though the server and client use exactly the same setup apart from the binaries, as I am connecting from a windows client) *fixed*

[jamespetts]

DeMangler,
thank you for checking this. I am not sure what is causing the mismatch (perhaps case sensitivity in the pakset file names?), but you can force join, ignoring the erroneous pakset mismatch by using the load game dialogue and typing in the server address (there might be an additional switch; but I do not immediately remember now).

I shall try connecting to this when I get home. Thank you very much for setting up a server; it is most appreciated.

[deMangler]

I noticed that even though I thought I was running a 9.6 windows exe it was in fact 9.5.
I think this is because I was mistakenly downloading the wrong zip (your github link to the windows latest zip is not working.)
I used the alternative link posted here http://forum.simutrans.com/index.php?topic=1894.msg70939#msg70939
Now I am running a 9.6 windows exe it does not report a pak mis-match and I can connect fine.
Thanks.

[Vonjo]

Nice. No problem so far. 

But I think you should password public service player.

[deMangler]

Nice. No problem so far. 

But I think you should password public service player.

Thanks for the tip, (and the info about using simgraph0.cc from standard btw).
I will definitely password the public service player when I start up a serious one, I will probably use obey timeline too on at least one game.
This game is mainly so I can get an idea of how large a map the hardware can handle. I normally play with huge maps and I think with multi-player, the bigger map the better.
At the moment the server is using about 30% of the total RAM (including swap) available to it (according to top).
I need to see how that changes when people build large networks and passenger / vehicle numbers increase.
I do not really want to go over 50% on the RAM for the game server alone, so we shall see.

dM

[Vonjo]

Ok, no problem. It means, to test the server, I can build factories and cities, right? So we can test the server under heavier load.

[deMangler]

Ok, no problem. It means, to test the server, I can build factories and cities, right? So we can test the server under heavier load.

Yes.

[jamespetts]

I should note that playing with the timeline off is very much not recommended for par128.Britain, especially the Experimental version.

[deMangler]

I should note that playing with the timeline off is very much not recommended for par128.Britain, especially the Experimental version.

I am glad you said that. I normally play with huge maps and obey timeline on from 1830. I hope to be able to do this with the server too when I have an idea what it can handle.
If anyone has any knowledge of the resource usage of simutrans experimental servers on linux with various size maps and populations I would be pretty interested.
In any case, I hope to be able to contribute to such knowledge soon.
dM

[jamespetts]

deMangler,

I think that you are the pioneer in this field so far - I shall be very interested in your results!

[jamespetts]

I have now started playing this as a provider of express coach services - "Brown Line". I am noticing desyncs when I try to change the names of stops, players or lines, but, aside from that, all is going well.

Edit: DeMangler - I suggest that you connect your dock in Pentchester to a 'bus stop to integrate with my 'bus network there to increase the ridership on your ferry.

Edit 2: One particular problem that comes from playing with the timeline off is that road vehicles get stuck behind horse and cart type private cars that should not exist in later eras.

[deMangler]

I have now started playing this as a provider of express coach services - "Brown Line". I am noticing desyncs when I try to change the names of stops, players or lines, but, aside from that, all is going well.

In total I have had 5 desyncs. 3 were when adjusting or renaming lines from within 'new line' in a depot. The other two were just when I was observing.
The server itself has been 100% stable with no CPU/memory use spikes or other signs of a problem.
As of 1869 the memory usage has increased from 80MB to 85MB since the game started in 1830.
Bandwidth usage:
1 client  25kbits/sec
2 clients 37kbits/sec
<edit> Hmm. actually there was iptraf over ssh in there too. My bad. Very approximate figures anyway.

Edit: DeMangler - I suggest that you connect your dock in Pentchester to a 'bus stop to integrate with my 'bus network there to increase the ridership on your ferry.

I figure I may as well make it public as that is something that has not happened yet in this game.

Edit 2: One particular problem that comes from playing with the timeline off is that road vehicles get stuck behind horse and cart type private cars that should not exist in later eras.

Yes, I had problems with that early on which was why I set up the ferry service between Birnwell and Brushliegh Docks.
Thanks to your technologically miraculous coach service the captain of the SS Great Britain has had to sell his daughter's virtue to pay for his aging mother's medical bills and import cheap slave labour from the furthest reaches of the empire to run the boilers.  

<edit> I just noticed actually that due to it's minimum load limit SS Great Britain is spending all it's time in the dock. Reminds me of when I used to live in Bristol I use to walk past the SS Great Britain regularly.
Now I live in Swindon I walk past a big statue of Isambard Kingdom Brunel lots. He certainly did leave a legacy.  A statue of him in pak128 Britain would be very appropriate. Anyway, off topic, blah blah.....

[jamespetts]

This is encouraging indeed! It's good to see that this remains stable. I am just about to push version 9.7 with a few bugfixes - perhaps we could start again with the new version and a map twice as large with 4x as many cities and the timeline on to see how the server copes with something a little more strenuous?

Edit: As the server is about to be taken down and restarted with 9.7 and a larger map, some things that I learned by playing this game:
* it pays to keep waiting times down on local 'bus routes, especially when they feed into longer-distance coach or train routes (do this by having several different direct, linear routes in a town rather than a single circular route, and have at least a couple of 'buses on each);
* ensuring that stations/stops don't get too crowded and have enough capacity is important to prevent losing large amounts of money through refunds;
* the more routes served by a stop, the larger that its capacity will need to be;
* the convoy spacing feature is very effective for local 'bus routes with more than one 'bus if the cnv/month value is set appropriately - something around 60-70 is good for a small town 'bus route with two 'buses;
* connecting to a city containing another player's 'bus network will often cause that network to become overcrowded; if that other player is not around to fix it, be prepared to add capacity of your own; and
* watch out for 'bus loading times - the single door type of 'bus takes longer to load than the double door type, which can impede average speeds, especially in larger towns.

[deMangler]

I have re-started the server with version 9.7.
There is a somewhat larger map and more cities / industries.
Also the public player is passworded this time and obey timeline is on, from 1830 (so could be a bit of a slower start than last time).
Again - all in the interest of testing so joining players be warned the server may be re-started for testing purposes before your empires have fulfilled all your dreams.....

Have fun.
dM

For those interested in the start and end savegames from the previous test server (v9.6) they can be downloaded here.

[deMangler]

Windows Client v 9.7 On-line Play Crash Report.

Created Coaching Stable
Clicked on single horse.

Popup:

Code Select Expand


FATAL_ERROR:
float32e8_t::operator *
(const float32e8_t & x)
const
Overflow in 1.#INF * 1.5
PRESS ANY KEY

Key pressed. Client Crashed.

Any subsequent login if the coaching stable is clicked above popup displays as before.
Also every time I reconnect after this crash the message stating the number of connected clients increments by one.
This happens whether I have previously entered my password or not.
I appear to be able to create and click on Staging Posts ok.

Server still appears stable.

dM

[jamespetts]

dM,

thank you for the bug report: this is related to the switch from fraction to synthetic floating point; I have reported this to Bernd, who wrote that code. I suspect that it is related to the fact that single horses have very little power. In the meantime, the double horse set does not have the same problem, so I suggest using that for the time being.

[jamespetts]

I have found a further bug with 9.7 - the maximum walking distance is incorrectly calculated when creating a saved game, resulting in passengers being able to walk to all destinations within 6,000 tiles (and thus not using the player's transport). I have fixed the bug in my 9.x branch, so it will be fixed for the next release, but, in the meantime, here is a saved game with a corrected value substituted manually.

[deMangler]

I have found a further bug with 9.7 - the maximum walking distance is incorrectly calculated when creating a saved game, resulting in passengers being able to walk to all destinations within 6,000 tiles (and thus not using the player's transport). I have fixed the bug in my 9.x branch, so it will be fixed for the next release, but, in the meantime, here is a saved game with a corrected value substituted manually.

I have re-started the server from the fixed savegame.
For some reason the server listing at http://simutrans-germany.com/serverlist_ex/ has begun showing zeropoint.cc as off-line quite often when it is really on-line, so try connecting anyway it may well be ok.
dM

[jamespetts]

I have found a number of critical errors in 9.7 (including one that gave rise to large negative revenues incorrectly, which stopped me from having enough money to buy trains for my railway line in this game, and one that made the maintenance costs of buildings far too low), so I have released a critical fix release in the form of 9.8. This also fixes the crashing issue with the single horse, although note that an erroneous speed may be displayed in the depot window when using the single horse. I strongly recommend that the server be upgraded to version 9.8. Apologies for the difficulties, and thank you for your help with testing.

[Bernd Gabriel]

dM,

thank you for the bug report: this is related to the switch from fraction to synthetic floating point; I have reported this to Bernd, who wrote that code. I suspect that it is related to the fact that single horses have very little power. In the meantime, the double horse set does not have the same problem, so I suggest using that for the time being.

single horses no longer produce illegal numbers in my master branch

[jamespetts]

Bernd,

tested - thank you very much! It's too small of a fix for an update in itself, but it will be in the next and subsequent versions.

[deMangler]

Server has been restarted with a new map and version 9.8.

[jamespetts]

Excellent! Probably shan't be able to do much until to-morrow evening, but this should be a rather more productive test than the false start that was 9.7. Thank you very much indeed for hosting this.

Edit: I notice that the towns and industries are rather few in number - a better game, I suspect, will be had with a somewhat denser map (and it will be a rather better test of the resource consumption, too). As it stands, whoever establishes a railway between Sackingport and Pondsand will make large quantities of money, whereas everyone else will be rather out in the cold.

[deMangler]

Ok - restarted with more stuff (than ever before!!!)
It did look a bit empty.
Maybe we can push some limits.  
dM

<edit>
After this instance had been running for 1 1/2 hours I noticed I could no longer log-in. I would get "server does not respond. Even after re-starting the server.
As far as resource usage goes the server appeared normal. No strange output to the console."



Re-starting the server without using the automatic save seems to work though.
I have now re-started the server with the same map at it's initial state to see if it repeats.

Before anyone asks... I don't have a save or the logs because I stupidly did not back them up the instant I noticed a problem, so they were overwritten. Ooops. Must remember to do that next time.
dM

[jamespetts]

Excellent! Should be an interesting test. I don't have time to log in myself until this evening, but should encourage others to do so to test and have fun! Thank you for your work on this.

[deMangler]

Here is a screenshot of the map from 1867, after the game has been running since 1830, or 2 days 6 hours.



Click to enlarge.



dM

[deMangler]

Server Crashed at 09/06/11  22:18 UTC.

Fortunately - this time the latest save appears to be working fine, so back up again, for now....

Log and save here.

dM

[sdog]

i think it went down with my client segfaulting. (sorry i don't have any debug information)

I sent a train to "withdraw" clicked on it when i expected it to be empty and gone was the client.

[deMangler]

The server seg-faulted as well. luckily I am an insomniac computer freak so I am still awake at 0630 am.  
I may have to write a script to periodically check if the server has stopped running and bring it up after doing backups and reporting, kind of like the one that updates the on-line/off-line image in the top of this thread.
I don't know of a way to get the server to autosave regularly, the setting in conf doesn't seem to do anything with the server (works with the client). It only seems to save when someone logs in.

Hope not too much work was lost in the game....
dM

[deMangler]

Server has segmentation-faulted again.

I suspect it is having memory problems. I will see what can be done, but I may have to try with a smaller map.

dM

[jamespetts]

Why do you think that this is a memory issue? How much memory is being consumed, and how much is free? A segfault means that the programme has tried to access a sector of memory that has not been allocated to that programme. That is almost invariably as a result of a bug in the code, which type of bug is common in C and C++ due to the manual memory management required in those languages. If the problem was insufficient memory, you would be more likley to get an out of memory error.

The difficulty with tracking down access violations/segmentation faults on server games is that it is harder to work out what triggered the crash and therefore find the fault. If people could record as best they can what it was that happened before the crash (and SDog's report above in that respect is useful; I shall have to look into issues with the replacer), that would be most helpful. I do not, however, recommend attempting a smaller map, as that would render less effective our ongoing test about resource utilisation, which test ought not be affected by unrelated crashes.

[deMangler]

There was about 10MB of physical memory free when it was crashing. There was plenty of free swap though.
You can see the memory situation of the server live in the link in  first post of this thread...... Or Here.

I agree it is too soon to try a smaller map, but I need to get a script written to backup the saves and logs and re-start the server automatically because it seems to be seg-faulting more often now and I will not be watching it.

It's nearly finished - then we will be all set.
Shouldn't be too long,

dM

<edit> ok. Here is the script if anyone is interested. seems to work so far.
I have it on a cron job every minute.

Code Select Expand

#!/bin/bash
# By deMangler for Simutrans Experimental Server, but will work for other stuff.
# As-is. No licence.
# Detects if server is not running and backs up log and save and restarts server.
# Backups are dated and numbered.
#
#  **! Define these variables correctly  - these are examples only !**
SERVER_LOCATION="/home/simutrans/simutrans-server/simutrans" #directory containing Server
WEBSTUFF_LOCATION="/home/simutrans/www" # location of status icon if any
BACKUP_LOCATION="$SERVER_LOCATION/bak"  # directory to save backups of log and save
SERVER_PROCESS_NAME="simutrans-experimental-server"       # command to run server
SERVER_PORT="13353" # port
SERVER_ARGS=" -log 1 -debug 2 -server $SERVER_PORT -nomidi -lang en" # Server Args

MAX_RESTARTS=20 # Maximum number of times to restart server and backup files.
#########################################################
if [ -e $SERVER_LOCATION/server-restart-blocked ]; then
    # Script will do nothing if a file called server-restart-blocked is in server directory
    exit # Naughty but nice. Oldskool.
fi


if pidof $SERVER_PROCESS_NAME > 0; then
    #update forum notification icon
    cp $WEBSTUFF_LOCATION/online.png $WEBSTUFF_LOCATION/teststatusico.png
else

    mkdir -p $BACKUP_LOCATION
    #update forum notification icon
    cp $WEBSTUFF_LOCATION/offline.png $WEBSTUFF_LOCATION/teststatusico.png
    DATE=$(date +%Y-%m-%d)
# Make sure that we are using a unique filename for this backup.
NUM=1
while [ -e $BACKUP_LOCATION/savebackup-srv-$SERVER_PORT-$DATE-$NUM.sve ]; do
NUM=$(($NUM + 1))
done
if [ $NUM -le $MAX_RESTARTS ]; then
   cp $SERVER_LOCATION/server$SERVER_PORT-network.sve $BACKUP_LOCATION/savebackup-srv-$SERVER_PORT-$DATE-$NUM.sve
   cp $SERVER_LOCATION/simu-server$SERVER_PORT.log $BACKUP_LOCATION/logbackup-srv-$SERVER_PORT-$DATE-$NUM.log
   $SERVER_LOCATION/$SERVER_PROCESS_NAME$SERVER_ARGS
else
   touch $SERVER_LOCATION/server-restart-blocked
   # Could mail yourself an alert here or something
   # echo "Hello. Server Crashed lots by the way. I give up. *Sigh*"
   :
fi
fi

[jamespetts]

Shall be interested to see how that gets on. Have you looked into using the official Net Tool?

[deMangler]

Sometimes the server hangs (or something) without crashing - at these times it still reports as on-line but it is impossible to log in. I have yet to find a way of detecting this automatically, so if this occurs please feel free to pm me and I will try to get it up and running again as soon as possible.
dM

Shall be interested to see how that gets on. Have you looked into using the official Net Tool?

I had no idea about an official net tool. Forum search brings up nothing. What am I looking for? What does it do?

[jamespetts]

See here for the nettool.