News:

Simutrans Chat Room
Where cool people of Simutrans can meet up.

LAN game lag and desyncs

Started by killwater, October 03, 2014, 12:07:39 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

killwater

Hello
I am trying to play a LAN game however I am getting a lot of desync and there is a lot of lag. The connection ping is 2-5ms. I am playing experimental package from the how to get post, version 11.35. Is there anything I can do to make it better? The lag is like 1 minute before the work done is displayed.

jamespetts

Lag can be caused by the client machine being a lot slower than the server and falling behind, rather than by network conditions. In extreme cases, this might also cause desyncs. Are you running a large map? What are the specifications of your client?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

killwater

The map is not very big - 256x256km so it should not be a problem however the server is indeed much faster. The client is AMD e2-1800 while the server is intel i5-3210M. Both have 4GB RAM.

jamespetts

What is the maximum fast forward speed that you get on that map on the client and on the server?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood

Do be aware that in 11.35 there is a critical problem with power networks. Any form of power distribution network will cause clients to quickly loose synchronization and be disconnected. This is currently why power is banned/not used on the big main server.

Junna

Quote from: DrSuperGood on October 04, 2014, 01:02:52 AM
Do be aware that in 11.35 there is a critical problem with power networks. Any form of power distribution network will cause clients to quickly loose synchronization and be disconnected. This is currently why power is banned/not used on the big main server.

This happens on newly created maps as well, where there is no pre-existing infrastructure.

My friend also has a 2-core 1.8 (much slower than my, the host's, computer) and we experience the same kind of awful unplayable lag. However, in fast forward, we have a comparable speed (around 80-85 mostly; though his occasionally dropping to 50) on the same multiplayer map loaded singly.

There are no power lines on the map, either.

jamespetts

That is rather curious. Does this persist once infrastructure is added? Is it, perhaps, an issue with the server's settings?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Junna

It does persist and usually in the form of very great delay to the client; it might take over 5 minutes or more for, when the client builds something, said things to appear. (Usually I, the host, sees the things built far before).

The settings are the default ones. I think I tried to increase frames behind once but didn't notice any improvement.

jamespetts

Very odd. Does this occur when using a saved game on your server?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Junna

Both newly created and loading a saved game.

jamespetts

That seems odd, as your experience is different to what is seen on the server. May I ask what your settings on the server actually are; perhaps you could copy and paste the network section into this thread? Also, make sure that the pakset simuconf.tab settings do not override the base simuconf.tab settings if you are assuming that the base simuconf.tab settings are the ones that apply and you are using a pakset other than Pak128.Britain-Ex.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

killwater

I can reaffirm Junna observations. The fast speed on client and server is similar and about 80. The lag time seem to fluctuate from zero to about 5 game minutes. I have also noticed some strange behaviour with loading coal - there is often a situation that only 225t is loaded despite having about 500t in the dock. Similar situation can be observed during unloading.

jamespetts

Can you post your server's network settings, too?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

killwater

Here it is:
################################### Network settings ##############################
#
#listen = 127.0.0.1
server_frames_ahead = 4
additional_client_frames_behind = 4
server_frames_per_step = 4
server_frames_between_checks = 256
server_announce = 1
server_announce_interval = 900
server_dns = 127.0.0.1
server_name = A generic Simutrans-Experimental server
server_comments = This is a Simutrans-Experimental server running with default settings.
server_email = maintainer@your.server
server_admin_pw
server_pakurl = http://your.domain/pakset.zip
server_infurl = http://your.domain/server-info.html
pause_server_no_clients = 1
#nickname = John Doe
chat_transparency = 75
unprotect_abondoned_player_months = 36
remove_dummy_player_months = 9

I took it from config/simuconf.tab and deleted hashed comments. If you need something else let me know. 

DrSuperGood

QuoteThe lag time seem to fluctuate from zero to about 5 game minutes
Are you saying that it still remains connected but orders take 5 minutes to execute? That would be symptomatic of the game running slow on the client.

Is the client giving problems a laptop? Simutrans is very CPU heavy and so there is a good chance that it will cause laptop CPUs to overheat as they are never designed to run at high clock speeds for long periods (cheap laptops especially). When the CPU overheats it will slow down, possibly to the point that the client can no longer keep up with the server. At that stage lag will start to occur and it could possibly grow to over 5 minutes with time.

killwater

Both are laptops. Client has about 15% load on CPU.
I have two games on another laptop(server+ client both in graphic mode) and I still experience desyncs. No lag though. The second laptop runs on about 10% load.
No power infrastructure in the game at all.

DrSuperGood

QuoteBoth are laptops. Client has about 15% load on CPU.
I have two games on another laptop(server+ client both in graphic mode) and I still experience desyncs. No lag though.
To handle desyncs use a cable network such as Ethernet instead of wireless since wireless is unreliable. Yes it may have good latency at times however starting a microwave or washing machine could cause packet loss and things. The only other solution is to set your client frames behind setting to something high like 15/20.

Overheating could easily be why there is the huge lag since the one client never can catch up with the server. Experimental is a lot more demanding than standard due to improved convoy physics, passenger mechanics etc so one can expect it to use more processor resources. Try starting the server at a slower game speed (half) and see if it happens. I could be totally wrong however since these things are very hard to diagnose.

killwater

As I mentioned it there should be no problem with overheating as the load on CPU is only about 15%. So unless it is so a disastrous design it should be perfectly fine. As for ethernet cable I do not have one so cannot check that one.

DrSuperGood

QuoteAs I mentioned it there should be no problem with overheating as the load on CPU is only about 15%.
That metric is meaningless as it could be anything from all X (where X depends on your processor) cores running at 15% to 1 core running at 100% with some other cores doing a little. Power consumption is related to the worst case core utilization as it cannot throttle back the clock speed without introducing a bottleneck.

Simutrans is mainly single threaded (as it was not a design consideration back when the projected started). The only part that is really multi threaded is the software render. The software render should perform the same in experimental as it does in standard so not be an issue however the core of experimental is probably considerably more demanding than standard.

Try running the server at half or third speed to see if you still get the 5 minute latency starting.

jamespetts

For reference, here are the settings of the current Bridgewater-Brunel server. Electricity supply has been reported to cause desyncs, so try removing it (if there is any) to see whether that helps.


################################### Network settings ##############################
#
# Synchronized networking is always a trade off between fast respone and safe
# connections. A more relaxed timing will cause delay of commands but is more
# likely to compensate for clients running slightly faster than the rest.
#

# Sets the local addresses Simutrans should listen on and use for making outgoing connections
# A DNS name may be specified, this will be resolved and Simutrans will attempt to listen
# on all of the addresses returned.
# By default it will use all local IPv4 and IPv6 addresses
# This setting has no effect if Simutrans has been compiled with the USE_IP4_ONLY flag set!
# The addresses listed will be tried in the order specified
listen = 46.32.231.222

# How much commands should be delayed before sending back to clients.
# A larger number will catch even clients running slightly ahead but cause delay.
# This is set by the server side.
#
# The sum of server_frames_ahead and additional_client_frames_behind should be
# evenly divisble by server_frames_between_checks for best network performance.
server_frames_ahead = 4

# How much extra delay in command execution on the client side, on top of server_frames_ahead.
# A larger number can compensate for larger fluctuations in communication latency.
# This is set by the client side.
#
# The sum of server_frames_ahead and additional_client_frames_behind should be
# evenly divisble by server_frames_between_checks for best network performance.
additional_client_frames_behind = 4

# In network mode, there will be a fixed number of screen updates before a step.
# Reasonable values should result in 2-5 steps per second.
#
# The sum of server_frames_ahead and additional_client_frames_behind should be
# evenly divisble by server_frames_between_checks for best network performance.
server_frames_per_step = 4

# The server sends after a fixed number of steps some information to the clients.
# Large values here means: reduced server communication (if that is of importance...)
# Small values should improve the timing of the clients.
server_frames_between_checks = 1

# Automatically announce server on the central server directory (http://servers.experimental.simutrans.org/)
# 0 (default) = off, 1 = on
server_announce = 1

# Interval of server announcement (if enabled)
# Value is number of seconds between server announcements, default is 900 (15 minutes)
# Minimum value is 60 (1 minute), for accurate listing it is recommended not to increase
# this value to greater than 3600 (1 hour)
# To disable announcements set server_announce to 0
server_announce_interval = 900

# Fully Qualified Domain Name (FQDN) or IP address of your server (IPv6 or IPv4)
server_dns = bridgewater-brunel.me.uk

# Name of server in server listing
server_name = Bridgewater-Brunel

# Additional information about your server (for display on the list server)
server_comments = This server is set up to run large Pak128.Britain-Ex games on a VPS with 3Gb of RAM available and one dedicated core.

# Email address of server maintainer (for display on the list server)
server_email = **REDACTED** (to prevent spambots)

# The password required for administering the server.
# NOTE: This should be changed from the default when used, for
# obvious reasons.
server_admin_pw  = **REDACTED**

# Pakset download URL (for display on the list server)
server_pakurl = http://www.bridgewater-brunel.me.uk/downloads/Pak128.Britain-Ex-0.9.1.zip

# Server info URL (for display on the list server)
server_infurl = http://www.bridgewater-brunel.me.uk

# Pause server when no clients are connected
pause_server_no_clients = 1

# Nickname when joining network games
#nickname = John Doe

# Chat window transparency (0=off, 25, 50 75 are possible)
chat_transparency = 75

# The number of game months before a player making no changes
# to its company is unlocked automatically to allow other players
# to take over.
unprotect_abandoned_player_months = 360

# The number of game months before a player that has never built
# anything substantive is deleted automatically.
remove_dummy_player_months = 24


Incidentally, the multi-threading is used mainly for loading/saving: multi-threading of the rendering is a newer feature in Standard not yet ported to Experimental, although GDL itself is multi-threaded, so some small aspect of the display end of things is multi-threaded by default.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Junna

The client's connection is indeed over a wireless connection that is intermittently unstable - but we do not all afford the luxury of easily correcting this, when the only modem you have is a wireless aged abomination. It should be noted that, however, there did not use to be significant problems with older versions of Sim ex. from about a year and some ago. Connection was at times unstable, but then only resulted in infrequent desynchs when connection truly was disrupted: it did not, as now, be stuck in a permanent state of extreme client delay from the moment of joining.

DrSuperGood

Write off the desynchronization as network stability issues and the odd experimental bug (it is not perfect, you will desynchronize if you issue an order just as the game saves). Due to you using LAN and not a mega massive sized map it should be no issue to re-join.

The growing lag is most likely due to the client being unable to keep up with the server. Try running in fast forward mode for an extended period (say 5-7 minutes) to see if the advancement rate drops. If so this would be overheating related otherwise we are pretty much back to square 1 (something else is cause).

killwater

#22
@DrSuperGood
This is a dual core processor and 15% CPU utilization is top value including everything running in the system. Power consumption should be considered for all cores at once as cooling is shared throughout them and so designed to withstand the load of all cores working simultaneously.  Anyway it makes only 30% utilization of one core so I still do not think this is an issue.
@Junna
The thing is in my case there a periods in which the delay renders the game unplayable. And sometimes it can be played for half a day without a single issue. You sound like it is a constant delay in your case- is it correct?

DrSuperGood

#23
QuotePower consumption should be considered for all cores at once as cooling is shared throughout them and so designed to withstand the load of all cores working simultaneously.
Not in laptops. They purposely design them with insufficient cooling to save power and production cost. This is a well known issue that plagues laptop gamming.

Power consumption of processors is based on the clock speed and to some degree the logic being executed. As such you are correct in saying that a processor running 1 core fully and the others practically idle will use less power than one with all cores fully utilized. However that difference is minimal compared with the difference clock speed makes. Power consumption is highly related to the clock frequency. If 1 core has to run at full clock frequency then everything does and in doing so power consumption is dramatically raised.

This is why it is important to try and make processes that balance load across all cores in mobile design.

Let me give you an example.
I use a desktop computer system which has excessive cooling on it (was custom built so did not save on cooling). Since it is air cooled it makes a noise proportional to the amount of cooling required by the dynamic fan controller. The fan controller responds to power/temperature changes to keep the processor operating well within its designed range. When running something single threaded like an old console emulator with no speed limit my fan runs very loudly trying to keep the processor cool despite only 1 of my 4 cores being used (ignoring HT cores). On the other hand I can play a game like StarCraft II which uses 2 cores however none are loaded to 100% due to bottlenecks in synchronization and the fan is less noise. The result is that 1 core running at 100% uses more power than 2 cores running at 80-90%.

Another example proving how what is run influences power consumption is with regards to a GPU I own (which is irrelevant to simutrans but proves the point). An 8800GT was a pretty powerful GPU at the time it was released, capable of "next generation" graphics at the time. It worked amazing running Direct3D 9 games at unbelievable quality. However along came StarCraft II and suddenly reports came in of the card literally going up in smoke. The reason is that the Direct3D 10 graphics used by StarCraft II were implemented in a very inefficient way on the GPU resulting in excessive heat production which the stock cooler on the card could not cope with. The result was you had to turn down shaders and limit frame rate to like 30 fps and then no overheating occurred. The card was more than powerful enough to run a lot of games as pretty insane settings for the time however it overheated when running that game due to the work load the game placed on it. Power consumption dependant on what is being run.

I would not rule out overheating unless you know your processor is within normal operating temperature. It could be something silly like the assembly used for software rendering the graphics generates more heat than usual instructions. It is hard to tell however your symptom of it lagging behind the server in a variable way up to 5 minutes does sound like the processor is running slow and unable to keep up with the server.

QuoteYou sound like it is a constant delay in your case- is it correct?
I do not have delay more than I specify in the client frame behind and server frame behind configuration. The only time I have experienced longer than normal delay was when my computer froze for a period of time to restart hard disk (I use mechanical drives) or due to changing underground views (for some reason doing that is demanding in experimental), however in both cases it quickly was eliminated when my client caught up.

Just an after thought...
I did encounter something similar with an very old game (Empire Earth) on a very old computer. It kept falling further and further behind all other clients until I was literally seeing results >10 minutes after they happened. I never did find the cause but I believe it was related to AGP video cards.

Junna

Quote from: killwater on October 07, 2014, 08:39:22 AM
You sound like it is a constant delay in your case- is it correct?

Indeed.

Said laptop does no longer have an operational battery (it is constantly plugged in), but it does have some overheating issues (being a laptop and quite old and worn).

killwater

After some testing it looks as it can be CPU throttling issue as DrSuperGood said. It looks that problems start to appear when there is another CPU consuming process in the background.

jamespetts

Thank you for reporting the nature of the problem: it does help to know the issue.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.