News:

Simutrans Forum Archive
A complete record of the old Simutrans Forum.

bridgewater-brunel.me.uk - Simutrans-Experimental - Pak128.Britain-Ex 0.9.1

Started by jamespetts, January 26, 2014, 01:35:08 AM

Previous topic - Next topic

0 Members and 4 Guests are viewing this topic.

asaphxiix

the command to use tcpdump would be 'tcpdump -i eth0 -w filename.cap' (you can also use filters to reduce the size of the file)

I also experience lag, though only sometimes, intermittently and it passes. I have quite a strong i5 and lots of RAM, and a stable wired connection with 77 ms to the server.

jamespetts

Quote from: TurfIt on February 09, 2014, 02:12:19 AM
I just repeated the test locally with your server config. No timing issues encountered, as expected since the values aren't that far off default.

Hmm - very curious. I don't suppose that it could have anything to do with the client being multi-threaded and the server single threaded, could it?

QuoteWhy change server_frames_ahead? The default should be fine, or even reduced. You're adding an extra 170ms latency to every client, even if they don't need it. If clients are ending up with 'execute in past' desyncs, they should adjust additional_client_frames_behind instead. That way clients with good connections aren't penalized.

I had adjusted this some time ago, having found that clients were disconnecting often with the previous setting. Do you notice an unacceptable degree of latency? I could try to adjust this downwards if you think that it would help.

Quoteserver_frames_between_checks = 256 is the default, but IMHO way too long. At 12 fps that's 21s between checks, a client could run way ahead in that time... I would drop that to 24-48 frames. A time check packet is a mere 98 bytes, times the number of connected clients, even once second, is still a bandwidth pittance.

Ahh, interesting - would increasing this value be likely to reduce desyncs due to the client running ahead?

QuoteI presume you changed server_frames_per_step to reduce CPU load? A setting of 5 and 12 fps is equivalent to 2.4 simloops. Offline the simloops target is 5.0. 2.4 isn't horrible, but things would be more responsive if you put frames_per_step back to 3 or 4. At the cost of extra CPU load of course.

This was indeed to reduce CPU load: the current game is in the fairly early stages, and the load is not too intense. I found that a well developed game in the 20th century on a map of this size could have really very high CPU usage that could cause desyncs due to the server running behind the client (presumably) and unresponsiveness without this setting being lowered. That being said, there have been one or two changes to make the code more efficient since then, but I do not know whether that will be enough to allow a higher setting of this value.

QuoteFor checking packets, try Wireshark or Tshark, or even good 'ole tcpdump would do.

Hmm - I am not familiar with these tools: do I need to restart the executable and attach one of these to the process somehow, as with gdb, or are they independent packet sniffers of some sort?

Thank you very much for your investigations here - most helpful.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

AP

Something seems very off with the city growth, we have vast urban sprawl with little or no urban transport. Either that or the cities started off far too close together for the growth settings.

By the time we get to rail, there are going to be no corridors left between towns.

Am also concerned that exclusively-passenger-services via canal seems to be profitable. Canals were mainly built for freight.

--

On an unrelated matter. The maintenance costs on the tool buttons in brackets e.g. Bridleway (1.6), what time base is that over? And how many of those units in a game-month? E.g. if I build a bridleway 10 tiles long, I should be able to calculate monthly maintenance ahead of time somehow and relate it to operating profit.

jamespetts

The high rate of city growth is a known issue: the maximum rate should be set to be a percentage of the total size. In Standard, and much earlier versions of Experimental, town growth would not occur without player transport. In more recent versions of Experimental, passengers can get to their destinations by walking alone, and this prompts town growth, whose rate is too high (the current system was not designed for a very long game). I tried to adjust this downwards for the pakset release in simuconf.tab, but evidently did not adjust it enough. Town growth will be looked at for the next major release.

As to the profitability of canal services, there were indeed passenger services by canal, so this should not automatically be unprofitable. However, the construction and maintenance costs of the ship canals necessary to support the Thames sailing barges currently used on the canals should be such that using those canals only or mainly for passengers should not be profitable. I should note that the pakset is not balanced yet.

As to the maintenance costs, these are monthly.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

asaphxiix

Quote from: jamespetts on February 09, 2014, 11:26:39 AM

Hmm - I am not familiar with these tools: do I need to restart the executable and attach one of these to the process somehow, as with gdb, or are they independent packet sniffers of some sort?


This is a separate process, you run alongside the game executable (as it transmits and receives packets:)

the command to use tcpdump would be 'tcpdump -i eth0 -w filename.cap' (you can also use filters to reduce the size of the file)

where filename.cap is a file you can then open on your computer using wireshark (which I believe is free), and allows you to analyze the packets transmitted or received by the server during the running of tcpdump.

"eth0" might be replaced with the interface name through which the default route goes. route -n show show all the routes, then use the interface name stated in the route of 0.0.0.0 (that's the default route).

I think it can only be run under root. Of course I'll gladly help doing this, but I'll need the root password for that (you can always change it later :)

TurfIt

Quote from: jamespetts on February 09, 2014, 11:26:39 AM
Hmm - very curious. I don't suppose that it could have anything to do with the client being multi-threaded and the server single threaded, could it?
Unlikely. Unless there's been extra multithreading code added to Experimental over that in Standard which is not yet multithreading any of the simulation related stuff. Experimental appears to still be missing the multithreaded drawing that was added to Standard last year, would be a big help to those running higher resolutions and with multi core processors - frees up a lot of CPU time to spend on the simulation instead of drawing...


Quote from: jamespetts on February 09, 2014, 11:26:39 AM
I had adjusted this some time ago, having found that clients were disconnecting often with the previous setting. Do you notice an unacceptable degree of latency? I could try to adjust this downwards if you think that it would help.
I do find the latency on this server more noticeable than others, bordering the annoying actually. From what I can tell, the server has a solid internet connection. I'm at the mercy of the usually unstable trans-atlantic links, and still am getting a rock steady 112 +/- 1ms ping. Hence if clients are discoing, they need to fix their connection and/or adjust their client settings for more margin (assuming of course we find/fix this timing anomaly I've found - nobody experiencing this will stay connected long when others are online and building things). Changing the server settings needlessly penalizes those with good connections.



Quote from: jamespetts on February 09, 2014, 11:26:39 AM
Ahh, interesting - would increasing this value be likely to reduce desyncs due to the client running ahead?
Decreasing would be an idea. I thought we'd discussed this before... yes we did - see Simutrans-Experimental - Pak128.Britain-Ex 0.8.4 server perfomance .


Quote from: jamespetts on February 09, 2014, 11:26:39 AM
This was indeed to reduce CPU load: the current game is in the fairly early stages, and the load is not too intense.
I'd be tempted to put it back to the default 4, atleast once past the crazy boat era in the game. IIRC that was the peak CPU usage during the last game. Would help the responsiveness getting the simloops back up a bit. Ideally this setting and all the others would be dynamically adjusted - another item on the long term todo list - and as usual if somebody else were to get there first...  ;)


Quote from: jamespetts on February 09, 2014, 11:26:39 AM
Hmm - I am not familiar with these tools: do I need to restart the executable and attach one of these to the process somehow, as with gdb, or are they independent packet sniffers of some sort?
asaphxiix has provided good info. If you have GUI access to your server, Wireshark might be the easiest for you to use. Ideally you'd do the capture when you're the only client connected, and your client is idle (not sending commands). Otherwise filter the capture to just the traffic to a single client. Even better capture the server sent packets, client received packets, and client log all at once.

First though, I'd like to see a log from anybody else showing the NWC_CHECKs - especially the 5 frame progression that should not be there. Mine run normally for minutes, then receives a burst of NWC_CHECKs, then back to normal for a while. Every burst throws my client timing way off, and if anybody else is online doing commands - disconnect.


TurfIt

Run your client with '-debug 3 -log' command line args.
simu.log will be created which is the log.

asaphxiix

plenty of those. the time difference initially grows up to 8000, then I think the pause ends, and there are a bunch of

Warning: karte_t::interactive:   sync_step=1359430

and then the NWC_CHECK starts slowly to decrease to zero, going negative, reaching -354 even, and then back to up to 8000, and dwindles in that range. so it's quite unstable, but this could be normal and seems to react positively to the sync_steps.

Log: http://www.filedropper.com/simu-copy

TurfIt

See my log in post #137. I'm looking for somebody else experiencing a burst of NWC_CHECK messages in a row, and then followed by the interactive messages with sync_step= counting up by 5. Or any other anomalies. Each NWC_CHECK should be immediately followed by an interactive message with sync_step counting up by 256 (for the current server settings).

EDIT: you edited to post your log while I was posting. It does indeed contain the problem I'm seeing. Now for James to obtain the server side of things...

asaphxiix

Yes I have those too I believe. You can see also in the file I posted last message.

Message: NWC_CHECK:   time difference to server 7476
Warning: karte_t::interactive:   sync_step=1365615  server=[rand=1827315087 halt=1 line=1 cnvy=2049 ind_dns_prop=3816 act_ind_dens=7022 traffic=2351262] client=[rand=1827315087 halt=1 line=1 cnvy=2049 ind_dns_prop=3816 act_ind_dens=7022 traffic=2351262]
Message: network_command_t::rdwr:   read packet_id=9, client_id=0
Warning: network_check_activity():   received cmd id=9 nwc_check_t from socket[480]
Message: NWC_CHECK:   time difference to server 7582
Warning: karte_t::interactive:   sync_step=1365620  server=[rand=3168147781 halt=1 line=1 cnvy=2049 ind_dns_prop=3816 act_ind_dens=7022 traffic=2351261] client=[rand=3168147781 halt=1 line=1 cnvy=2049 ind_dns_prop=3816 act_ind_dens=7022 traffic=2351261]
Message: network_command_t::rdwr:   read packet_id=9, client_id=0
Warning: network_check_activity():   received cmd id=9 nwc_check_t from socket[480]
Message: NWC_CHECK:   time difference to server 7790
Message: network_command_t::rdwr:   read packet_id=9, client_id=0
Warning: network_check_activity():   received cmd id=9 nwc_check_t from socket[480]
Message: NWC_CHECK:   time difference to server 7917
Warning: karte_t::interactive:   sync_step=1365625  server=[rand=2912924728 halt=1 line=1 cnvy=2049 ind_dns_prop=3816 act_ind_dens=7022 traffic=2351263] client=[rand=2912924728 halt=1 line=1 cnvy=2049 ind_dns_prop=3816 act_ind_dens=7022 traffic=2351263]
Message: network_command_t::rdwr:   read packet_id=9, client_id=0
Warning: network_check_activity():   received cmd id=9 nwc_check_t from socket[480]
Message: NWC_CHECK:   time difference to server 8116
Warning: karte_t::interactive:   sync_step=1365630  server=[rand=1175234442 halt=1 line=1 cnvy=2049 ind_dns_prop=3816 act_ind_dens=7022 traffic=2351263] client=[rand=1175234442 halt=1 line=1 cnvy=2049 ind_dns_prop=3816 act_ind_dens=7022 traffic=2351263]
Message: network_command_t::rdwr:   read packet_id=9, client_id=0
Warning: network_check_activity():   received cmd id=9 nwc_check_t from socket[480]
Message: NWC_CHECK:   time difference to server 8237
Message: network_command_t::rdwr:   read packet_id=9, client_id=0
Warning: network_check_activity():   received cmd id=9 nwc_check_t from socket[480]
Message: NWC_CHECK:   time difference to server 8449

not sure what you mean by the sync_step counting up to 256, but this seems just like your messages.

jamespetts

Quote from: TurfIt on February 09, 2014, 06:01:03 PM
See my log in post #137. I'm looking for somebody else experiencing a burst of NWC_CHECK messages in a row, and then followed by the interactive messages with sync_step= counting up by 5. Or any other anomalies. Each NWC_CHECK should be immediately followed by an interactive message with sync_step counting up by 256 (for the current server settings).

EDIT: you edited to post your log while I was posting. It does indeed contain the problem I'm seeing. Now for James to obtain the server side of things...

Is it server logs that you are after, or tcpdump statistics? I have command line access to the server, but there is no GUI installed. Should I simply upload the raw output from the command posted above? If so, for how long should I leave tcpdump running?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Sarlock

Quote from: AP on February 09, 2014, 12:53:25 PM
By the time we get to rail, there are going to be no corridors left between towns.

Am also concerned that exclusively-passenger-services via canal seems to be profitable. Canals were mainly built for freight.

Passenger canals aren't really making money right now.  They are making a small profit, if there is enough population density in its catchment area, but when offset against the maintenance cost of the canals it's probably a money loser still.  My monthly maintenance cost is 200,000/month, which is a huge cost, mostly to support my canal infrastructure.  There is no way my canal passenger services are earning 1,000,000+ in revenue to offset their part of that.

The main reason I have set up extensive passenger canal networks is to get ready for rail, since now I already have nicely laid out right-of-ways through the cities (which by then will be well within the urban area-assuming the game isn't reset by then) and serve as a break-even (if I am lucky) way to aggregate passengers to my profitable long-distance connector lines.  A quick guess puts my passenger/mail service revenue at about 5 million/year-probably about 3 million in profits.  The long-distance lines is where the big profits are, but you need to have lots of feeder connections to build up sufficient demand.

I am very surprised that no one on the western islands has set up any significant passenger networks.
Current projects: Pak128 Trees, blender graphics

TurfIt

Quote from: jamespetts on February 09, 2014, 06:14:06 PM
Is it server logs that you are after, or tcpdump statistics? I have command line access to the server, but there is no GUI installed. Should I simply upload the raw output from the command posted above? If so, for how long should I leave tcpdump running?
Packet capture is what I want to see, the server sides logs are not sufficiently detailed - why the heck aren't the logs entries timestamped???
If no gui, you're obviously stuck with command line, try posting the raw tcpdump. I can filter locally. Running for a few minutes should be fine.
WARNING: the capture may contain passwords. Don't be entering any while doing this to play it safe.

And ideally run Wireshark locally on your client at the same time, and post that along with the client simu.log.
WARNING: if capturing on a windows client, windows is very very very very very chatty. Definitely filter to only the comms with the server. Otherwise there'll be all sorts of who knows what you probably don't want posted...

jamespetts

Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

TurfIt

This is the output file created with the -w option? It should be the raw packets, not just this semi decoded print.
Might be usable, but there's so much junk in here, and it's not the correct format for the tools to filter. A nice pcap format file would be nice which is what the -w should give???

jamespetts

Ahh, I had trouble with the -w option. I had the error:


tcpdump: filename.cap: Permission denied


The error was the same whatever file name that I specified.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

asaphxiix

was it run from root? perhaps try to another folder, where there are write permissions.

jamespetts

It was indeed run from root, and I tried a known good folder (the same as I used without the -w option where it worked without difficulty).
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

asaphxiix

that's quite strange. what version and distro is running?

Googling, I see some stuff about a process called AppArmor, i'm guessing you don't have that but better to make sure.

the best lead seems to be trying to create (touch) the file you are writing to and setting full permissions for everyone (chmod 777 filename) on it.

Also try the -Z option?

TurfIt

-p option could be tried too, but I'd expect a different error if setting promiscuous failed. We don't need a promiscuous capture for this.

EDIT:
OK, this capture does indeed appear to show the server sending NWC_CHECKs ~200ms apart. Without the full packet data from the -w file, I can't be certain, but there's definitely bursts of packets being sent that match the timing signature of bursts received by my client.

Not sure where to go from here. Any chance you can try copying your whole simutrans server install to somebody else's server somewhere? Running the same binary elsewhere would point the finger to simutrans or your host/OS environment...

jamespetts

To answer the question - I am using Ubuntu 13.04. Apologies for not having been able to respond more quickly - I have been working on the forge cost feature and latterly making dinner and pudding, taking photographs of the pudding, and uploading said photographs to Flickr.

Perhaps it is time for a distribution upgrade?

Turfit - thank you very much for your analysis. When you ran the server, did you use a Linux or Windows version, command line or graphical, 32- or 64-bit? If other than a Linux command line 64-bit version, I wonder whether the variance in behaviour somehow arises there. If you have a 64-bit Linux environment, I could copy the whole server install (save that I will remove the admin password from simuconf.tab) to a .tar for you?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

TurfIt

Looks more like a cake or something than what I'd call pudding. But then you do have weird names for things over that side of the pond.  ;D

I just ran the graphical x86 windows program compiled by you as the server on an old laptop. I don't have 64 bit linux environment at the moment, but it's easy enough to create one...  a .tgz would be fine.
I'd also still really recommend 32 bit for Simutrans. It really is not designed for 64. But then it's not really designed for MSVC either, doesn't really matter until you're pushing the performance limits, but then maps on this server tend to!

jamespetts

Ahh, we use "pudding" as a general word for what I believe that you call "dessert", although this particular pudding is a close relation of a sponge cake (the distinction being that it is steamed rather than baked, and served hot in a bowl with custard or similar rather than cold on a plate on its own).

As to using 64-bit: I do that because the server is a Linux 64 bit server, and the 32 bit version of the application has been found to be unstable on Linux (but not Windows) running a 64-bit architecture. Also, some of the very largest maps are now coming close to the practical limit of memory available to 32 bit applications.

The server files are here [Edit: now deleted]. That is an enormous file, however: over 800Mb at the maximum possible compression ratio that I could obtain. Please let me know when you have downloaded it so that I can delete it. I have omitted the folder, which is on the server, for the old Pak128.Britain-Ex 0.9.0, and I have also omitted the log files and custom script setups for starting/stopping the game.

I hope that this is sufficient. I should be very interested in your findings. Thank you again for taking an interest in this.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

TurfIt

I have the file. Might be a couple days before I can look into it. I see you included source, presume this is the source you compiled from for the current running server? And simutrans/simutrans-experimental-11.18 is the executable? With .17, 16, 15, etc the previous builds?

EDIT:
Did you mean Ubuntu server 12.04.4 LTS?  or 13.04?   12.04.04 LTS is latest long term, and 13.10 latest otherwise...

jamespetts

I did mean 13.04. The next release of Ubuntu is an LTS, is it not? I was planning on skipping 13.10 and waiting for 14.04.

The source code included is indeed from the current executable version, and the executable actually run is the one simply named "simutrans-experimental", which is a build of 11.18. The source code should be the same as on my Github "master" branch.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

sdog

James, could you ask one of the players on Bridgewater-Brunel to briefly describe a particularily good game on Simutrans. With a focus on those aspects you consider most interesting. As the interest in Carl's project shows, a lot of people are very interested in the complexity that airses from a compact set of rules. The historical development, and how it reflects real transport development could be very interesting as well.

However a large number of people interested are passive, and not joining the server, thus it happens a bit out ouf the view of a wider audience. A compact summary might lead to further interest, and present also to non players what the serious aspects of simutrans are. (That might also help someone who has to provide a good reason for playing such a game.)

TurfIt

OK. Fired up Ubuntu server 64bit 13.04, started the server, connected a windows 32 bit client, and the timing went wonky immediately. Now to find the culprit...

EDIT: was able to duplicate with Standard, but only by making the CPU beg for mercy with the OP test game - pegs at 100%. So, despite your experimental save only consuming ~25% CPU, it's behaving overloaded. I suspect Experimental is doing too much in step(), and hence not calling ::interactive regularly enough. What the OP game does to Standard is massively overload the passenger generation/pathfinding, and hence end up spending most of its time in step().

So, I believe this tracks right back to the fundamental architecture of the program, and therefore could very well be unfixable. That said, I'll try looking into the behaviour of ::interactive when it's not being called regularly, and see if something can be done there. Otherwise, try to shorten up the time Experimental is spending in step()!



Also, when I compiled the Experimental server without DEBUG=3, the client discod immediately on the first checklist check with a mismatch. Both rand= and traffic= were different. I don't know how your windows x86 binary I was using for the client is compiled, but it shouldn't matter. Debug, non-debug, optimized, non-opt, all combinations thereof, shouldn't matter. Client should still stay connected to server, so you've got some other thing going on too...

jamespetts

Sdog - as you will see, I have merged your request with the server thread so that people can see your request directly. Players in this server game: if anyone feels inclined to produce a write-up as Sdog requests, this might be very interesting. Thank you for suggesting this.

TurfIt - thank you very much for investigating this: it is much appreciated. You are correct in deducing that there is indeed more work being done in passenger generation and pathfinding in Experimental than Standard. I do not know a great deal about the details of the way in which karte_t::interactive() works and how it fits in with the networking and timing: I have never really delved into such low-level parts of the programme. May I ask - is this the same thing as calling INT_CHECK(), or is this something else? If so, where is and is not safe to call INT_CHECK? Perhaps adding more calls to INT_CHECK() will help. If this is not the same thing, I should be very grateful if you could explain the difference.

As to a checklist desync when the server is not running with DEBUG defined, that is odd, as whether DEBUG is defined makes no difference in Windows, and synchronisation is maintained although the server is an optimised build with the debug symbols stripped. The definition of DEBUG causing desyncs would tend to suggest an assert with a side effect, but this cannot be so, since it does not make a difference in Windows: I must say, I am quite stumped as to why this might occur in Linux but not Windows.

Thank you again for all of your investigations - they are much appreciated.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

TurfIt

INT_CHECK is only applicable to offline games. Online, the interrupts are disabled as the execution on the server and all clients must be done in lock step. I think I see a couple things that can be done to improve the situation, but I'm out of time tonight to even describe them...

ӔO

The best part about the online game is that you can see what other players do to be successful.

I wouldn't have known of the high speed corner exploit if it were not for the first game.
My Sketchup open project sources
various projects rolled up: http://dl.dropbox.com/u/17111233/Roll_up.rar

Colour safe chart:


sdog

Quote from: jamespetts on February 11, 2014, 11:37:45 PM
Sdog - as you will see, I have merged your request with the server thread so that people can see your request directly. Players in this server game: if anyone feels inclined to produce a write-up as Sdog requests, this might be very interesting. Thank you for suggesting this.

Thank you James.

And thanks in advance to whomever would  write such a history of a game.

To clarify it a bit: I should post it to the blog, in its full length. Then write a synopsis to post on social media, linking to the blog.

jamespetts

Quote from: ӔO on February 12, 2014, 04:09:45 AM
The best part about the online game is that you can see what other players do to be successful.

I wouldn't have known of the high speed corner exploit if it were not for the first game.

This reminds me that I need to fix this for the next major release.

TurfIt - thank you for the clarification. I shall be interested to know your thoughts when you have time to record them. Your input is much appreciated.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

TurfIt

I did try out some changes to client timing. So far so good, but I need to think on possible undesirable implications on the one change, and need to tame down the adjustments a bit as they're currently a bit aggressive. Timing after the initial join is all over the place, but it seems the cause is server side. After 30secs or so I settles right down and runs quite nicely.

I also joined with my modified client to your server, and things were still pretty good, except every now and then your server seems to go to sleep... My client has to slam on the brakes to avoid overrunning, then your server takes off like a Bugatti Veyron leaving my poor client pedaling like mad trying to catch back up. I don't get such extreme pauses running the server locally...

Also, feel free to split all this timing discussion off to a development forum. Leave this thread for gameplay discussion would be good IMHO.