News:

The Forum Rules and Guidelines
Our forum has Rules and Guidelines. Please, be kind and read them ;).

[ENDED GAME] Bridgewater-Brunel no. 1 - Great Britain sized map (no. 2)

Started by jamespetts, December 05, 2017, 09:03:46 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

DrSuperGood

22:41 @ 02/01/2018 GMT and the server is only achieving 0.3 MB/sec upload.

jamespetts

Quote from: DrSuperGood on January 02, 2018, 10:42:24 PM
22:41 @ 02/01/2018 GMT and the server is only achieving 0.3 MB/sec upload.

Thank you for the note. This is certainly not consistent with the issue relating to the upload speed specifically being the compiling of the nightly builds, although can you confirm that this is actually causing the precise same symptoms as you earlier reported (viz. the long lag between command and action)? I want to understand whether this is the same or a different issue.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood

Quotealthough can you confirm that this is actually causing the precise same symptoms as you earlier reported (viz. the long lag between command and action)?
No I cannot as this was being observed from my automated update program downloading the latest nightly zip, which has achieved speeds up to 10 MB/sec doing the same operation in the past.

jamespetts

Quote from: DrSuperGood on January 03, 2018, 01:15:50 AM
No I cannot as this was being observed from my automated update program downloading the latest nightly zip, which has achieved speeds up to 10 MB/sec doing the same operation in the past.

Ahh, I see, thank you for the clarification. This may or may not be a separate issue in that case. I do notice that my server statistics do record multiple occasions of the server using all of its bandwidth, so it may well be that your Java downloader might well help with this.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood

#109
QuoteAs the game develops, it will become more demanding on computational resources. Slower computers might not be able to keep up
My i7 920 cannot keep up with the server anymore unless zoomed in very close to the map, hence why I was reporting lag as I was falling further and further behind. It is basically reaching the stage I cannot play anymore. The game only uses ~20% of my processor but it must not be fast enough for the main thread.

jamespetts

#110
Is this something that has changed since the latest nightly build, or has this been accumulating for some time? Also, can you check your per core CPU usage?

Edit: Further, can I ask what your reported figure for frames per second is?

Edit 2: I have now committed some optimisations to the most CPU intensive part of the game, the passenger generation algorithm.

Edit 3: I have restarted the server with the optimisation; I had to revert one of the optimisations as it appeared to be causing thread deadlocks (there are some issues with "thread_local" - I may try to implement it without "thread_local").

The server is set to use a framerate of 30fps. With my i7 950 (overclocked), I can keep up when zoomed in, but not when zoomed out. When I zoom out, my framerate drops to ~28fps, and so the game drops behind. If I zoom all the way in, the game catches up by running at 40-50fps until it catches up to the server, and the lag disappears.

The graphics system is really very inefficient, but I do not have the expertise to optimise it; I know that an attempt to make it hardware accelerated failed a few years ago because doing it in a way that does not actually make the CPU use more effort would require a fundamental rewrite. I do not know how significant that features such as transparency and the animated sea are for these purposes.

A short-term solution might be simply to reduce the server's target framerate, but that does make everybody's game much less smooth and pleasant to play. I wonder whether increasing the server's setting for the number of sync steps per step might also assist?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood

QuoteIs this something that has changed since the latest nightly build, or has this been accumulating for some time?
Made worse by recent nightly. Obviously because the server was running slow before as it was a debug build which you have now fixed. Before both it and I were slow so the difference was low. Now the difference builds up fast.
QuoteAlso, can you check your per core CPU usage?
Meaningless as modern OS load balancing means all cores are used equally. This is to evenly distribute heat across the surface of the chip so as to allow clock boost technology to operate efficiently. A bottlenecked/always running single thread will appear as using all 4 cores 25% each, and running on none of the hyperthreading cores.

QuoteEdit: Further, can I ask what your reported figure for frames per second is?
FPS varies from <20 when zoomed out to slightly >40 when zoomed really close in into the middle of nowhere on land. Sync step frames get as low as 1.8 zoomed out to reaching 4.2 when zoomed in very close to the middle of nowhere on land. Middle of nowhere on land is higher performance than middle of nowhere on water as water is animated and forces a full frame redraw. Performance is slower on land in cities than in middle of nowhere.

QuoteThe server is set to use a framerate of 30fps. With my i7 950 (overclocked), I can keep up when zoomed in, but not when zoomed out. When I zoom out, my framerate drops to ~28fps, and so the game drops behind. If I zoom all the way in, the game catches up by running at 40-50fps until it catches up to the server, and the lag disappears.
Equally well if I ran on a modern generation i7 at 4GHz I would always be able to catch up as I would be a lot faster. An i7 920 not overclocked is slower than an overclocked i7 950.

For the short term the only viable solution might be to add a dynamic frame rate system to the server, similar to what StarCraft II uses. In StarCraft II if a client cannot keep up with all other clients it drops the game speed (ticks per second) so they can. This would mean that when I play both the server and all clients slow down so my CPU can keep up.

jamespetts

#112
Quote from: DrSuperGood on January 11, 2018, 07:44:36 PM
For the short term the only viable solution might be to add a dynamic frame rate system to the server, similar to what StarCraft II uses. In StarCraft II if a client cannot keep up with all other clients it drops the game speed (ticks per second) so they can. This would mean that when I play both the server and all clients slow down so my CPU can keep up.

That is a very interesting idea. Would you have any idea how one might code this?

In the meantime, I am trying again with a different version of the optimisation abandoned earlier; but the solution in the interim might well be simply to reduce the target framerate for the server.

May I ask what framerate that you get at a moderate zoom setting?

Edit: I have now pushed the revised optimisation, which now works correctly. However, I still get only ~26fps zoomed all the way out. Would it be a good idea if I were to decrease the target FPS on the server to 15?

Edit 2: Testing has determined that the sea is a significant part of the issue here: when I am zoomed all the way out but have no water on my screen, I can keep up with the server at ~30fps; when there is water on my screen, however, I fall behind. I should note that this is using my 1440p monitor. With my 4k monitor, I get ~26fps even over land zoomed all the way out, and ~16fps over water. I suspect that the animation of the sea is part of the problem here - the graphics engine really is horribly inefficient.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood

If the game is running at 30 sync steps per second then that is clearly the problem... Lowering it to 15 will probably solve it at least for now.

Water tiles have to be redrawn each frame so cause an entire screen redraw. Land tiles do not hence save on fillrate. Fillrate is tied to CPU time and memory bandwidth.

TurfIt

Quote from: jamespetts on January 11, 2018, 07:51:54 PM
the graphicsSimEx simulation engine really is horribly inefficient.
FTFY.

Given the disparagement expressed above, I undertook some el quickie timing tests with the save as downloaded from the server.
Some weird stuff since last time I compared backends - not sure if SimEx specific, or standard afflicted too.... will need to look.

opengl backend just crashes on startup. not maintained - might as well just delete it...
GDI - used to be ~5% slower than SDL. Now 75-100% slower! wow. Time to get rid of this one too.
SDL - still the best. Unless you have a Mac, or run certain window managers in Linux, or want to display non latin character languages, then no go. But English on Windows, timings can't be beat.
SDL2 - used to also be ~5% slower than SDL, Now 12-44%. But, I can reduce that 44% which is when all zoomed out down to the same 12% slower at normal zoom by disabling dirty tile updates. But at a cost of 27% slower at normal zoom. hmmm.

Other performance enhancers:
  Disable water animation - 10% faster, and much less nausea.
  Set the correct #threads for my CPU, 12%. (WTF is the default at 6? Very few hexacore CPUs still. Engaging HT for the display threading just slows things down. And triple WTF does the client desync from the server when 'fixing' the thread count????)

And the FTFY:
In SimStandard, largeish games show timings of 20% spent sync_stepping objects, 80% on the display. The server SimEx game is showing 66% objects, 33% display (@1:1 zoom). Hence you're barking up the wrong tree blaming the display for the games performance woes.

Raw data:

GDI ss: 80010, 2388, eye: 0.00, way: 0.00, syn: 7.68, sst: 7.69   fr: 2388, dis: 7.28,  tot: 14.96
SDL ss: 80036, 2386, eye: 0.00, way: 0.00, syn: 7.74, sst: 7.75   fr: 2386, dis: 4.19,  tot: 11.94
SDL2 ss: 80019, 2383, eye: 0.00, way: 0.00, syn: 7.83, sst: 7.83   fr: 2383, dis: 4.72,  tot: 12.55
SDL2 6thread ss: 80026, 2389, eye: 0.00, way: 0.00, syn: 7.97, sst: 7.97   fr: 2389, dis: 5.31,  tot: 13.28
SDL2 8thread ss: 80025, 2385, eye: 0.00, way: 0.00, syn: 8.01, sst: 8.01   fr: 2385, dis: 5.33,  tot: 13.34
SDL2 use_hw ss: 80024, 2387, eye: 0.00, way: 0.00, syn: 7.44, sst: 7.45   fr: 2387, dis: 6.01,  tot: 13.45

GDI MaxZoomOut ss: 70002, 1862, eye: 0.00, way: 0.00, syn: 7.00, sst: 7.01   fr: 1862, dis: 21.36,  tot: 28.36
SDL MaxZoomOut ss: 70005, 2085, eye: 0.00, way: 0.00, syn: 6.96, sst: 6.97   fr: 2085, dis: 10.95,  tot: 17.91
SDL2 MaxZoomOut ss: 70027, 2072, eye: 0.00, way: 0.00, syn: 7.00, sst: 7.00   fr: 2072, dis: 15.85,  tot: 22.85
SDL2 MaxZoomOut 8thread ss: 70001, 2082, eye: 0.00, way: 0.00, syn: 6.65, sst: 6.65   fr: 2082, dis: 16.11,  tot: 22.76
SDL2 MaxZoomOut no water anim ss: 70017, 2087, eye: 0.00, way: 0.00, syn: 6.97, sst: 6.97   fr: 2087, dis: 14.35,  tot: 21.33
SDL2 MaxZoomOut use_hw ss: 70025, 2090, eye: 0.00, way: 0.00, syn: 7.01, sst: 7.01   fr: 2090, dis: 12.35,  tot: 19.36

sst: average ms spend sync_stepping objects, fr: number of frames in run, dis: avg. ms per frame. tot:(al).

DrSuperGood

The entire problem is that the simulation of Simutrans Extended is a lot more resource intensive than Standard. Although one can optimize it here and there, there is no avoiding the facts that more time must be spent on the simulation because the simulation is more complex. The problem is that the simulation is now so complex that the 33% of time spent on display is a considerable bottleneck to game performance.

QuoteSet the correct #threads for my CPU, 12%. (WTF is the default at 6? Very few hexacore CPUs still. Engaging HT for the display threading just slows things down. And triple WTF does the client desync from the server when 'fixing' the thread count?)
I asked the same question... The threads are not just used for graphics, but rather for graphics and updating game state. Threading logic should be such that number of threads is client configurable and the result of an action is independent on the number of worker threads specified for it. However the way it is currently implemented appears that the number of threads does effect the outcome of actions and hence all clients have to use the same number of threads as the server.

HT for display threads should not matter much. Yes it will be slower due to additional synchronization, but it should be mostly trivial. A far larger performance impact would occur having more threads than available execution units due to context switching overhead.

jamespetts

Thank you for that profiling - that is very interesting, especially the consequences of the number of threads. I have now fixed the problem of the number of threads having to be set in simuconf.tab to be identical to that on the server: the game will simply defer to the server's when joining a network game and retain that value until it starts or loads a new game. The reason that the number of threads has to be the same on the client and on the server is that the passenger generation is multi-threaded, and each thread has its own random number generation seed, so that number of threads will affect what actually happens in the game. I am in the process of modifying the threading system so that the passenger generation system (which does not run concurrently with anything substantial in the main thread because it would cause conflicts if it were to do so) uses the number of threads specified in num_threads, but other multi-threaded systems, such as the private car and convoy routing, use 1 fewer thread than the specified num_threads, as these do run concurrently with the main thread in places.

Dr. Supergood is correct about the significance of graphical performance in Extended, incidentally - the much greater map size combined with the much greater depth/complexity of simulation makes the graphics performance far more critical in Extended than in Standard. I did not mean to be critical of those who worked on the current graphics system: it is of its time and I doubt that anything better could sensibly have been written in 1997-1999. If it were possible to recast the engine now to take advantage of modern methods and technology, that would be splendid, although is rather beyond my abilities.

On a map as large as the Bridgewater-Brunel map, most of the time is spent in passenger generation as a result of the large number of alternative destinations that it is necessary for passengers to have, together with the time spent by the game in hashtable lookups for routes on player networks. The performance of the hashtable and weighted vectors (for the buildings, which are accessed when passengers pick somewhere at random to go for each iteration of their destination search) are therefore critical, although I am not sure whether they can be improved.

As to the relationship between threads and execution units, my approach to threading in Simutrans has been to have a bespoke multi-threading model for each subsystem that requires multi-threading, and to have a set pool of threads for each of those subsystems (e.g., passenger generation, convoy routing, etc.). The number of threads in that pool is equal to the number of threads minus one (to take into account the main thread; although, I am now changing that for the passenger generation to make that equal to the number of threads, as that does not run concurrently with anything else of any significance). Some of these multi-threaded subsystems run concurrently with all sync steps (e.g. the path explorer, which uses only one thread, the convoy routing and the private car route finding) and some do not (the route unreserver for railways and the passenger generation). This is all independent from the number of threads allocated for the graphics, of course. Quite how this fits into the relationship between the number of threads and the number of execution units I do not know, but it is certainly considerably better in performance terms than it was when all of these things were single threaded.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood


jamespetts

My apologies: there was some code that I added yesterday that worked on Visual Studio but would not compile on GCC. I have now fixed that and am recompiling on the server now. It should be back up again in a few minutes.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood

Server is unstable. Crashed twice in the last few hours. Only user activity was line maintenance work such as adding convoys, removing convoys, adding new mail lines etc.

jamespetts

There was a bug that is consistent with this description fixed in the latest nightly build.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Rollmaterial


jamespetts

My apologies for this: this was due to a compile error in the Linux builds, which I have now fixed. The server is running again.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

AP

My company seems to have had its password stripped out at some point, and been edited by other players. Are passwords being deliberately removed after a certain time period, or is it still a bug like we had on the canterbury server?

Also, my line "west interconnecter ferry" has a weird issue - ships at dock report an error "too complex" and will not depart. I cannot send them to depot (they can't find one) and I cannot enter their schedule menu to see whats up or change where they are going. I've added extra waypoints to the route in case it' the distance between ports causing an issue but that doesn't seem to have solved anything. Thoughts?

Lastly, it seems changes to the passenger generation or something have affected my company rather badly, but other companies dont look unduly affected? Is there a change that people have made to accomodate the new balancing? I can't immediately see what has been done by other players, all the routes look the same as before. Yet my margin is well negative now. I've tried tweaking passenger demographics for routes, unsure if that will affect anything substantial.

jamespetts

As to the passwords, there is a system that will unlock companies if they have been inactive for a certain period of time (in game years) to allow other players to take over companies of players who have become inactive so as not to clog up the limited saved game slots.

If you think that the issue with the ferry is a bug, I should be grateful if you could post a separate bug report in the usual way.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood

The error message is pretty self explanatory. You made a line that is far too complicated. If it should be considered that is open to debate and might be a bug.

There was a large reduction in passenger generation of ~1/3. This was attributed to stage posts and other unmanned stops no longer inherently generating passenger traffic.

The problem is your network was not very profitable to start with. Where as it only cost companies like mine ~30% of my yearly profit, it caused you to flip into negative numbers.

AP

Quote from: DrSuperGood on January 20, 2018, 09:42:55 PM
The error message is pretty self explanatory. You made a line that is far too complicated. If it should be considered that is open to debate and might be a bug.
The line has 4 stops and a mirror instruction. It is probably the simplest shipping line I have.

If the profitability issue is a simple as stated I'll have to simply review line provision, that's fine. Just wanted to be sure I wasn't missing some underlying cause / cleverness before I start that task. It does seem other players are generating more profit from very similar lines to mine, I'm unsure what the difference is, convoy spacing or something perhaps. I'm still trying to get the hang of the new passenger dymanics, so wasn't going for optimising before, but I guess now need to.

Ves

I think I noticed on some of your lines that you have quite high waiting times on some lines. Try get that down to just a couple of hours and see what effect that does. That being said, my ships are going in negative revenue currently. It's the road network that keeps me in positive income.

jamespetts

I should note for clarification that the error message about a route being too long/complex refers to the route between any given two points in the schedule, and is not a complaint about the complexity of the schedule.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood

Most of my ships make me excessive amounts of profit, to the point I decided to throw a lot of it away raising (enlarging) an island out of the ocean.


AP

Quote from: DrSuperGood on January 22, 2018, 01:08:08 AM
Most of my ships make me excessive amounts of profit, to the point I decided to throw a lot of it away raising (enlarging) an island out of the ocean.
I noticed this, but couldn't figure out how you were doing it. I assumed it was a bug operating in your favour... ;-)

DrSuperGood

QuoteI noticed this, but couldn't figure out how you were doing it. I assumed it was a bug operating in your favour...
The bug is called offering efficient services to people at affordable prices. Unlike yours which often have overcrowding or cause other player's stops to overcrowd.

Old ships are slow, so really long distance routes do not work.

jamespetts

Quote from: Rollmaterial on January 22, 2018, 07:01:08 PM
Server is down.

It seems to be running as far as I can tell, and two people appear to be connected.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.


jamespetts

I should note that the server is temporarily offline for a memory upgrade.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

The server is now back online following the upgrade; hopefully, players will be able to notice improved performance and reliability.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.


jamespetts

#138
The game seems to have been terminated by the server because it was taking too much memory, and this appears to have occurred during a load/save cycle, with the consequence that the saved game had become corrupted, in turn meaning that the automatic re-start failed because it would always try to load a corrupted saved game and quit with an error.

I have now restarted the server from the latest backup, but the out of memory issue is very likely to be due to a memory leak, since I have recently upgraded the server's memory from 4Gb to 6Gb. I have spent some time investigating a possible memory leak, but have so far been able to make very little progress in finding the cause.

Edit: I think that I have now fixed the memory leak, which was related to pedestrians and private cars.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

DrSuperGood

Could players please stop unloading all their mail at my (slot 4 Far East Company) stops that are servicing single towns for local pickup. You can even unload to my hubs directly (saves you on upkeep and hops) as long as you keep the mail waiting for your lines at it under 500 in total. If you cannot promise such limits then rather unload all that mail in your own stop nearby one of my hubs and then use a local shuttle service to move it across to my hub, letting you store as much mail are your lines require.

What some of you are doing is like gathering together 500 vans filled with mail and then dumping it in some local village post office to be delivered. Where as my hubs are like large post office centres handling thousands of vans per hour, the local post offices are not. If the game had any physics at all the poor post office would literally explode with all that mail!

Unless one does this I cannot garuntee that return mail is delivered to your networks at all, which might negatively effect profitability.