News:

Simutrans Forum Archive
A complete record of the old Simutrans Forum.

poor game flow in large maps

Started by Colin, September 01, 2009, 10:50:53 AM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

Colin

I've got a game going on a 1024x1024 map  with several large towns and 8 fast trains, (4 Northbound 4 Southbound) Multiple trams running in all full growth towns (15000 or over). The game has become extremely laggy with no seeming cause. It can happen at the start of a month, two days in or halfway. All not needed processes are shut down and the graphics card is running at 'Turbo'.

The same game played in Experimental suffers no lagginess at all.

Maybe a look should be taken at Knightlys 'New Path Search' to see if it can be incorporated into Standard? Or maybe a new type of search path could be written to suit Standard?
I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

Isaac Eiland-Hall

I suspect this is not a pak64-specific issue...

Meanwhile, have you compared the performance of the GDI executable vs. SDL?

Colin

Quote from: Isaac.Eiland-Hall on September 01, 2009, 11:08:30 AM
I suspect this is not a pak64-specific issue...

Meanwhile, have you compared the performance of the GDI executable vs. SDL?

Yes I have, It doesn't make any noticable difference. I haven't tried it with PAK128 as I only have PAK128 Britain which is a EXP game. I would try PAK128 but it takes weeks to get to the stage where I have PAK64.  But you could be right it may be a Simutrans-Standard problem.
I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

gerw


Colin

I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

whoami

#5
(Moving to Development...)

Direct link to the savegame is http://simutrans-germany.com/files/upload/Colin%27s%20Standard%202611.sve; (EDIT:) it is loadable with plain pak64/r175.

knightly

I have just tried Colin's save game.

There was a brief period of reduced responsiveness immediately after loading, but soon afterwards the game ran smoothly. When I tried to add a halt or adjust certain convoy's schedule, responsiveness was reduced again shortly afterwards for quite some time.

So, probably the reduced responsiveness is caused by the re-routing of all existing ware packets in one single world step after set_schedule_counter() is triggered by actions like halt construction/deletion or schedule changes.

gerw

Quote from: Knightly on September 02, 2009, 08:57:08 AM
There was a brief period of reduced responsiveness immediately after loading, but soon afterwards the game ran smoothly. When I tried to add a halt or adjust certain convoy's schedule, responsiveness was reduced again shortly afterwards for quite some time.

So, probably the reduced responsiveness is caused by the re-routing of all existing ware packets in one single world step after set_schedule_counter() is triggered by actions like halt construction/deletion or schedule changes.
I've noticed the same, while trying Colin's game. So it's the rerouting of goods, which makes the game laggy...

Colin

Quote from: Knightly on September 02, 2009, 08:57:08 AM
I have just tried Colin's save game.

There was a brief period of reduced responsiveness immediately after loading, but soon afterwards the game ran smoothly. When I tried to add a halt or adjust certain convoy's schedule, responsiveness was reduced again shortly afterwards for quite some time.


Reduced by quite a lot and when building a game, because one is continually adding halts or new schedules, it happens with repeated monotony. Every few game days actually, and not counting Auto Save which I've extended to every 3 months. Brave little Aussie fellow that I am. ;D ;D ;D
I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

knightly

Quote from: gerw on September 02, 2009, 09:04:10 AM
I've noticed the same, while trying Colin's game. So it's the rerouting of goods, which makes the game laggy...

Gerw, is it possible to spread the load of re-routing ware packets across a few steps?

Quote from: Colin on September 02, 2009, 09:06:30 AM
Reduced by quite a lot and when building a game, because one is continually adding halts or new schedules, it happens with repeated monotony. Every few game days actually, and not counting Auto Save which I've extended to every 3 months. Brave little Aussie fellow that I am. ;D ;D ;D

Colin, I understand how poor the responsiveness would be when you continue to add/remove halts or adjust schedules, because I have a similar PC specifications as yours ;) The game almost freezes during such global re-routing.


gerw

Quote from: Knightly on September 02, 2009, 09:13:59 AM
Gerw, is it possible to spread the load of re-routing ware packets across a few steps?
It's currently spread across 32 steps, see spieler_t::step. But maybe this number should depend on the time, the rerouting needs.

knightly

Quote from: gerw on September 02, 2009, 10:47:49 AM
It's currently spread across 32 steps, see spieler_t::step. But maybe this number should depend on the time, the rerouting needs.

Thanks a lot for pointing out. :) It is quite incredible that after spreading re-routing across 32 steps, the game is still so unresponsive.

gerw

Quote from: Knightly on September 02, 2009, 11:34:41 AM
Thanks a lot for pointing out. :) It is quite incredible that after spreading re-routing across 32 steps, the game is still so unresponsive.
But one can think of a worst case, where all the 'expensive' stations (with many waiting goods) are rerouted in one step...

z9999

At least, there wasn't this problem in 99.17.1.
But 100.0 and after versions have this problem.

I heard many people saying 100.0 and after versions are too heavy.

whoami

#14
My observations with Colin's savegame:
1) After loading of the savegame, and also after deployment of a new schedule or a new goods category for an existing schedule, ST becomes unresponsive for some seconds, with down to 2(!) fps and 100% cpu/core utilization. => ST seems to recalculate routing for all goods packets, not just for the goods categories served by the new schedule/convoy. This should be fixable by filtering before re-routing.
2) The previous observation especially applies to the change of routing in an isolated network, which seems to still lead to re-calculation of routing on the whole map. For isolated freight lines/networks, it may be useful to limit the refresh, e.g. by walking through the graph of stations *and* connected convoys, also filtering(!) by goods category (see above), marking the stations (also convoys due to their contents?) for refresh, and refreshing only the marked ones in the next loop (removing the marker here again).
1)+2) could be changed as described (or by other means), but a large, fully interconnected P+M network will still not benefit directly from such changes, only indirectly by optimized routing for certain other cases. Further spreading of re-routing calculation may help.
3) side-note: station coverage setting in the savegame is 3 (=> 49 tiles to check at P+M+G generation), which doesn't matter for re-routing, but the additional calculation effort for newly generated passengers+mail increases overall load.

Colin

#15
I have just read whoami's post and it seems to me that he has hit the nail on the head, as they say. If his suggestions can be implemented we should see a, not vast, but quite substantial in crease in game flow. (Incidentally my FPS went down to 1)

I could set the station coverage back to 2 but surely that would lead to an increase in Trams/Tram lines/Stations, creating even more route finding?

I have question re this particular game.

modnote by whoami: this part is now discussed in http://forum.simutrans.com/index.php?topic=3203.msg31352#msg31352:
I have a passenger train (Not shown in the uploaded game)  Which travels between two stations only. It leaves one station with over 3000 pasengers and arrives at it's destination where they all disembark. It makes a loss of over 41 thousand (not in travel costs) when the passengers leave. It returns home with over 300 passengers and makes a profit of over 22k when the passengers leave. Have I got a setting in wrong in the config.tab or is this another bug?
I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

whoami

#16
As I wrote, my suggestions (if they matter at all) will not help (much) in the case of the passenger/mail network.

Quote from: Colin on September 03, 2009, 08:29:17 AM
I could set the station coverage back to 2 but surely that would lead to an increase in Trams/Tram lines/Stations, creating even more route finding?
Directly, it will only increase routing calculation effort to a smaller extent, because of the larger number of stations ("hops"). But usually, you will have more different lines, more interconnection points and more transfers, and that makes a big difference.

QuoteIt makes a loss of over 41 thousand (not in travel costs) when the passengers leave. It returns home with over 300 passengers and makes a profit of over 22k when the passengers leave. Have I got a setting in wrong in the config.tab or is this another bug?
(I assume that you don't refer to ST-experimental.)
pay_for_total_distance = 2 can lead to such a result.
Also, cost deduction at switching of the month might be involved.

Colin

#17
Quote from: whoami on September 03, 2009, 11:22:18 AM
As I wrote, my suggestions (if they matter at all) will not help (much) in the case of the passenger/mail network.
Directly, it will only increase routing calculation effort to a smaller extent, because of the larger number of stations ("hops"). But usually, you will have more different lines, more interconnection points and more transfers, and that makes a big difference.

Do you mean that if I make multiple smaller Tram networks in my cities instead of one large network it would improve the flow of the game?

In Experimental the train made $721 profit on the outward journey with 3078 passengers as apposed to a $41k loss in Standard.

Additional info re-train behaviour.

Train travels A-B-A.
From A to B
journey 1. 3078 passengers, loss on arrival at B -$41k plus.
Journey 2. 3078 passengers, loss on arrival at B -$42k plus.
Journey 3. 3078 passengers, loss on arrival at B -$43k plus.
From B to A
journey 1. 325 Passengers, profit on arrival at A $22k plus.
Journey 2. 476 Passengers, profit on arrival at A $16k plus.
Journey 3. 124 passengers, profit on arrival at A $4k plus.

This doesn't make sense and it doesn't matter if I set the pay_for_total_distance to 0, 1, or 2. I realise that these settings may not affect a current SVE game. But it doesn't detract from the nonsensical profit and loss that is being generated.
I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

whoami

Quote from: Colin on September 03, 2009, 06:11:00 PM
Do you mean that if I make multiple smaller Tram networks in my cities instead of one large network it would improve the flow of the game?
More different lines means more schedules and transfers to check, so goods routing calculation will need more CPU time. From this aspect, a larger station coverage can support in reducing calculation effort.
The performance-wise effect of the station coverage setting depends, however, also on your network layout: The more tiles with houses are covered by several stations, the slower the game may become, because a generated passenger will check more than one station for suitable connections. But the developers can provide a better, substantiated analysis on this complex problem.

As your savegame doesn't contain an extremely large network, there may be a performance bug. Additionally, Z9999 may be right when he said that ST has become "too heavy".

Regarding the obscure revenue: It's unrelated to the program performance problems, so could you please open a new topic for that, also providing the savegame with the train? After that, I will move my own replies and any remaining related text there.

Colin

Quote from: whoami on September 04, 2009, 07:39:38 PM
As your savegame doesn't contain an extremely large network, there may be a performance bug. Additionally, Z9999 may be right when he said that ST has become "too heavy".

Oh dear! I thought my SVE contained a fairly massive network. I must be really naive.

Quote from: whoami on September 04, 2009, 07:39:38 PM
Regarding the obscure revenue: It's unrelated to the program performance problems, so could you please open a new topic for that, also providing the savegame with the train? After that, I will move my own replies and any remaining related text there.

Ok I'll upload my SVE with the latest additions and then open a new topic.
I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

prissi

The station coverage is not important.

The problem in your savegame are not the size of the map but the shear amount of waiting persons. You have several stations with more than 300000 people waiting. Updating such a monster will of course take some time.

I will look into the code to make it smoother, although no promises ...

Colin

Quote from: prissi on September 05, 2009, 11:38:36 AM
The station coverage is not important.

The problem in your savegame are not the size of the map but the shear amount of waiting persons. You have several stations with more than 300000 people waiting. Updating such a monster will of course take some time.

I will look into the code to make it smoother, although no promises ...

Thank You Prissi, I will look forward to any improvements you can make.

In my next game (which may be awhile away due to this one being nowhere near completed) I will reduce the Passenger Level.
I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

Colin

@ Prissi,

Does nightly 2640 contain your changes?
I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

knightly

Hi Colin :)

According to the source code update history, Prissi's changes for speeding up re-routing have already been incorporated. So, does it improve the responsiveness of your game?

Knightly

Colin

Quote from: Knightly on September 07, 2009, 11:36:01 PM
Hi Colin :)

According to the source code update history, Prissi's changes for speeding up re-routing have already been incorporated. So, does it improve the responsiveness of your game?

Knightly


No, if anything it's getting worse. I'm sorry to say.
I may not agree with what you say, but I will defend to the death your right to say it

Thought for the day

When you are up to your backside in alligators, it is difficult to remind yourself that your initial objective was to drain the swamp.

z9999

Quote from: Colin on September 08, 2009, 04:07:40 AM
No, if anything it's getting worse. I'm sorry to say.

Same here.
What we want is not to solve jumping convoi, but to solve unresponsive.
In recent simutrans, idle time is zero during long time even though convoi keep moving well.

knightly

#26
Quote
void spieler_t::step()
{
   if(  halt_list.get_count()>0  ) {

      uint32 it = halt_iterator_start;
      slist_iterator_tpl <halthandle_t> iter( halt_list );
      while(  it>0  &&  iter.next()  ) {
         it--;
      }
      if(  it>0  ) {
         halt_iterator_start = 0;
      }
      else {
         uint32 units_handled = 0;
         while(  units_handled<8192  ) {
            if(  !iter.next()  ) {
               halt_iterator_start = 0;
               break;
            }
            halt_iterator_start ++;
            // iterator until 8192 passengers were handled
            units_handled += iter.get_current()->sum_all_waiting_goods();
            iter.get_current()->step();
         }
         INT_CHECK("simplay 156");
      }
   }
}

After looking at Prissi's changes, I think one of the reasons why the unresponsiveness problem has aggravated lies in the fact that >= 8192 ware (not ware packets) are re-routed in each step. For hub stations with large-sized ware packets (i.e. large menge), this will indeed help to prevent too much re-routing work in one step; but with non-hub halts, they usually have only small-sized ware packets (please see the attached image), so it needs to process many non-hub halts before the limit of 8192 can be reached.

IMHO, re-routing workload depends not on packet size (menge), but on packet count. More packets, more re-routing work, and vice versa. So, I would suggest to change from counting ware menge to counting the number of packets processed. Of course, the limit of 8192 also needs to be reduced accordingly.


prissi

The problem in Colin save are that there are several stations with 300000 passengers. Updating (rerouting) them takes nearly a second on my computer. During this no user interaction is possible, since such can be only done outside a step. The only other way would be that the stations themselves only handle 8192 each step, and indicate the result then. But this would require more complex logik, as changes of the passenger numbers must be also taken into account (i.e. something like a dirty flag for ware in stations.) I will look into this next week.

knightly

#28
Edit : Post removed as the provided screenshot was wrong.

HeinBloed

Knightly,

I really only stumbled accross this thread randomly, but looking at your screenshot, I strongly suspect that you fell victim to a display bug: the sum of passengers of the packages displayed is never 300k, not even close. I guess lots of destinations with letter >B are missing.
(previously known as "tttron")

prissi

Display works only with integer coordintes with 16bit; therefore I think the scrollbar just messed up. It is very easy to time those stations and they have like 4000-30000 packets and cn take up to 800ms or rerouting (on this computer here).

knightly

Prissi,

Really sorry! I didn't realise that the list is incomplete. I loaded Colin's save game again and looked at the same station, and the list kept growing gradually when I tried to scroll down.

Nevertheless, I think it is more logical to count the number of packets instead of the sum of menge. Hope you can consider this suggestion when you look at the code later.

Knightly



Dwachs

[offtopic]
this:

Quote from: tttron on September 09, 2009, 12:23:14 AM
I strongly suspect that you fell victim to a display bug: the sum of passengers of the packages displayed is never 300k, not even close. I guess lots of destinations with letter >B are missing.

should be fixed in revision 2642.
[/offtopic]
Parsley, sage, rosemary, and maggikraut.

gerw

In the current implementation, if the goods of a station are rerouted, simutrans does a BFS for each packet. Since no heuristic is used, it is possible to do this in only one BFS, until the target for each packet is found.

I will try to implement this, if there is any free time next week...

prissi

Please check again with curretn nightly. Should be much smoother (only lagging on my eeePC).