[11.24] Loading deadlock

[jamespetts]

A player in the online game has reported a bug in which vehicles at a stop sometimes fail to load. It is at the time of writing visible with convoy no. 7049. I am writing this here before working out a fix because the problem emanates from TurfIt's system for improving the performance of loading vehicles and I should like to garner input from TurfIt before proceeding.

The fundamental problem appears to be this: the new system checks which lines have already been loaded in any given step. If it is found that vehicle Y from line X has not loaded anything in any given step, then all vehicles from line X are skipped in that step's loading check (assuming that they are at the same point in their schedules) on the assumption that if vehicle Y loaded nothing, then that was because there was nothing to load. However, it is sometimes the case that vehicle Y will load nothing because vehicle Z is already partly full or for other reasons. If it happens that vehicle Z is checked first in each step, then vehicle Z will load nothing and will stop vehicle Y from loading anything because the check will be skipped. This stops either vehicle from loading anything, creating a deadlock.

Any thoughts on how best to overcome this issue would be much appreciated.

[Sarlock]

Interesting.  This is likely happening on my lines but I haven't noticed it due to the sheer volume of ships.

[Sarlock]

Update: I see it now.  Happening quite frequently, actually, just hard to notice.  I was wondering why many of my main stations were overcrowded, I thought it was just from a reluctance to add additional ships to the map, but I see now that many ships are leaving overloaded stations completely or nearly empty when they should have picked up a full load from that station.

[jamespetts]

The user who reported the initial problem has given more details, which I share here, with his leave, in order to assist with finding the problem:


Here's a picture that should help explain things:




1. You have two main hub stations on each side of the map, because the map is so big.  (I actually have more, but this is just a simplication)
2. All vehicles deliver goods from sawmills to a hub (either the west or east hub).
3. At each hub there are also ships waiting to deliver goods from the hub to factories.
4. Between the east and west hub are TWO shipping lines.  This is the key.

4a. There is a shipping line that waits for 100% load at west station and goes to east station  [let's call this "W>E line"]
4b. There is a line that waits for 100% load at east station and goes to west station [let's call this "E>W Line"]
4c. As you can see both lines are EXACTLY the same, except one is supposed to wait at the west station and one is supposed to wait at the east station.

THE PROBLEM:

* Let's say I have 5 empty ships waiting at West station (the "W>E Line").
* Then a bunch of ships deliver goods to West station wanting to go to East station.

What should happen:
* All these newly delivered goods should load onto the empty ships waiting at West station.  They should load onto the "W>E Line" ships that are waiting empty.

What does happen:
* Instead, what sometimes happens is the station gets full and the goods refuse to load onto the empty ships.  They refuse to load onto any "W>E Line" ships.

However, if I delete the "E>W Line" everything starts working, which makes me think the game is incorrectly trying to assign routing to the "E>W Line" and refusing to acknowledge the ships that are there and ready to ship on the "W>E Line"

[TurfIt]

If it is found that vehicle Y from line X has not loaded anything in any given step, then all vehicles from line X are skipped in that step's loading check (assuming that they are at the same point in their schedules) on the

Only other vehicles on line X loading the same goods type would be skipped since the cargo should go into the first vehicle.

assumption that if vehicle Y loaded nothing, then that was because there was nothing to load. However, it is sometimes the case that vehicle Y will load nothing because vehicle Z is already partly full or for other reasons. If it happens that vehicle Z is checked first in each step, then vehicle Z will load nothing and will stop vehicle Y from loading

If Z is first, why will it load nothing and block Y? What are these other reasons?

The user who reported the initial problem has given more details, which I share here, with his leave, in order to assist with finding the problem:

I setup a test game with this arrangement. NFF.
If there's a simple map (online game way to big to test with) out there showing the problem, it would be necessary to have...

[jamespetts]

TurfIt,

thank you very much for your response. The person who reported this problem has offered to set up a highly simplified test game to demonstrate it, which I will upload here when it is available;

James.

[ӔO]

oddly enough, my hub and spoke cargo system works just fine.

[TurfIt]

I do see a probable problem with different convoi types carrying the same cargo type on the same line. Haven't tested yet, and might not get to it for a few days.

[jamespetts]

Thank you for your input, TurfIt. Do not worry about the few days; we are all busy. I will upload the sample game when the user who spotted the bug provides me with a copy. Thank you for your help with this.

Edit: The user has not managed to create a sample game yet, but tells me that he is working on it, but has now reported a very unresponsive experience when logging into the server, which I can reproduce, even though the server CPU load is at 50.1% and the game runs at acceptable speed in single player mode on the client; the game appears to hang for tens of seconds and then race ahead for a few seconds. Might this be related to the timing patch?
This was an unconnected issue now (partly) solved.

[jamespetts]

The user has now been able to reproduce the problem and produce a saved game, derived from the server game with a number of things deleted for simplicity, which is here. His description of the issue as manifested therein is as follows:

Use the previous picture & post for reference as to what is going on.  This is based on the server.  I removed a lot of stuff, but you can still see the problem at (3676,531).  The station is labeled: "Center"

Focus ONLY on stations: "Center" and "SE"
Focus ONLY on lines: "Center > SE" and "SE > Center"

Background:
* Planks are shipped through this station "Center" going to either the east or west side of map.
* You can see the specific problem if you look at the goods waiting in station "Center"
* Station "Center" has planks waiting to be shipped to station "SE"
* As described in the previous post, there are two lines connecting stations "Center" and "SE"
** "Center > SE" waits at center station (100%) and ships to SE
** "SE > Center" waits at SE station (100%) and ships to Center

Problem:
* As you can see, there is an empty ship from the "Center > SE" line that will not fill up.
* If you add more ships to line "Center > SE" they also will NOT fill up.

Cause:
* If you delete the "SE > Center" line then the ship will fill up.
* OR, if you add a ship from the "SE > Center" line and then manually adjust the schedule so it docks with the station "Center" then it will fill up.

This proves that the game is incorrectly assigning routing to the wrong line "SE > Center" when there are empty ships waiting to ship on the "Center > SE" line

Many thanks to the reporting user, and I hope that this assists.

[TurfIt]

This has nothing to do with the loading performance improvement patch.
Instead this is a manifestation of [11.24] Passengers/cargo make sub-optimal decisions about which convoy to board .

Specifically convoi 4098 which is waiting at "Center" station is part of line "Center -> SE". The path explorer has determined that line "SE -> Center" is the preferred line for transport from "Center" to "SE". The logic conditions for the cargo to board the non-preferred line are not met.
The path explorer has also determined the average waiting minutes for this link at 1764.5 minutes with a journey time of 1322.6 min. Convoi 4098 has a listed journey time of 4843.5 min, and the cargo waiting has only been there 686.4 min, so will not load. Once the cargo has been waiting for 10587.0 min, it will load.

This journey time appears to be garbage, and if time spent waiting at a station for cargo gets included in the journey time, then this will never fix. i.e. Cargo doesn't load because the journey time is too high --> convoi sits around waiting for cargo --> journey time gets higher --> cargo doesn't load --> ...

And the waiting_too_long logic is nonsensical. The higher the previous waiting time, the longer it needs to wait this time, and around we go again in a self escalating loop.

When you rewrite this convoy selection routine, I urge you to move as much as possible out of hole_ab for performance reasons.  The decision needs to be made much sooner at a higher level than when it gets to checking each cargo packet.

[jamespetts]

Thank you very much for that investigation and advice - that is very helpful. I will look into that in due course. This will require a major release to deal with.

However, the issue that I described in the first post appears on the face of it unconnected; it may well be that the user who reported the bug was unable to reproduce that, and instead reproduced something else.