48,000 jobs and no production

[zook2]

On the BB sever there was a car factory, next to but outside a medium-sized town. It produced quite successfully for many years and it grew larger several times. After one such growth event it had 48,000 open jobs but stopped producing. Several players supplied bus and train routes to nearby cities, but the number of employed never rose about a few dozen, and those always disappeared quickly. It never again made another car and finally shut down.

[Matthew]

I have got exactly the same issue on one of my maps. A car factory has 48,000 open jobs and no matter how fast I ship in commuters, I can never reduce the open jobs by more than a few dozen. It has never produced a car. This car factory was present from the start of a new map created in 1945, so the upgrade and field growth aspects of your situation weren't present in mine.

Links to the savegame and pakset are in this thread if anyone wants to see it; it's in the town of Pakenham.

[Matthew]

I have done more investigation into this issue and I have bad news. CarFactory1950 is essentially unplayable and it probably causes errors in the Extended program.

Signs: unplayability
I did manage to succeed where Zook2 says B-B players struggled, and got the Car Factory working (right-click for full images):


For a decade the Jobs Available total stayed in the 47,975-48,000 range. I eventually managed to get the factory started once about 9,000 commuters were registering, and as you can see it is currently running with about 13,000 commuters, which makes the factory barely operational (5% = about a dozen cars a month).

However, this required an enormous effort. Firstly, I am running trains and buses from all these towns into the station near the Car Factory, with a tramway for the final stretch:



These are very large cities by Simutrans standards (the largest city in B-B5 was about 150,000 in this era), and they are supplemented by commuters by plane from other areas of the map too. I was fortunate that by chance there were 400,000 people living fairly close to the Car Factory. But inevitably the more places you connect, the more irrelevant traffic you get on the convoys intended for Car Factory commuters. I cannot be certain exactly what factor finally got the Car Factory started, but I think it was one of two things.

Firstly, I manually built a load of level 36 tower blocks near the Car Factory and also built a 'New Town' of level 24/36 tower blocks around a railway station just along the mainline from the Car Factory:




These large buildings approximately never appear automatically in Extended (another bug!), but they mostly house exactly the Low class people that the Car Factory wants, which is very helpful in the 1950s when normal city growth overwhelmingly creates Medium commuters.

Note that this strategy isn't available to Bridgewater-Brunel players, who can't place these buildings and can only create new towns with the wrong class of commuter. And they are unlikely to be as lucky as I was, with the factory appearing near a cluster of big cities.

Secondly, the car ownership levels also naturally rise during the 1950s and this might have also played a role as more commuters could drive to the Car Factory. However, there is also contradictory evidence. For unrelated reasons (I was fed up of private car unstable equilibria ruining my bus routes TT), I halved the car ownership levels in privatecar.tab.... and the number of commuters arriving at the Car Factory actually increased. So I think private cars might have been hindering more than they were helping, and they were certainly hindering me from having fun. The road next to the Car Factory was insanely congested and I cannot clear the road tile that gives access to it. I used bypasses, dual carriages, roundabouts, traffic lights, one-way systems, and manual deletion to try and handle the private cars and buses going to and from the town with the Car Factory, but they keep coming. And as private cars tend to do, they're polluting their environment, specifically the log file....

Symptoms: private car logging

Log messages

As I have played on this map, the following warning and error have appeared more and more in the debug log:

Code Select Expand

Warning: obj_t::~obj_t():    couldn't remove 0x7abf7c2d0150 from 2106,1967,15
ERROR: vehicle_base_t::leave_tile():    'typ 65' 0x7abf98199030 could not be removed from 2106 1967
For help with this error or to file a bug report please see the Simutrans forum:
https://forum.simutrans.com
(2106,1967,15) is the access tile for the Car Factory and "typ 65" is "roaduser" vehicles, a.k.a. private cars or citycars. Last week these messages were appearing about 4 times per real-life minute and testing today it's about 5 times per minute. I am not saying my road layout is perfectly optimal, far from it, but it seems plausible that these messages are a result of excessive traffic from the Car Factory swamping that tile.

Each message occurs for a different vehicle every time, i.e. neither is ever run on the same vehicle more than once (assuming the object addresses are stable). Based on a small sample, I think the warning and the error occur for completely different sets of vehicles, again assuming the addresses don't change.

I have tried to find the root cause of each message and I will show my working as my C++ skills are limited.

obj_t::~obj_t()
This warning comes from /obj/simobj.cc:93. The ~obj.t() destructor looks up the vehicle's tile and tries to remove it from the tile's objlist through a call chain that ultimately reaches objlist_t::remove(), which we'll explore later. If this fails, the destructor tries to find another tile (boden) at a different Z-height of the same XY co-ordinates. If that succeeded, then it would log a warning that I never see.

If the destructor hasn't found this object, it then searches the objlist of every tile on the entire map for it, which must be hurting performance. Again, if the vehicle was removed, then it should show a warning message, which I never see. So these vehicles are never getting removed from the objlist by this process.

The function never reaches a return statement since it can't find the object in an objlist. However, since this is a destructor, the function still returns and AFAIK the object is still deleted.

I set watchpoints for both messages in a debugger. Here's the interesting parts of a typical backtrace for this one:

Code Select Expand

Thread 1 "simutrans-exten" hit Breakpoint 2, obj_t::~obj_t (this=0x7fff340a04a0, __in_chrg=<optimised out>) at obj/simobj.cc:93
93                      dbg->warning("obj_t::~obj_t()","couldn't remove %p from %d,%d,%d",this, pos.x , pos.y, pos.z);
(gdb) bt
#0  obj_t::~obj_t (this=0x7fff340a04a0, __in_chrg=<optimised out>) at obj/simobj.cc:93
#1  0x0000555555c91a56 in vehicle_base_t::~vehicle_base_t (this=0x7fff340a04a0, __in_chrg=<optimised out>) at vehicle/../vehicle/vehicle.h:62
#2  0x0000555555caf165 in road_user_t::~road_user_t (this=0x7fff340a04a0, __in_chrg=<optimised out>) at vehicle/simroadtraffic.cc:66
#3  0x0000555555cb07a7 in private_car_t::~private_car_t (this=0x7fff340a04a0, __in_chrg=<optimised out>) at vehicle/simroadtraffic.cc:431
#4  0x0000555555cb07f8 in private_car_t::~private_car_t (this=0x7fff340a04a0, __in_chrg=<optimised out>) at vehicle/simroadtraffic.cc:431
#5  0x0000555555c5a220 in karte_t::sync_list_t::sync_step (this=0x55557d0332c0, delta_t=100) at simworld.cc:4251
(BTW the line number for frame #2 seems to be wrong, perhaps due to preprocessor or overloading shenanigans.)

Frame #5 has the first real game logic in the call chain; the object is deleted by the third line of this snippet:

Code Select Expand

             case SYNC_DELETE:
                currently_deleting = ss;
                delete ss;
                currently_deleting = NULL;
                /* fall-through */

I'm completely unfamiliar with the private car state machine, but it seems that it's trying to delete the cars and necessarily succeeds despite the unsuccessful search of objlists.

vehicle_base_t::leave_tile()
This error message is at /vehicle/vehicle.cc:L242.

We take this path if the vehicle does NOT have the not_on_map flag AND (it doesn't have a grund OR obj_remove(this) returned false). That last function ultimately reaches objlist_t::remove(), which we mentioned earlier. And although the code is different, the next steps are effectively the same. We look for another tile (boden) at a different Z-height of the same XY co-ordinates. If that succeeded, then it would give a warning message ("should have been...") that I never see. If it fails, then we likewise search the objlist of every tile on the entire map for it, and again the absence of a success message in the log proves that the search always fails.

So if we put the two messages I do see in the log together, the game is searching the objlists on all 25 million tiles on my map roughly four times a minute. Ouch!

Watching for this error in the debugger, this is the interesting part of a typical backtrace:

Code Select Expand

#0  vehicle_base_t::leave_tile (this=0x7fff34010900) at vehicle/vehicle.cc:242
#1  0x0000555555cb49f6 in private_car_t::hop (this=0x7fff34010900, to=0x5555a1e9e3a8) at vehicle/simroadtraffic.cc:1343
#2  0x0000555555cb9b3a in vehicle_base_t::do_drive (this=0x7fff34010900, distance=7104) at vehicle/vehicle.cc:334
#3  0x0000555555cb10a1 in private_car_t::sync_step (this=0x7fff34010900, delta_t=63) at vehicle/simroadtraffic.cc:543
#4  0x0000555555c5a1e7 in karte_t::sync_list_t::sync_step (this=0x55557d0332c0, delta_t=63) at simworld.cc:4246
#5  0x0000555555c5a454 in karte_t::sync_step (this=0x55557d02dbc0, delta_t=63, do_sync_step=true, display=true) at simworld.cc:4319
#6  0x0000555555be5087 in interrupt_check (caller_info=0x555555d635c0 "simworld.cc:5497") at simintr.cc:113
#7  0x0000555555c5ecce in karte_t::step (this=0x55557d02dbc0) at simworld.cc:5497
#8  0x0000555555c752ab in karte_t::interactive (this=0x55557d02dbc0, quit_month=2147483647) at simworld.cc:11232
A quick glance suggests that private_car_t::hop() is called when a private car moves to another tile. The function doesn't actually check whether leave_tile() succeeds, so AFAIK this bug isn't actually stopping the cars moving to another tile.

Partial diagnosis: private car errors

objlist_t::remove()

Calling objlist_t::remove() on backtraced objects in the debugger returned FALSE.

Both log messages indicate this function is failing to complete. The definition is at /dataobj/objlist.cc:L509. Thankfully someone wrote a quick return path for tiles with empty or minimal objlists, but normally the function searches the objlist for an object and if it's found, moves everything above it in the objlist down one place, effectively deleting it. The fact the function is returning false shows that these private cars are not in the objlist for the tile outside the car factory, even though the game expects them to be there (presumably based on the co-ordinate member variables of the private car objects).

This function makes use of two member variables of the objlist. capacity is an uint8, the "Number of items which can be stored without expanding // zero indicates empty list!" according to the code comments. top is another uint8,which is "0-based index of the next free entry after the last element therefore also the count of number of items which are stored".

So it seems likely that objlists are limited to 255 objects. That is made more plausible by these lines in objlist_t::grow_capacity():

Code Select Expand

else if(capacity>=254) {
// capacity exceeded ... (and no need for THAT many objects here ... )
return false;
}
The objlist_t is deep in the bowels of Simutrans and I have very little understanding of it. But at first sight it seems that it assumes you'd never have more than 255 objects on a tile. Well, the access tile for the Car Factory has a tram stop, which I think means at least 4 infrastructure objects, plus the occasional tram, plus many pedestrians. And that tile also has an unknown proportion of 13,000 commuters from the Car Factory generating private cars there too..... I can see how that might exceed the code's assumption that you'd never need more than 255 objects on a tile. And the Car Factory has the potential for 3½ times as many commuters!

But what I haven't been able to work out is why these private cars aren't present in the objlists. I don't think it's when private cars enter the road tile from another one, because vehicle_base_t::enter_tile() has an error message that would be logged. I'd plausibly expect that there's some part of the passenger generation code creating private car objects, but I've tried following a few plausible call chains (from both that end and the objlist_t::add() end) without getting anywhere so far, perhaps because my IDE is nearly as confused as I am by all the overloading and inheritance in the vehicle classes.

And TBH I can't really see the point in spending any more time investigating the exact mechanism by which this bug occurs. We know that it only occurs on this single tile on the map, which is the road tile for this colossal Factory. Shrinking the factory seems very likely to fix it and should be much faster to test.

No reproducible unmodded case
Nonetheless, at the weekend, I spent an hour trying to create a reproducible case on a small and unmodded map, but I failed, because of the same staffing shortage problems that initially prompted this thread. I'll spare you the gory details unless you ask, but basically it would take several hours to build a save and network big enough to get the Car Factory chain working. I am not going to pursue this without a good reason.

(EDIT: This section was previously titled "No reproducible case"; the bugs are reproducible but in a modded game.)

Cross-contamination?
But that does raise the theoretical possibility that the errors and warnings in the game are due to my modifications to the game and pakset. I can't definitely rule it out, but I think that's unlikely. My own code modifications were to the UI, the timing of autosaves, and city growth; I haven't touched factories, vehicles or passenger generation. I did use martin509's industry generation branch, but he's confirmed that it doesn't touch passenger generation. My other changes to the pakset were to namelists, regions, and unrelated factories (making sheep farms bigger to feed the Car Factory!), none of which should affect internals like the object destructor. Decreasing private car ownership rates in privatecar.tab is relevant, but should have helped the problem.

Remedy?
It seems more sensible to focus on the hypothesis that CarFactory1950 is just too big and see whether making it smaller helps.

I do this with reluctance, because I can see why you'd want to have a very large car factory in pak128.Britain-Ex. Car factories were widely seen as the pinnacle of industrial power in 20th century Britain. The current value is a good faith attempt to represent that. But even in real life they weren't this big! The biggest I know of were Austin's gargantuan Longsight plant, which had about 25,000 workers at its peak, and Ford Dagenham, with up to 40,000. But bear in mind that Ford Dagenham was in a city of 8 million, the biggest in the world! B-B5's population peaked at about 4½ million, so expecting 1% of the population to work at a single plant is way out of scale. And while we all know the pakset balance is very much a work in progress, my Car Factory has 195 workers per car per month, which also seems excessive.

And ultimately, a factory that can't be operated in multiplayer games and causes bugs when it's operated in single-player just isn't fun.

So rather than spending more time trying to track down the cause of the bug, I hope to find out if drastically shrinking the number of commuters required to operate the factory at full capacity stops the bug(s) occurring, with a view to making a Pull Request. If I succeed, the Extended .dat reference could include guidance on a safe maximum number of Factory commuters. Does that seem a sensible way forward with this issue?

[prissi]

Assuming that the moving logic is still the same as standard, private cars are moved by a list at each sync_step and removed if they return not true for the sync_step(). This was done to avoid exactly what you describe, i.e. the removing from the sync_list and from the tile is synchronised.

The leave_tile() still allows to enter the next tile was to remedy errors with an overfull obj_list. But then the car (if enter fails) should probably set the obj_not on map error. However, since at max 200 pedestrians are allowed on a tile, and less than four cars will fit (as cars must spawn or placed on empty tiles), I fail to see how one can reach this condition. Are you sure it is a car and not a pedestrian? Those are added last and can pile up on stop tiles near busy stops. Standard discards pedestrians whenever they do not fit, but I would not rule out some errors there as this limit is rarely reached.