Topic: Matching freight network capacity with freight demand

[Jando]

The thing I struggle with most on my current map (all industries placed manually as public player, cross-connect industries flag switched on) is to match freight network capacity with demand. That's about the whole network, but the crucial thing is the last part of the network, from station or freight yard to end-consumers. When a new town is connected to the network - and with it multiple end-consumers, bakeries, markets, grocers, that sort of things - it always creates a huge spike in demand. Huge compared to what the end-consumers will actually consume once they got the stock delivered: it's very common that a end-consumer when first connected will demand stock that will equal multiple years of consumption of the same end-consumer. I've seen cases with up to 12 years of consumption demanded by end-consumers.

Of course that means that the same end-consumer will not demand anything at all for some years after the items from the spike have been delivered but instead consume the stock without any demand for new deliveries. That makes it very difficult to design the freight network. A new market would need a dedicated train service initially delivering the hundreds of fish, fruit and vegetable during the initial spike, but once these items have been delivered it only needs a packhorse to keep the link alive. That packhorse of course will never leave it's cargo bay. I've started to group end-consumers by location now, allowing that pack horse to keep the link to multiple end-consumers, else I'd need to bulldoze half a town to make room for cargo bays with pack horses that never leave the bay anyway.

And that cycle continues, after the stock is consumed I see a new spike and then again years of no demand at all.

A suggestion about what I would prefer:
If an end-consumer (thus an industry that produces nothing) has less stock than it's nominal consumption (first line of industry window, max. XYZ units per month) then let it demand that nominal consumption once per game month. Eventually - after deliveries happened - it will have more than that nominal consumption in stock. When it does let an industry demand what it consumed at the end of a game month.

[Vladki]

This sort of spikes have been dealt with in standard - JIT=2, but this never got to extended. Instead James has made some other more sophisticated algorithm for orders. Maybe it is time to reconsider if something simpler would not be better?

[DrSuperGood]

I do not think extended is designed to work with cross-connect factories.

[Jando]

I do not think extended is designed to work with cross-connect factories.

What I described in my original post happens without cross-connect as well.

[accord2]

I do agree with this post. It simply makes cargo transport not profitable... I think in this case James needs to put first gameplay quality and not realism.

Also, I don't think balance will solve this problem, since it is much more than that.

[Jando]

I think in this case James needs to put first gameplay quality and not realism.

Hehe, Accord2, for sure the current mechanic is not good for gameplay, but it's highly unrealistic as well. I believe my suggestion in my OP would lead to a far more realistic mechanic. Here's a screenshot from what just happened at some consumers, all of them around 8-10 game-years old.



Three industries pictured. Storage and transit graphs shown.
- The hardware shop, average consumption around 1 unit per day/month, with 100 units in stock demands 58 new units.
- The clothes shop, average consumption around 0.8 units per day/month, with 37 units in stock demands 73 new units.
- The market, average fruit consumption around 5 units per day/month, with 185 units in stock demands 328 new units. Same with vegetables, average vegetable consumption 4 units per day/month, with 42 units in stock demands 140 new units.

That looks all rather odd to me.

[Ranran]

I suppose some balancing issues can be pakset issues or both extended and pakset.

I guess many of the recently reported issues may have been caused by this adjustment from August 2017.

By this adjustment, the industry of pak128.britain set abnormal numerical values in dat, and as a result, it has many contradictions with the extended code specification.
For example, a farm can not create more than 400 fields, but not only is the minimum field set beyond that, but the maximum value is set to 1900.
So I am confused what is going on... (´・ω・｀)

[jamespetts]

This is a complex issue, as all issues relating to balance are. First of all, to deal with Ranran's point: I had not realised when adding the larger number of fields for farms that there was a hard limit of a 9 tile radius. I have now abolished this hard limit from the next nightly build, so hopefully farms will have their correct production.

However, I do not believe that this is relevant to the initial issue. That issue seems to be that the storage capacity of consumer industries is too high, which, coupled with a system in which industries demand goods until their storage is full, can lead to initial spikes in demand. The system is designed to regulate the demand thereafter, but this will not affect the initial spikes. One thing to bear in mind is that all times should be treated as on the short timescale not the long timescale, so a game year is actually 76.8 hours which makes (assuming 16 hour days, as we do not simulate the night) 4.8 days. 12 game years would be 57.6 days, which would be too much for a market stall with fresh fruit or meat, but probably not too much for a hardware or furniture shop.

Can anyone identify any particular industries whose internal storage is too high? If so, I can look into recalibrating these.

As to demand regulation more generally. it is possible that, when introducing the new visitor based demand system for industries, I had not adjusted the demand regulation algorithm to take account of the fact that the industry's demand is based on actual visits rather than a hard-coded number, which would mean that ongoing demand is too high. However, initial demand is necessarily an intractable problem, since it is not knowable how many customers will visit the consumer industry when it is selling products until it actually starts selling products.

If the only demand spike issues are initial, then this suggests that the problem is not that the demand regulation did not take into account visitor numbers. However, if demand spikes persist indefinitely, then this would suggest that I would need to look at adapting the demand regulation algorithm to take account of visitor numbers.

I should be very grateful if people could let me know what they are experiencing as to demand spikes (i.e., initial only or indefinitely persistent) so that I know whether to look into this aspect further.

Thank you for your feedback on this so far.

[Jando]

Hello James, and many thanks for looking at the issue!

That issue seems to be that the storage capacity of consumer industries is too high, which, coupled with a system in which industries demand goods until their storage is full

I do not believe that storage capacity of consumers has anything to do with the issue, instead storage capacity of industries seems to have no effect. Consumers commonly demand much, much more, even hundreds of times the size of their storage capacity. If you can please look at the screenshot I posted in reply #5. There you have a market (storage capacity for fruits: 2 crates), yet the market has 185 crates in stock and has now demanded 328 more. With that demand delivered the market will have about 250 times it's storage capacity in stock. That seems to be true for all industries: storage capacity has little or perhaps even no effect on demand at all.

If the only demand spike issues are initial

No, spikes are not only initial. In the screenshot in reply #5 you see consumers that are in operation since 8-10 game years that are creating a new spike.

Thank you again, James, for your good work on Extended!

[jamespetts]

Thank you for letting me know: that is helpful. Are you able to upload a saved game in which these demand spikes can reliably be reproduced so as to assist me in testing for the causes of this and devising remedies? That would be most helpful.

[Jando]

Are you able to upload a saved game in which these demand spikes can reliably be reproduced so as to assist me in testing for the causes of this and devising remedies?

Hello James, can be observed in every developed freight map. Here's a link to my current Pinterridge map (link valid for 60 days): https://simutrans-germany.com/files/upload/AA_Demand.sve

How to observe:

1. Load above map, hit pause.
2. Map is centered on Pinterridge Freight Yard that is connected to almost all producers on that entire map. Just across the street to the north of the cargo bays are 3 free tiles (that I just bulldozed), switch to public player and place a grocer on one of these free tiles, the grocer will be in range of the freight yard there.
3. Switch back to default player and examine grocer, unpause game. The grocer will immediately demand 22 canned food. Why 22? Because that is the output storage size of the cannery, thus the grocer immediately demands all canned food in storage there.
4. Cannery will produce more canned food and the grocer will demand 4 canned food more when they become available, for a total of 26 canned food, that is 13 times the input storage of the grocer and about 52 times it's future daily/monthly consumption.

You can repeat this test by placing a hardware shop or a builders' yard on one of these free spots, the mechanism will always be the same. A placed hardware shop will initially demand 54 crates (because that is what the hardware factory has in stock) and will continue demanding until it has demanded 62 crates. That is about 5 times it's input storage and about 60 times it's daily/monthly consumption. Same with a builders' yard, you would see it demand all steel and all concrete available in the respective factories.

Obviously the above shows how the initial spikes happen. Why we also get spikes years later like shown in the screenshot in my reply #5 in this thread I have no idea.

[jamespetts]

I have been looking into this recently. I have just pushed a modified demand algorithm that is intended to address this issue. However, the new demand algorithm is not enabled by default on existing saved games. To enable it, you need to change the just_in_time value in the "economy" tab of the advanced settings dialogue (key "i" in Pak128.Britain-Ex) to a value of 2 or 3 (2 and 3 have different behaviours; 3 ignores the consumer industry's input storage capacity entirely whereas 2 does not; 2 is the default for new games with the next nightly build).

The existing algorithm (which still operates when just_in_time = 1) will cause the consumer industry to demand enough goods to fill its input storage and then only demand as much as it needs to fulfil its consumer's estimated demand, taking into account transport lead time.

The modified algorithm, with just_in_time = 2, will demand only enough goods to satisfy estimated consumer demand without running out of stock, taking into account the lead time for the goods, save that the consumer industry will not demand so many goods that the input storage is likely to overflow at any point (taking into account future estimated demand).

With just_in_time = 3, the industries behave in the same way as with just_in_time = 2, save that the consumer industry's input storage size is ignored.

I should be grateful if people could test this to see whether these changes have any effect on the demand spikes reported here.

[Jando]

Many thanks for your work on this, James. I will test this over the next days and report back. Thanks again!

[jamespetts]

Splendid, thank you.

[Jando]

The existing algorithm (which still operates when just_in_time = 1) will cause the consumer industry to demand enough goods to fill its input storage and then only demand as much as it needs to fulfil its consumer's estimated demand, taking into account transport lead time.

The modified algorithm, with just_in_time = 2, will demand only enough goods to satisfy estimated consumer demand without running out of stock, taking into account the lead time for the goods, save that the consumer industry will not demand so many goods that the input storage is likely to overflow at any point (taking into account future estimated demand).

With just_in_time = 3, the industries behave in the same way as with just_in_time = 2, save that the consumer industry's input storage size is ignored.

Hmm, had time now for a few tests. Not sure what to think about it. Firstly, whether I run just_in_time with 1, 2 or 3 makes no difference for an initial spike at all, a newly connected bakery demands the same amount of flour under either setting. It will demand as much flower [edit: flour, lol] as is available at the grain mill, 116 in my case, no matter whether 1, 2 or 3 is used for just_in_time.

Here is my test game: https://www.jschuh.eu/Shared/Industrial1.sve
Map shows Hartcester Freight Yard, connected (amongst other things) by trains to grain mill and brewery. Load map, place bakery using public player, bakery will demand 116 flour. Reload map, set just_in_time to 2, place bakery, demand 116 flour, reload map, set just_in_time to 3, place bakery, demand 116 flour. So there's no effect I can see with a newly connected industry.

I do see changed demand though with some other - already connected - industries. Same saved game, locate stop Tatsand Shops West (a public stop), there's a pub 3 tiles east of the stop. Having just_in_time at 1 the pub will not demand any new beer, setting just_in_time to 2 or 3 and it will demand 54 new beer, means all beer in stock at the brewery at that time.

But then I don't really understand the difference between modes 1, 2 and 3 too. From your post it seems to me that the only difference between them is the handling of the industries' input storage. However, like I said earlier in post #8 here in this thread:

I do not believe that storage capacity of consumers has anything to do with the issue, instead storage capacity of industries seems to have no effect.

All industries on all my freight maps commonly demand much, much more items than their input storage size.

Thanks for your work on this, James!

[accord2]

In my current game of 20 years (started in 1750) with this feature I see no difference from before. Industries produce and consume the same amount of goods as before. I did change to just_in_time = 2 from original =1.

It feels commercial buildings (as bakeries) don't consume much, even if they have a great amount of costumers. This causes other suppliers to slow down/halt.

[jamespetts]

Thank you for the testing and feedback. I am afraid that I have not been able to reproduce the behaviour that you report with the saved game supplied.

I tested according to the following method:

(1) modify just_in_time to 2;
(2) pause the game;
(3) save the game;
(4) place a bakery near Hartcester Freight Yard (crossconnect being enabled, this is automatically linked to the map's only grain mill);
(5) save the game again;
(6) unpause the game.

The bakery's internal storage is 11 and it initially calculates its maximum in transit number as 128. The bakery in fact demands 79 sacks of flour in this configuration, which is within the maximum of the sum of the maximum in-transit number and the internal storage. After a second or two, the maximum in-transit number is recalculated to 13.

Re-loading the saved game produces a different result: the maximum in-transit quantity of 13 is calculated immediately, and only 13 sacks of flour are demanded and put in transit. Can I check that you have definitely downloaded the latest executable? If you have, can you re-test using the exact method above to see whether you get the same results?

Incidentally, it is not correct that the consumer industry's input storage has no effect on the demand, although it will only have an effect in specific circumstances.

[Jando]

Can I check that you have definitely downloaded the latest executable? If you have, can you re-test using the exact method above to see whether you get the same results?

Yes, I have the newest nightly. The difference between what you did and what I did is that you saved the game twice, once after you modified just_in_time, and once after you placed the bakery. The 2nd safe is the essential one it seems, what the bakery demands after unpausing the game depends on whether the game was saved after placing the bakery or not. Just the save, no reload.

- This sequence: load my saved game, pause, set just_in_time to 2, place bakery, unpause game leads to the bakery demanding 79 flour and half a second later another 37 flour for a total of 116. Note no save in that sequence.
- Saving the game (just the save, no reload) after placing the bakery and before unpausing the game leads to the bakery demanding 13 flour.

The above seems to happen no matter whether you save after modifying just_in_time or not.

Edit: Additional info. The save action may also be responsible for the demand spikes we see with existing industries, like shown in the screenshot in post #5 in this thread. Just saving a game can cause an industry to create a demand spike.

[jamespetts]

I have been carrying out further tests. The issue is complex, but I have pushed some modifications to my Github repository which will be incorporated into the next nightly build which should help to regulate the flow of goods to industries better.

First of all, the maximum in transit figure was not correctly recalculated in some cases. This has now been corrected. Secondly, in some cases, it was possible for the maximum in transit figure to go into the negative. This can happen when an industry is built on the same tile as an industry that has been deleted and demands the same type of goods from the same factory, and those goods are subsequently re-routed. This has also been corrected.

Finally, I have also modified the method of calculating the maximum in transit percentage for consumer industries to take into account their actual consumption based on the number of visiting passengers (consumers) in proportion to their visitor demands.

I should be grateful if you could re-test to see whether this improves matters. Do be careful to check that there are not already goods en route to a now deleted industry at tile 1246, 274 when testing, as this caused me substantial difficulty before I understood what was happening. If you replace the industry too soon, all the goods (the numbers having been calculated under the old algorithm) will be sent to the newly built industry irrespective of the new algorithm.

[Jando]

Thanks for you work on this, James!

I'll report back tomorrow.

[Jando]

Nothing to report yet, sorry, I need more time ... so many items from old demand spikes still in the system (either on consuming industries or in transit) that it's hard to judge the effect of the new system, will continue testing and report back in a few days.

[Jando]

Some first results/observations. I created a new map, year 1800, zero industry from map creation. I set just_in_time to 2 and placed a few industry chains myself as public player:

- a market (supplied with fruits, vegetables and fish),
- a tavern (supplied with cider and beer),
- a bakery (supplied with flour).

Interesting here are the numbers in the consumption line of the industry window, after creating the industries I get:
- fishing port consumption: live fish 0/0(5)/4
- tavern consumption: beer 0/0(2)/2, same for cider
- bakery consumption: flour 0/0(4)/4

Made lines, built stops, bought vehicles, etc. After 2 in-game days I observe:
- The fishing port consumption now is 8/129(31)/4. Interesting is that the fishing port has demanded all live fish as soon as it became available at the fishery. It stopped doing that when the first live fish from the fishery arrived at the fishing port.
- The tavern consumes cider just fine. Cider consumption line reads 4/1(3)/2. Beer however is a problem, the consumption line is 0/0(0)/2. The tavern does not demand any beer, the brewhouse is stocked.
- The bakery flour consumption line reads 0/0(1)/4. The bakery does not demand any flour, the windmill is stocked.

I'd guess the max. in-transit number can go below 1 and if it does the industry cannot demand anything. Saved game after these 2 in-game days is here: https://www.jschuh.eu/Shared/Crawport1.sve

[jamespetts]

Thank you for spotting the issue with the zero demand rounding error - I have pushed what I believe to be a fix to this. I should be grateful if you could re-test with to-morrow's nightly build.

[Jando]

Thank you for spotting the issue with the zero demand rounding error - I have pushed what I believe to be a fix to this. I should be grateful if you could re-test with to-morrow's nightly build.

Thanks for looking at the issue, James, much appreciated! Fix partially works, bakery and tavern are now demanding 1 flour and 1 beer. There still seem to be other minor rounding errors though, for example the bakery showing 1 flour in stock and the status "no material" in first line of industry window at the same time.

Fix works good enough however to judge how demand at small industries in the early years (1800 in this case) works out. Two observations:

1. All small industry chains (here defined as industries with 0, 1 or 2 as maximum in-transit number) follow a common demand pattern: demand 1 or 2 items (1 or 2 as defined by max. in-transit), consume these 1 or 2 items until out of stock, then demand another 1 or 2 items. None of these industries demands new items when not out of stock. Industries with max. in-transit numbers of more than 4 do not show this pattern. Easily observed at the windmill in the above saved game: mill demands 2 grain, when delivered consumes grain and produces flour, when out of grain demands 2 new grain.

2. No industry attempts to fill it's input storage. The small chains from 1) cannot because they never demand items when not out of stock, the larger chains (best observed with the market in the above saved game) have so high in-transit numbers that their demand exceeds their storage capacity by a large margin.

I wonder if the max. in-transit number needs to have a minimum value like: input storage capacity - items now in stock - items already in transit.

[ACarlotti]

I haven't checked recently, but a while back I found that some of the rounding of goods amounts was potentially quite confusing. In particular, I think the amount of an input good in stock is rounded down in the display; if the amount to order is determined by rounding down the amount needed, then this would mean that there isn't at least 1 unit of unserved demand until there is at most 1 unit in stock (which will immediately decrease to less than 1, rounding to 0).

[jamespetts]

Thank you for spotting that: I had written code intended to avoid precisely the situation described in (1), but there was an error with it: I have now fixed it, and the fix should be applied in the next nightly build. I should be grateful if you could re-test.

As to (2), this is not inconsistent with the intended behaviour: industries will only need to fill their input storage in so far as their maximum input storage is in fact the amount that they need to keep in stock at any one time so as not to run out in the time that it takes for goods to arrive from the producer industry.

[Jando]

That looks much better, now, James!

- Windmills are behaving fine now with the latest nightly. They show max. in-transit as 2, slowly lowering that number to 1 once they have 3 or 4 grains in stock.

- Bakery works fine, max. in-transit is 1, the bakery demands a new flour before it runs out of stock.

- Tavern max. in-transit for beer is at 0, tavern demands a new beer the moment the in-stock number at the tavern goes to 0. Probably an effect of rounding the numbers I assume, showing 0 but having, say 0.3 beer still in stock. I've seen a case where the daily consumption-graph for beer at the tavern went from 1 to 2 while the tavern showed 0 beer in stock. Thus I assume not really out of stock but merely a rounding effect.

- Same for brewhouse and stock of woodchips, max. in-transit is 0 as well. Brewhouse shows stock as 0 for quite some time before it demands 1 ton of new woodchips. But I've seen no case of the brewhouse actually stopping production.

- No odd demand spike observed so far.



Will be testing more over the weekend and report back, currently making an industry test map in 1930 to judge the effect of the changes with more demand and faster (truck) transport. Thank you so much!