Interim industry improvements

[jamespetts]

I have been working this afternoon on an interim solution to the industry issues that have been reported. As I understand it, the main reported problem has been a low ratio of consumer to producer industries, and a low number of industries overall.
The current system relies on a map wide industry density proportion system, which tries to keep the ratio of population to industry (weighted by the 1 divided by the "distributionweight" of each industry) at a constant level and also make sure that all consumer industry demand is fulfilled.
I have set up the industry-consumer-calibration branch on Github with an alternative system, loosely based on the existing increase_industry_every parameter used in Standard and optionally retained in Extended, but with a new algorithm.
In the new algorithm, the game will try to maintain a constant number of consumer industries in each individual town per unit of town population. Thus, with increase_industry_every set to 2,000, the game will try to maintain one consumer industry in the town per 2,000 in population. These industries will be built specifically in the town (if there be room for them) rather than anywhere on the map. If necessary, producer industries will also be built to satisfy demand, or alternatively existing producer industries with excess capacity and that are close enough will be linked to the new consumer industries.

I could not find a satisfactory algorithm for taking into account the industries' distrubutionweight, so anything more sophisticated will have to wait until the full town growth work. Because this measure is an interim solution, it may be somewhat more crude than the previous mechanism (albeit this was apparently not giving good results). However, the previous mechanism can be reactivated by setting increase_industry_every to zero.

With the new algorithm, it is possible to generate a map with no industry, and new industry chains will be built in the first few years and months until the requisite ratio has been reached. With the old algorithm, it is necessary to spawn enough industry at the outset, as the ratio of industry to population at the start will continue into the later game.
Before I commit this to the master branch, I should be grateful for anyone with self-compiling abilities to test this and comment on whether this is a workable interim solution to the industry issues. Remember, the purpose of this is to be a quick and lightweight change pending a full reworking of town growth, which will include a more comprehensive treatment of industry, so it will not be of utility to suggest coding changes that will greatly increase the amount of work necessary to be done.
The intention is to have this code in place before starting a new game on the Bridgewater-Brunel server.
Thank you all in advance for any feedback.

[Mariculous]

This sounds simple and yet powerful. I will give it a try starting with 0 industries and doing some artificial town growth.

[jamespetts]

This sounds simple and yet powerful. I will give it a try starting with 0 industries and doing some artificial town growth.

Splendid, thank you.

[Mariculous]

There is an excessive "const", so it won't compile:
https://github.com/jamespetts/simutrans-extended/blob/16a14a62a0b3a06231e1b77fa71ea2c64cf8b8a5/simcity.cc#L3902

Fixed this, now it seems to be working.

I just used the stephenson-siemens save as an example. Well, I guess we maybe need to tweak the industry_increase_every value...

I grew a few cities to get started, which grew quite a lot industries.
However, the more important point than the tweakable number is that most consumer industries shared the same producing factories.

Well, at least at a first view...

I continued growing each city once to spawn their industries.
For some cities it took quite a while to grow, so there must be some kind of quite compute intense stuff going on.
I did not finish growing all cities, because the game just froze for 5 minutes at some time, so I killed it, but when I had a look at the number of industries for the last time, there were 8304 industries on the map :O

The number itself is not an issue as we can tweak it, but the created city chains are a problem!
Just one example was a brewery: production of 474 base units per month producing 668% the amount in beer. It had exactly 1 pubs connected, consuming ~46 beer per month, whilst it could serve above 100 pubs!
There were many further breweries connected to a few pubs, where each of them could serve at least 50 pubs.

For sure I have just picked the pub chain as an example but this was also true for further chains.

The problem about this is we are wasting a lot of production capacities, so we will spam the whole map full of industries, so I guess whilst still promising it is currently practically not usable.

Further note, it seems there is a fixed number of consumers spawned depending on the number of inhabitants.
Some consumer industries, for example the supermarket, will sell a huge amount of products but it will also spawn a huge industry chain.
To better balance out the industry/inhabitant ratio, it should be better to use a products/inhabitants ratio to spawn consumbers rather than a consumber_industries/inhabitants ratio.

Further I noticed when deleting any industry using public services magic hand, it will simply place new industries within the city at the next city growth.
Whilst kind of odd, I guess it's fine as an interims solution.

[Mariculous]

I had som further investigations using city_grow_every

Observing a single type of goods, it generally seems that industry chains are fine if the amount of required goods increases from producers towards consumers, because it is always ensured any consumer can be served enough goods, which I guess is the assumption inherited from standard, to paksets city chains.
Obviously Pak128.Britain-Ex does not fullfull this assumption.

If the amount of required goods decreases towards the consumer at any point in the chain, there quite often will be new industry chains spawned whilst there are still a lot of capacities unused in other industries of that same type.
E.g. in my latest observations, there had been 3 breweries, each connected to three pubs, 1 industry connected to two pubs and 4 breweries connected to one pub each. Some pubs even got served by multiple breweries.
I did not mention the only brewery connected to more pubs yet, as that one existed for a long time already and was manyally linked to manually spawned pubs.
Each of the breweries could produce enough goods to serve the whole map with ease, the largest industry would use 12% of its production capacity, whilst the smallest one would use 24% of its production cpacity.

Furthermore, an industry chain will always ensure that the theoretical consumption of an industry can get satisfied, instead of trying to allow for producing as much as demanded from connected consumers, or a alittle bit more.
Thus, the larger industry from the example above will be connected to 4-times as many farms as it would require to serve the demand.
Connecting a few more than required for the demand is fine imho, but there definitely is some consideration of the demand, instead of theoretical maximum production required to prevent industries from spaming farms around the map.
Climbing the industry chain further up, there is the steelworks. There were 6 Steelworks each serving exactly one brewery and nothing else, 1 steelworks serving the only furniture factory on the map, 3 steelworks serving one car factory each and 1 Steelworks serving a car factory plus two breweries.
Any of these steelworks could serve all of the breweries and the furniture factory with ease. The Steelworks to car factory ratio is fine assuming car factories theoretical production. For sure, each of the two car factories had exactly one car dealer connected.

Additionally, any end consuming industry will not often, if ever improve their service to offer additional goods. None of the pubs offered cider, whilst there were multiple orchards with free cider capacities spread around the map.


I guess this is a general issue in industry chain logic rather than in the interims industry improvement approach, as it behaves just the same when using the create industry chain tool.
However, without improvements to the industry chain logics, the interims solution is impractical.

A relatively simple approach could be prefering connecting to existing industries with free capacity rather than spawning new ones when spawning new ones, additionally lazily adding suppliers to industries as soon as a new consumer gets connected.
Strictly doing this would however also lead to some kind of odd industry chains because, for example, there would only spawn a new steelworks, if none of the steelworks on the map had any free capacity.
So I guess there should be a likelihood depending on the distance and the relative amount of freee capacity of a city, to spawn a new one instead of connecting to an existing one.

[jamespetts]

Thank you for your work in looking into this: it is much appreciated. I only have time to respond very briefly now, and can look into this more in due course. However, there are one or two points that it is helpful to set out now.

Firstly, the basic re-linking logic is indeed largely taken from Standard; it attempts to make sure that the producers produce enough for the consumers to consume, but it appears to be too conservative in this. One thing to bear in mind, however, is the industry range logic in Extended, meaning that industries can only form links with other industries within range. As newer types of industries become available, this range gets longer, so, even if one steel mill could supply a whole map on the basis of its output, it might well not be able to do so on the basis of its range.

Can I ask you to clarify this part of the post?

Observing a single type of goods, it generally seems that industry chains are fine if the amount of required goods increases from producers towards consumers, because it is always ensured any consumer can be served enough goods, which I guess is the assumption inherited from standard, to paksets city chains.
Obviously Pak128.Britain-Ex does not fullfull this assumption.

If the amount of required goods decreases towards the consumer at any point in the chain, there quite often will be new industry chains spawned whilst there are still a lot of capacities unused in other industries of that same type.

In particular, can you explain what you mean by "increases from producers towards consumers" here? That is a little ambiguous and it would help me to understand more precisely what you meant here.

In relation to actual demand rather than maximum theoretical demand, this is a potentially unsolveable problem, at least not without a complex heuristic approach yet to be invented. The problem is that actual demand is based on how many visiting passengers actually turn up at the industry in any given time period. The linking of industries occurs at creation time, before any passengers have arrived at the consumer industry, this these data are not available to be used in demand computation. Further, I cannot at present devise any workable heuristic beyond the mechanism which is already in the game of having consumer industries scale their demand levels within the .dat file specified range based on the relative size of the town in which they are spawned.

I am also not sure what you mean by "free capacity of a city" here, and it would be helpful if you could elaborate.

In any case, thank you again for testing this: this is most helpful.

[Ranran(retired)]

Firstly, the basic re-linking logic is indeed largely taken from Standard; it attempts to make sure that the producers produce enough for the consumers to consume, but it appears to be too conservative in this

I remember seeing similar discussions in the standard, at which time the following were also issues:
Currently we can set InputSupplier[], but not vice versa.
I suppose controlling downstream as seen from upstream will need to add new functionality.

[wlindley]

Compiled and testing. — In 1750, building an Apothecary shop triggers creation of a local Vegetable farm, but that farm doesn't supply the same town's Market that thus remains without a Vegetable farm source.

Likewise, building a Tavern triggers a Common Brewhouse and some Grain fields but fails to connect the existing local Orchards, who make cider but have no consumer, leaving the Tavern without any cider sources either. 

A more complete balance would probably need some code so newly created consumers would look for existing local suppliers of all their desired goods; and that newly created sources would look for local consumers of all their supplied goods. 

[Mariculous]

"increases from producers towards consumers", means that a single industry in the chain does always produce fewer goods than a single industry consuming goods of that type can consume.
E.g. this is the case for the farm-brewery relation, where a single farm, which is the producer, will produce less goods than a single brewery, which is the consumer.

"free capacity", I did not write "...of a city", as it is the unused capacity on an industry, means the amount of goods that an industry could produce in addition, if it had enough conumers and suppliers.

it attempts to make sure that the producers produce enough for the consumers to consume, but it appears to be too conservative in this.

I don't think this way round is too conservative. I had not found any end consumer indsuytry that one could not serve by 100% with the good types produced by the linked industries. However most often there is only one or two industries linked to any end consumer, so it can most often only offer one type of products to clients.

About the range, I was aware of this but did not expect exceeding ranges to affect the 1024*1024 stephenson-siemens save in 1985.
However, I had a look at the paks now. The Steelworks of that time have a Range of 175 and the Brewery has a Range of 210.
If this distance is measured in manhattan distance tiles rather than in cartesian kilometers, this may be the cause.

To verify this, I have just started an empty 1024*1024 map where I placed a 3x3 grid of towns around the map and spawned many pubs in the central town.
All of them connected to the same brewery. When there were ~10 pubs connected, a new brewery was spawned nearby, so there was a beer consumption of roughly the "units per month" number of the brewery connected.
As far as I understand, the "Max. units per month" displayed at the top of the industry window is multiplied by the goods production multiplier below, so it may be that the industry mistakenly does not take care of that number, effectively allowing an industry to only produce the above given "Max. units per month" due to lack of consumers.

I did another testing in a "naturally" grown 256*256 map with 4 towns and placed a few pubs in the largest town.
The fourth pub I placed spawned somewhere outside of the town due to farms occupying the the city. I was only 69 tiles in manhattan distances away from the brewery but spawned a new brewery. That brewery did not connect to any Steelworks, whereas the steelworks was only 78 tiles away, so it should have connected to it.

I guess this could be the cause for excessively spawning industries.

In relation to actual demand rather than maximum theoretical demand, this is a potentially unsolveable problem

Why don't we spawn industry chains (here the pub chain again), that are able to supply the current "max units per month" of end consumer industries/shops (here pub, for which that displayed number is somewhere arount 40) and add new suppliers (here brewery) as soon as the currently connected breweries can not further increase their actual production by connecting further suppliers (grain farms) to serve the requested demand.

As there were multiple points that may or may not exist from the observations which will lead to spamming the map with industries, here is a short list of the obove described ones:
- industry Range of large industries may be too low to get enough consumers connected (assuming manhattan distance in tiles is given in the dat)
- industry chain may not take care of products production multiplier.
- new industries get spawned although there is a sufficient industry in range.
- industries spawning producers for their theoretical maximum consumption instantly, although they most often require only a fractal of this to serve the theoretical maximum demand of consumers.

[jamespetts]

I have spent some time this afternoon looking into this in some detail. I have pushed a number of changes to the master branch, and merged these into the industry consumer calibration branch. First of all, thanks especially to Freahk for having analysed the issues with the current industry generation algorithm so precisely - this has allowed me to test and modify the algorithm to ameliorate these issues.

First of all, I fixed a bug in which a fixed industry limit from Standard had been used as the limit to an industry's range, overriding the values specified in the .dat files. Note that this is a value in km and is converted into tiles on loading (if I recall correctly).

Secondly, I fixed a bug in the industry increasing density algorithm in which the missing inputs of the goods were not properly kept track of in industries that had multiple inputs, such as markets.

Thirdly, I added a mechanism to check, when building a consumer industry, which goods produced by suppliers are currently in the most surplus, and choose a consumer industry based on what goods are most in oversupply (this is somewhat randomised, so it will not always simply build a consumer industry with an input of whatever is the most oversupplied type of goods at present).

Fourthly, having checked, so far as I can tell, the consumption percentage values are taken into account when determining production levels for new industry construction.

Testing these three changes between them, this does seem greatly to improve the inter-linking of industries, especially on larger maps where there are more industries overall. Now, far more producing industries have multiple consumers, and far more multi-type consuming industries are linked to multiple producers of different types.

I wonder whether the change on the consumer-industry-calibration branch is actually necessary in light of these changes. I will thus defer discussion of making the algorithms unique to this branch more sophisticated until this has been considered further. Any testing and analysis of these new changes (both on the master and consumer-industry-calibration branches) would be most appreciated.

I do want to get industry generation right before starting a new game on the Bridgewater-Brunel server so that players there can have a good freight based game as well as transporting passengers.

[jamespetts]

Some further testing suggests that this new algorithm in the master branch works well with map generation, too, but I suggest setting the number of industries to 16x the number of towns, at least in 1750.

[Vladki]

Is it possible to adjust that during the game? On server game oil refinery has just closed recently, so it might be interesting to see if the chain recovers or not. And if not what will replace it

[Mariculous]

I did the testing on a local stephenson-siemens game, so I guess the answer is yes.

Additionally, a population related amount of industry density, more preciesely a relation in between the amount of potential consumers and the amount of producible goods seems to be better than a fixed industry density to me.

I will give the branch another try, but don't expect it to happen before next weeks tuesday.

[wlindley]

The latest seems to give a far better distribution of industries, with good re-use of existing suppliers. Kudos!

[jamespetts]

Thank you all for testing.

Vladki - I am not sure what you are wanting to adjust, but the new algorithm will apply to existing saved games.

Freahk - I am not entirely clear on whether you think that the currently adjusted master algorithm or the consumer-industry-calibration branch algorithm for determining industry density is preferable - it would help if you could elaborate.

[Mariculous]

I am not quite sure how the current algorithm works but as far as I understand it will maintain a constant industry density.
Thus, I guess consumer-industry-calibration is closer to what would be preferable imho.

To be more preciese, I would prefer it if each inhabitant has a demand for goods, averaged out for sure, so the industry growth algorithm will always try to place enough industries so the player is able to serve that calculated demand.
In short: The more inhabitants there are on the map, there more goods can be produced by worlds industries. More goods does not neccessarily mean more industry buildings, as newer ones will be able to produce more than older ones.

[Vladki]

Vladki - I am not sure what you are wanting to adjust, but the new algorithm will apply to existing saved games.

I wanted to adjust the industry density. You said that for new game 16x number of towns is recommended. How is it set for older games? Or is it somehow automatic during the game?

[jamespetts]

I wanted to adjust the industry density. You said that for new game 16x number of towns is recommended. How is it set for older games? Or is it somehow automatic during the game?

The industry density settings are not affected by anything on the master branch - the consumer-industry-calibration branch is what affects industry density settings.

The current industry density system does not allow for adjustment of this figure manually after map creation. The way that it works is as follows: on map creation, an industry density proportion figure is calculated. This is one divided by the distribution weight of each industry divided by the total population. The game will then try to maintain the same industry density proportion; so, as the population grows, new industries will be generated. As industries close down as they become obsolete, the actual density will be reduced, and so more industries will be generated. Conversely, as existing industries become out of date and, instead of closing, be upgraded, the distribution weight of those industries may be lower (which usually goes with higher productivity values, especially in later years), which will reduce the number of industries being built as population increases; or other industries may close without replacement. This is calibrated to simulate industrial decline in the latter 20th century in Pak128.Britain-Ex, along with rapid industrial growth in the 19th century.

[Spenk009]

Just toyed around with the build. The change in clustering, transportation distance and growth in areas of player presence is very nice. There is also a regular occurrence of two industries (sometimes sole industries in the town) receiving goods from the same producer, even if unintended, this is very fair and imo realistic.

There is the question of placement of industries in towns, where I've found that they are created on the edge of town (both in this branch and master). Is there a way to twiddle the renovation factor for these to allow a more centered placement? A market 1 mile from the centre of town (2 miles from the other side) doesn't appear too realistic, even if convenient for the player. If this is something reserved for the town growth overhaul, this observance is irrelevant.

My compiler complained that there was a duplicate const:

Code Select Expand

simcity.cc:3908:15: error: duplicate 'const'
  const uint32 const increase_interval = welt->get_settings().get_industry_increase_every();
Removing it (the second const) allowed simutrans-extended to build.

[jamespetts]

Spenk - thank you for your feedback. I presume that you are testing with the consumer-industry-calibration branch? A lot of the changes that have made a difference to the connexion of industries to one another and the chains are also in the master branch now, and I am not sure whether the changes in the consumer-industry calibration branch are necessary as the balancing of the overall density of industry is more precise and subtle in the master branch.

As to the placement of industries in towns, the consumer-industry-calibration branch was not intended to affect this. This is planned for a future major overhaul of town growth, but this will be a very large project.

[Spenk009]

I have switched back to the master branch and the mentioned const error has been fixed. My apologies for missing that. It's a great addition to gameplay especially for the industry placement plausibility to the player.