Tweaking industry growth to be consumption-centric.

[martin509]

Hello, I post here to bring a proposal!

In my time tinkering with a local copy of Simutrans Extended's source code, I found a few points where I was dissatisfied with the economic simulation of the game - namely, games would spawn in with far too few consumers. A beer brewery capable of churning out several hundred crates of goods a month, fed by a huge array of farms, would spawn, all to be routed to.. one single pub that can barely handle a fraction of the output. I'm sure many of you have run into this problem yourselves.

To cut to the chase, I have made a few tweaks to the game's world generation and its economic growth simulation to revolve around the consumer industries first and foremost. These tweaks are fairly simple and therefore work robustly from the testing I have done. I think they would be good material for a pull request, and was directed by Jamespetts to post them for discussion here. The details are as follows:

Worldgen tweaks

At world generation, when industries are initially seeded via increase_industry_density, every time an industry chain with producers, etc is added, the game then calls the same function with a new flag that forces it to only add consumer industries. This is repeated until the function fails (e.g. it can not add any further consumers) and then the cycle repeats until we reach the industry cap.

This has some major upsides (namely, at worldstart you can fully utilize most of the capacity of the world's industry) and some minor downsides (you need a much higher number of industries to achieve a diverse set of goods to transport compared to before, since the game is padding out the number with many small consumer industries).

Industry growth tweaks

This has a bit more risk of introducing bugs, and somewhat breaks savegame compatibility, but I believe it to be worth it. The current industry growth algorithm is, essentially, as follows:

At the start of the game, we calculate the overall ratio of industry to population (counting all industries, including producers and secondary industries) and save that as 'industry density proportion'. Then, every month, we calculate the current industry density on the map, compare that to industry density proportion multiplied by the population, and if it's lower, we add new industry chains with a random chance. If it's higher, we close industries with random chance.

The tweak I currently have running ties into the above worldgen tweaks by calculating industry density proportion only using consumer-only industries (stuff such as builders' yards, coal merchants, etc). If we do not have enough consumer industries, then we first attempt to add some more at the start of every month, and if that fails (likely because there are no overproduced goods to fill in consumption for), then we add new complete industry chains. Codewise, this is adding no new values to keep track of, but is changing how industry density proportion is calculated, which will somewhat break savegame compatibility (old savegames will experience a large expansion of industry if they were previously in equilibrium).

Code status

Currently, everything I have running is roughly 95% done. It is feature complete, but there is not a system of converting over the industry density proportion value from old savegames, I have not contributed to Simutrans before so the code itself may not meet style guidelines, and I still need to transfer it over to Github.

In mentioning this to James, he brought up the idea of it breaking multiplayer synchronization. I have not tested this in multiplayer or even looked at multiplayer code, but as far as I can tell nothing that I have touched is multiplayer-facing (most of it located in simworld.cc) and am open to help on this front if this gets to the code review stage.

One last concern (and the most salient one) is that as I understand it, the current intent is to tie in industry growth to city population growth later on down the line. This does not do that, but is not a massive upheaval to the current system and does more generally reorient the economy around consumption of goods. As such I do not believe this will hurt those future plans for a rework, but I am open to feedback!

[jamespetts]

Thank you very much for your contribution - it is always good to see new people interested in contributing to the code base.

This is going to need some careful thought. First of all, the intention of the current code is to ensure that consumer industries always have sufficient supply somewhere in the world. The idea is that industry production is demand led, which is based on reality. With only the description given, it is difficult to know whether the modification breaks this.

The future industry change planned is that consumer industries will be generated as part of the town growth algorithm (which it is planned to overhaul completely) and not part of the industry algorithm, leaving only industries higher in the chain (and power stations, which have a unique place in the game even if they are consumer industries in other respects) to spawn using the current code. Thus, any changes to the industry code need to be able to work with this planned new system. On the face of it, it sounds as if the changes that you have implemented would not work with this system, since they would compute the industry density proportion using only consumer industries, which in future will not be connected to this system at all. It may be better to think about how to create a separate way of generating consumer industries that can then be superseded cleanly by the town growth overhaul if you are after making a temporary change to the industry code pending that time.

As to multiplayer compatibility, it is not only things that directly affect the multiplayer code that can affect this: anything that in any way affects the simulation code can potentially affect multi-player compatibility. This is because the way in which the multi-player synchronisation works is for the game to run independently on each player's computer as well as the server's computer at the same time. Any tiny divergence in what happens on one client's computer will make that client lose synchronisation. Thus, anything that is not fully deterministic may do this, including calling simrand() in a place where it will not necessarily always be called at exactly the same point on the client or server (e.g. during world generation - use sim_async_rand() instead), using sim_async_rand() where simrand() should be used, using any floating point numbers (other than the software-only floating point class specific to Simutrans) anywhere in simulation code, or anything that affects the game state but is not saved/loaded fully. This is why any changes need very thorough testing on a well developed multi-player game. Finding bugs causing losses of synchronisation in multi-player gaming can be fantastically difficult (on one occasion, one single bug took six months of intensive work to find), so it is important to be very careful not to do anything that might introduce these bugs in the first instance.

Thank you again very much for your interest in working on Simutrans-Extended.

[makie]

I found a few points where I was dissatisfied with the economic simulation of the game - namely, games would spawn in with far too few consumers. A beer brewery capable of churning out several hundred crates of goods a month, fed by a huge array of farms, would spawn, all to be routed to.. one single pub that can barely handle a fraction of the output. I'm sure many of you have run into this problem yourselves.

This makes me unhappy in standard, too.
I would be happy if we had the program change in standard as well.

[Matthew]

Hello, I post here to bring a proposal!

Welcome to the International Simutrans Forums! This is one of the most promising debut posts we've had!

In my time tinkering with a local copy of Simutrans Extended's source code, I found a few points where I was dissatisfied with the economic simulation of the game - namely, games would spawn in with far too few consumers. A beer brewery capable of churning out several hundred crates of goods a month, fed by a huge array of farms, would spawn, all to be routed to.. one single pub that can barely handle a fraction of the output. I'm sure many of you have run into this problem yourselves.

You are absolutely right. I also find it very frustrating. And so do other players, as has been clear for a long period of time; it's been discussed many times in the Bridgewater-Brunel chat. Eight years ago, James kindly responded by adding a 25% chance of consumer industries being constructed. Unfortunately, this did  not really change the situation much, and six years ago he  increased the chance of consumer industries being built, as a "temporary fix", but this has not resolved the issue either. This is the most helpful place you could contribute!

In single-player games, I have tried adding additional consumer industries manually using the public player's "Increase Industrial Density", manual placement, and "Link Industry" tools. But this doesn't work in the long term. I made a patch to watch what happens and because it increases actual_industrial_density above target_industry_density, the main result is a wave of factory closures (wiping out your existing transport networks). As new factories are generated, the balance trends back towards the distributionweights. So manual intervention is like the tale of King Canute and the the tide; this has to be fixed in code.

So I really welcome another attempt to resolve this.

Codewise, this is adding no new values to keep track of, but is changing how industry density proportion is calculated, which will somewhat break savegame compatibility (old savegames will experience a large expansion of industry if they were previously in equilibrium).

...there is not a system of converting over the industry density proportion value from old savegames,

It's great that you are aware that breaking savegame compatibility is highly undesirable. People play Extended games for (real-life) years. And in the particular case of Bridgewater-Brunel's very large map, we don't want to add another 50MB of industry if we can avoid it, because the savegame size and memory consumption are already barriers to potential players.

But it must be possible to calculate a rough rule-of-thumb ratio between the old and new values of target_industrial_density by induction and use that to convert the value in old savegames? I think the extra effort would be well worth it.

This has ... some minor downsides (you need a much higher number of industries to achieve a diverse set of goods to transport compared to before, since the game is padding out the number with many small consumer industries).

In principle, this would definitely be a price worth paying, because it would make the freight chains economically more viable and far more realistic.

I still need to transfer it over to Github.

I would encourage you to put it up on GitHub as soon as you can, even if it's incomplete, because I would be interested in pulling it and doing some alpha/beta testing.

The idea is that industry production is demand led, which is based on reality. With only the description given, it is difficult to know whether the modification breaks this.

My reading is that the proposed design should indeed ensure that consumer industries always have supply. But obviously we don't know yet whether the implementation does actually achieve that.

The future industry change planned is that consumer industries will be generated as part of the town growth algorithm (which it is planned to overhaul completely) and not part of the industry algorithm, leaving only industries higher in the chain (and power stations, which have a unique place in the game even if they are consumer industries in other respects) to spawn using the current code. Thus, any changes to the industry code need to be able to work with this planned new system. On the face of it, it sounds as if the changes that you have implemented would not work with this system, since they would compute the industry density proportion using only consumer industries, which in future will not be connected to this system at all. It may be better to think about how to create a separate way of generating consumer industries that can then be superseded cleanly by the town growth overhaul if you are after making a temporary change to the industry code pending that time.

One last concern (and the most salient one) is that as I understand it, the current intent is to tie in industry growth to city population growth later on down the line. This does not do that, but is not a massive upheaval to the current system and does more generally reorient the economy around consumption of goods. As such I do not believe this will hurt those future plans for a rework, but I am open to feedback!

Martin's proposed system would generate as many consumer industries as necessary to meet the proportionate-to-population target (target_industrial_density, which I think ultimately goes back to summing up the population figures in citybuildings). Then it uses existing code to generate enough manufacturers and primary industries to supply those consumer industries. When the mapwide population increases, the target_industrial_density increases, so new consumer industries are generated, and then manufacturers and primary industries are generated to supply them.

James' plan for growth still seems to involve generating consumer industries according to the number of, and values in, citybuildings (not necessarily in strict proportionality to population but presumably there is some link, via land value or whatever?). But those consumer industries still need suppliers, so you're still using the existing code to generate enough manufacturers and primary industries to supply those consumer industries, right? When individual towns grow new consumer industries would be generated, but that growth must also increase the mapwide population (again, not necessarily strictly in lockstep).

Given the principle that industry should be driven by demand, I don't see anywhere target_industry_density is needed once the future town growth system is implemented.

So, if I have understood it, it seems that Martin's proposal fills the slot where your town growth changes will eventually go. The two fill the same role in the economic system, but that's a good thing. Martin can improve the game for the rest of the 2020s without changing any of the assumptions that the town growth plan relies on.

Even it's only temporary for another six-to-eight years, that's probably three Bridgewater-Brunel maps and thousands of hours of player time, so this patch is still well worth doing.

[jamespetts]

Matthew - thank you for your thoughts. Martin - I think that we had better have a look at the code on Github before deciding how well that this integrates. Bear in mind that it will need careful review for compatibility with the future 15.x branch as well as the current live codebase.

[martin509]

Alright, the pull request is now up! Feel free to look over the code and tell me what parts completely break everything.

One thing I noticed in the testing prior to putting up the PR is that one shortcoming is this system only fills in consumer-only industries. For the majority of goods that follow a strict producer -> manufacturer -> consumer hierarchy this is not a problem, but for particular important ones like steel, this does cause the game to spawn a large number of builders' yards instead of, for instance, canned food factories. Unfortunately I don't have a great idea of how to solve this at the moment.

Welcome to the International Simutrans Forums! This is one of the most promising debut posts we've had!

Thank you!

But it must be possible to calculate a rough rule-of-thumb ratio between the old and new values of target_industrial_density by induction and use that to convert the value in old savegames? I think the extra effort would be well worth it.

Perhaps. Potentially what could be done is calculating the theoretical consumer-industry density implied by all of the production in an old savegame. I haven't tested rule-of-thumb ratios either - my uneducated guess would be 1:3 consumer:total.

Even it's only temporary for another six-to-eight years, that's probably three Bridgewater-Brunel maps and thousands of hours of player time, so this patch is still well worth doing.

As someone who doesn't interact with this community much I have to say that 'only temporary, just the next 6-8 years' is very amusing to someone used to more normal game development timelines.

[prissi]

On the topic of consumer oversupply:

There was once an unfinished patch (pre-dating experimental), which took all the industry's good demands and tried to decide what kind of industry was needed next to fulfil all needed. However, some paks (like pak128) have a deliberate shortage or certain goods, which was not factored in and it did spawn way to many factories due to the demand of some consumers. But his would need also to take account of the conversion rates of the factories.

That would need to actually touch the factory_builder which this patch did not touch.

[jamespetts]

Martin - thank you for posting that. The implications of this are complex and far-reaching and very careful consideration needs to be given to how best to proceed.

I see that there are two categories of changes, one in each commit: (1) the changes to how the industry density proportion is computed; and (2) the changes to industry placement when force_consumer == 2.

The industry density proportion changes definitely need further consideration. Currently, the code takes 1 / distributionweight of each industry in proportion to the world population as the means of determining when to create new industries or close existing industries. This keeps the number of industries in proportion to the world's population. Exempting all non-consumer industries from this would effectively allow an unlimited number of non-consumer industries to be generated. It would (unless some other mechanism should handle this, but I am not sure that it does) fail to achieve the intended effect of reducing the number of different producer industries as new producer industries with a lower distributionweight become current and replace old producer industries with a higher distributionweight to replicate, e.g., the way in which smaller numbers of larger factories and farms replaced larger numbers of smaller factories and farms in the 19th century.

Can I ask - are there intended to be mechanisms that achieve these objects by different means? I am staying with my parents at present so do not have access to the computer on which the IDE is installed, so it is difficult for me to review the code thoroughly until I get back. Is that what the second part of the code was intended to achieve?

The modifications to the industry density proportion would also require recalibration of the industry density proportion in the saved game, as you anticipate. If the things that the industry density proportion now does for producer industries are to be done instead by another mechanism, the industry density proportion will need to be recalibrated by reference to consumer industries. There will need to be a method written that works out the ratio of the industry density proportion to the current industry density including producer industries, then applies that ratio to the modified industry density proportion affecting only consumer industries, then saves that new value as the industry density proportion whenever a saved game older than that which contains the changes should be loaded.

In terms of the second part of the code, it has been a while since I have worked on the industry generation part of the code, so I cannot immediately remember (and I do not have easy access to my IDE for a few days, so I cannot easily check) how the current logic for force_consumer works. I note that force_consumer must already be an integer rather than a boolean, but we seem to be using a magic number directly rather than an enum: I cannot recall whether this is already in the code base, but it is not ideal. I note that increase_industry_density() is called with force_consumer == 2 in one new place in the code (and loops this up to thrice). What exactly is the intention of that part of the code?

Thank you again for your work on this.

[martin509]

Martin - thank you for posting that. The implications of this are complex and far-reaching and very careful consideration needs to be given to how best to proceed.

I see that there are two categories of changes, one in each commit: (1) the changes to how the industry density proportion is computed; and (2) the changes to industry placement when force_consumer == 2.

That is correct, yes. I will note that the force_consumer=2 changes are independent of the other ones and can be put into the game without needing a rework of industry density targets, but not vice-versa.

The industry density proportion changes definitely need further consideration. Currently, the code takes 1 / distributionweight of each industry in proportion to the world population as the means of determining when to create new industries or close existing industries. This keeps the number of industries in proportion to the world's population. Exempting all non-consumer industries from this would effectively allow an unlimited number of non-consumer industries to be generated. It would (unless some other mechanism should handle this, but I am not sure that it does) fail to achieve the intended effect of reducing the number of different producer industries as new producer industries with a lower distributionweight become current and replace old producer industries with a higher distributionweight to replicate, e.g., the way in which smaller numbers of larger factories and farms replaced larger numbers of smaller factories and farms in the 19th century.

Can I ask - are there intended to be mechanisms that achieve these objects by different means? I am staying with my parents at present so do not have access to the computer on which the IDE is installed, so it is difficult for me to review the code thoroughly until I get back. Is that what the second part of the code was intended to achieve?

On the first point, while this does not directly cap the number of non-consumer industries, generation of new ones is still bottlenecked by production, as increase_industry_density will not add extra production for a good that is already overproduced. As long as the map's total consumer density is at the target, it will not generate new industry.

On the second point, I have not substantially touched the code for industry upgrading/retirement, so thank you for catching that. Looking it over, it does appear that the probability for an industry to upgrade versus retire is in fact affected by the current industry density. Replacing this logic would probably involve instead determining probability to upgrade vs. retire based on how much of a production shortfall exists in a factory's goods (while preserving the existing logic for consumers). This is incorrect. It appears the current logic for factories deciding to upgrade does not directly take industry density into account apart from 'if upgrading would take us over the target density, don't'. However it does affect which upgrade to choose in cases where multiple are available, via a complex formula that I'm not sure I fully understand (link here) and I'm not sure isn't broken (testing it with example values in Excel seems to spit out very small values that do not change much with varying industry density, to the point that dividing density by 3 only affects it by a few percentage points(?)).

The modifications to the industry density proportion would also require recalibration of the industry density proportion in the saved game, as you anticipate. If the things that the industry density proportion now does for producer industries are to be done instead by another mechanism, the industry density proportion will need to be recalibrated by reference to consumer industries. There will need to be a method written that works out the ratio of the industry density proportion to the current industry density including producer industries, then applies that ratio to the modified industry density proportion affecting only consumer industries, then saves that new value as the industry density proportion whenever a saved game older than that which contains the changes should be loaded.

Yes, this is basically correct. My completely uninformed intuition is that a 3:1 ratio would be approximately on target as a rule of thumb (since for any given amount of consumer industry you probably have, on average, an equal amount of producer and manufacturer industry), but I have not tested it (potentially, quaternary industries such as coal/iron->steel->canned goods->consumer could mess with this ratio). In theory, what will occur to old savegames where there is too little consumption is that they will lose some production but experience a substantial increase in consumption to compensate - if nothing changes then there isn't much use to the system, after all.

In terms of the second part of the code, it has been a while since I have worked on the industry generation part of the code, so I cannot immediately remember (and I do not have easy access to my IDE for a few days, so I cannot easily check) how the current logic for force_consumer works. I note that force_consumer must already be an integer rather than a boolean, but we seem to be using a magic number directly rather than an enum: I cannot recall whether this is already in the code base, but it is not ideal.

I originally wrote this portion many months ago, and did not have any intent of putting it in a PR at the time, so I did not bother with changing force_consumer to an enum. It was in fact already a magic number, for reasons unknown to me - I'll probably change it to an enum soon just to make things clearer.

In regards to how the logic actually works, the current logic is that if force_add_consumer (which has a 75% chance of being true if force_consumer == 0) is false, that enables a set of logic for adding a producer industry chain. If it succeeds in that (e.g. if it finds an undersupplied good to add) then the function returns. If force_add_consumer is true, then increase_industry_density skips ahead to a set of logic for adding a consumer industry, which it selects from the list of oversupplied goods. This list is generated by, for each factory that produces the good, summing up the total production and subtracting the total number of production contracts (or global production if we are not using contracts).

With the new logic, force_consumer==2 forces force_add_consumer to true no matter what, and in addition, there is an additional segment of logic for if force_consumer==2. This is because adding consumers to oversupplied goods at worldgen time does not seem to affect their removal from the oversupplied_goods list generated at the start of increase_industry_density(), at least when using production contracts, which results in a loop to add consumers until we can't anymore never actually terminating and instead just adding industries forever. This logic manually checks if a good is oversupplied by comparing factory production against consumer industry consumption (instead of contracts), and then adds consumers using a slightly different call of build_link() than usual (namely, with the flag to add producer industries set to false so it does not infinitely escalate).

I note that increase_industry_density() is called with force_consumer == 2 in one new place in the code (and loops this up to thrice). What exactly is the intention of that part of the code?

This part of the code is for industry generation during worldgen. Every time increase_industry_density() is called normally, we then call increase_industry_density() with force_consumer=2 repeatedly until it fails (returns 0 industries built) 3 times in a row. This is so that every time producer industries are built, we fill in consumers until we no longer have overproduction.

[jamespetts]

Martin - that is very helpful, thank you. I think that I am going to have to spend a lot of time investigating the code in the IDE after I get home and testing to consider the implications of this fully, as I need to re-acquaint myself with the complete context before I can make sensible decisions about this. 

In general, we need a sensible system for making sure that the different producer industries (including different industries producing the same product) are in proportion to each other in accordance with the distributionweight, and that the industry density generally on the map is in proportion to the population. We need industries to close down when it is realistic for them to do so (the eventual plan being to simulate ports and importing/exporting).

[martin509]

An update: I think I have managed to (through great trials and tribulation) enhance the world-gen aspect of this to not just fill in with consumers, but also with appropriate manufacturers!



In this case, there was a steel mill generated at worldstart, naturally making a lot of steel. Steel is mainly consumed by other manufacturers, and just shipping it to builders' yards (which is ordinarily what the world would end up filled with) is pretty boring. What we instead do is allow non-consumers (such as furniture factories, canneries, etc) along with appropriate consumers for those (furniture shops, grocers, etc) to be generated in the 'fill in consumers' stage. These new manufacturers lack any suppliers at this stage (for instance the canneries don't have any fruit, fish, etc), but to avoid an infinite escalation we wait until we are done filling in consumers to then call increase_industrial_density() with a flag to only add producers, which fills in the missing supply chain links. It's not quite done but given I finally have it actually working I think it's feasible to add to this PR's scope.

Edit: Another update! I think it's properly balanced out now - it's hard to fully show in one screenshot but I changed the consumer-infill to favour final goods (ie goods not consumed by any factories), and the end result is that you end up with a very diverse distribution of consumers and minimal bottlenecks for factories.

[martin509]

Alright, the last few days have seen a lot of progress tightening up the code - I'm now at a point where I'm very happy with the world generation and think it's great in practice and 95% bug-free. If you're at all interested, please feel free to go ahead and test out the branch on Github for yourself!

I ended up having to overhaul the code for completing incomplete supply chains, as it did not properly handle undersupplied goods or bottlenecked consumers (if we have a factory taking in 100t of inputs, but only 10% of its consumption is used, why add more than 10t of inputs to the supply chain? etc). This was a bit painful but is nearly airtight in practice, and helps minimize the sheer number of suppliers generated (although for goods that require small farms, etc this can still result in hundreds of farms on the map).

For intermediary goods like steel, when choosing to add a new manufacturer we first check if that manufacturer's goods are oversupplied, and if so we instead add a consumer for that good. This results in a list of consumers for a given industry that looks like, for instance, steel mill -> cannery -> 8 grocers (each time we try to add a cannery, we find out there is an oversupply of canned goods and add a new grocer to consume it instead) -> some builder's yards -> a new cannery -> more grocers to fill in that one  -> yet more builder's yards -> a hardware factory -> infill with hardware stores until that factory is satisfied -> etc.

Here are examples of what this looks like in practice for medium-sized worlds with 60 consumers generated in the years 1800, 1900, and 1950 (unfortunately yes I can see the steelworks with incomplete supply chain in the last image, I'm still tracking that bug down).


In a nutshell, so long as you have a decent number of consumers to generate, there will be very high diversity in supply chains for you to work on completing, and those supply chains will all be free of bottlenecks. James, I would like to apologize for the mountain of code changes you will need to review but I promise it is all very worth it and not broken.

[jamespetts]

Thank you very much for that. I have spent some hours running some tests on this, albeit it is hard to test this thoroughly within a reasonable amount of time as industry generation changes slowly over a long period in the game. I have yet to test two critical things, which are (1) compatibility with the 15.x branch; and (2) network safety (compatibility with online games). The latter will require checking for losses of synchronisation at month ends, so setting up a suitable test case is likely to take a very large amount of time.

Conceptually, I think that the principle of calibrating the distribution weight to consumer industries and then having producer industries works, given that which producer industries are built is still randomised based on their distribution weights, and that the overall demand for producer industries is set by the consumer industries, which in turn are determined by the distribution weight.

However, as discussed, we do need to recalibrate the saved games' target industry density (and thus update the saved game version of this branch so that the game knows when to do this) so that this works with existing saved games.

There is also an issue with map generation: in the master branch, the map generation setting for the number of industries will control the number of industries. With the new code, it controls the number of consumer industries, so the UI is now misleading. Either we need to modify the code so as this number controls the actual number of industries generated, or we need to modify the UI so that it explicitly refers to controlling only the number of consumer industries generated. I should be interested in people's views on that latter point.

In any event, thank you for your work on this - it is much appreciated. Once we have a solution for both of the above issues, I will be able to start work on network stability testing and 15.x branch merge compatibility testing.

[martin509]

On the topic of the code now controlling the number of consumer industries, this was done mainly because if I kept to the normal behaviour of targeting a total industry count, it became very difficult to adhere to that target, as adding one consumer might add dozens of supporting industries. At least with targeting a specific number of consumer industries, the game will always add the same number of consumers. Unfortunately I am not sure how to change the UI to match right now, as finding the relevant places in translation files seems very opaque to me.

As another note to place on the long list of potential scope creep here, this change may require pakset rebalancing to be done, as it has revealed some major imbalances in production rates that presumably were not a huge deal before now. Certain industries, especially from the year 1945 onwards, have production or consumption wildly out of pace with their associated producers and consumers. This is most noticeable with supermarkets, which require hundreds of crates a month of various finished goods that result in hundreds upon hundreds of farms spawning, and on the flip side, oil refineries produce an absolutely massive amount of petrol that can often cause a map to generate with nothing but petrol stations as consumers, due to the code attempting to fill in the huge oversupply. I've been testing changes to the pak128-britain pakset on my own and have found it relatively easy to tweak production rates so that they're not wildly out of balance, but am concerned about whether it may break with the more realistic intent baked into the current figures, so would love some input here.

[jamespetts]

On the topic of the code now controlling the number of consumer industries, this was done mainly because if I kept to the normal behaviour of targeting a total industry count, it became very difficult to adhere to that target, as adding one consumer might add dozens of supporting industries. At least with targeting a specific number of consumer industries, the game will always add the same number of consumers. Unfortunately I am not sure how to change the UI to match right now, as finding the relevant places in translation files seems very opaque to me....

A lot of the Simutrans code is somewhat eccentrically organised, which does make it harder to understand than it might be.

To find the location of a game text, have a look at the en.tab file. If you search for "Number of industries:", you will find that this translates to "No. of Factories". The format of the file is to have the original text on one line and the translation on the immediately following line. If you then search the code base for "No. of Factories", you will find an entry at line 209 of welt.cc, which sets the "factory_count" member of the settings class, defined at line 109 of settings.h.

Changing only the English translation of this to "Number of consumer industries:" is problematic because all the other languages will still have a text that means "Number of industries," and it is not satisfactory i9f asking for 6 industries results in 40 industries being generated, as can easily be the case. We need, I think, either to change the base text, or, if possible, modify the code so as to generate (at least approximately) the total number of industries requested, although this may be difficult.

As another note to place on the long list of potential scope creep here, this change may require pakset rebalancing to be done, as it has revealed some major imbalances in production rates that presumably were not a huge deal before now. Certain industries, especially from the year 1945 onwards, have production or consumption wildly out of pace with their associated producers and consumers. This is most noticeable with supermarkets, which require hundreds of crates a month of various finished goods that result in hundreds upon hundreds of farms spawning, and on the flip side, oil refineries produce an absolutely massive amount of petrol that can often cause a map to generate with nothing but petrol stations as consumers, due to the code attempting to fill in the huge oversupply. I've been testing changes to the pak128-britain pakset on my own and have found it relatively easy to tweak production rates so that they're not wildly out of balance, but am concerned about whether it may break with the more realistic intent baked into the current figures, so would love some input here.

This does get complex. The principle is always that calibration of all numerical uses real world values where available. If using real world values does not get real world behaviour, instead of fudging the numbers, the principle is to understand why there is a disconnect and fix it by simulating the thing in reality that gives a different outcome to that observed in the game. In this case, that is likely to be importing/exporting, and importing/exporting is a planned eventual feature. 

[jamespetts]

I have now had the opportunity to run network stability testing. Unfortunately, I find that the new industry generation branch on Github is not network stable: in other words, in a simple test game, I cannot maintain synchronisation to a server. Testing with the master branch with the same saved game, I am able to maintain network synchronisation.

The test procedure is to load the saved game as a server (check the box in the load game dialogue), and then run another instance of the same binary executable, open the load game dialogue, type net:127.0.0.1 in the text box and click "OK" or press enter. This will connect the second (client) instance over the loopback interface to the first (server) instance. The client instance needs to be able to remain connected to the server instance stably for a long period. With the new industry generation branch, unfortunately, we get a loss of synchronisation with the server after a short period of time in my tests.

Network synchronisation is critical, so this does need to be addressed before I can integrate this. Unfortunately, this sort of bug can be very hard to diagnose.

[martin509]

A lot of the Simutrans code is somewhat eccentrically organised, which does make it harder to understand than it might be.

To find the location of a game text, have a look at the en.tab file. If you search for "Number of industries:", you will find that this translates to "No. of Factories". The format of the file is to have the original text on one line and the translation on the immediately following line. If you then search the code base for "No. of Factories", you will find an entry at line 209 of welt.cc, which sets the "factory_count" member of the settings class, defined at line 109 of settings.h.

Changing only the English translation of this to "Number of consumer industries:" is problematic because all the other languages will still have a text that means "Number of industries," and it is not satisfactory i9f asking for 6 industries results in 40 industries being generated, as can easily be the case. We need, I think, either to change the base text, or, if possible, modify the code so as to generate (at least approximately) the total number of industries requested, although this may be difficult.

Thank you for finding the relevant place in the code. I'll see about getting a translation in for all the languages in the files. Unfortunately I am not a polyglot so I am thinking I will probably have to rely on Google Translate, comparing my results with whatever the translation for 'consumers' (as seen in the industry list screen) is to make sure I'm on the right track.

This does get complex. The principle is always that calibration of all numerical uses real world values where available. If using real world values does not get real world behaviour, instead of fudging the numbers, the principle is to understand why there is a disconnect and fix it by simulating the thing in reality that gives a different outcome to that observed in the game. In this case, that is likely to be importing/exporting, and importing/exporting is a planned eventual feature.

So far, in terms of issues I've seen in the pakset, many of them can in fact be tracked down to unrealistic or arbitrary numbers, both in terms of low consumption and in some industries not producing enough. The planned addition of imports/exports would be nice to have but does not impact the overall balance super greatly, as from what I can tell even just adding more realism would help a lot. I'd post a writeup here (the short version is that agricultural yields don't reflect modern agriculture, some end consumers also have unrealistically low consumption, and supermarket consumption in particular could probably stand to be roughly halved) but I will probably save it for when I think the changes I have in mind are ready for a PR to the pakset.

I have now had the opportunity to run network stability testing. Unfortunately, I find that the new industry generation branch on Github is not network stable: in other words, in a simple test game, I cannot maintain synchronisation to a server. Testing with the master branch with the same saved game, I am able to maintain network synchronisation.

The test procedure is to load the saved game as a server (check the box in the load game dialogue), and then run another instance of the same binary executable, open the load game dialogue, type net:127.0.0.1 in the text box and click "OK" or press enter. This will connect the second (client) instance over the loopback interface to the first (server) instance. The client instance needs to be able to remain connected to the server instance stably for a long period. With the new industry generation branch, unfortunately, we get a loss of synchronisation with the server after a short period of time in my tests.

To test this out, I have booted up a local server with that method using the provided savegame, but unfortunately (so far this is after leaving the game running for two months and building a few small lines with a total of three clients running, with all clients running in -debug mode) I did not get the same loss of synchronization. If you could provide a simu.log file of such happening on your machine I'd be happy to look at it.

On another note, I have started looking at the code for savegame versions. All of the relevant code for adjusting industry densities based on game version appears to be based off major game versions. I am not sure how big of a deal incrementing that version number to 15 will be. In terms of approaches for savegame correction, I am leaning towards one of two approaches: either apply a fudge factor of say, dividing industry density proportion by 3-4, or recalculate the industry density and set that as the new target proportion. The first approach would probably cause some amount of new consumer infill, while the second might lean more towards retiring excess production. I will defer to you as to which approach you would prefer (same with the version increment).

[ceeac]

You can use the -heavy 2 command line parameter to diagnose desyncs, then use the -compare command line parameter to check which objects are not in sync - this might give a clue about the cause.

[Matthew]

I've done some testing of this branch over the last week and I noticed two things.

Translations

Unfortunately I am not sure how to change the UI to match right now, as finding the relevant places in translation files seems very opaque to me.

Changing only the English translation of this to "Number of consumer industries:" is problematic because all the other languages will still have a text that means "Number of industries," and it is not satisfactory i9f asking for 6 industries results in 40 industries being generated, as can easily be the case. We need, I think, either to change the base text, or, if possible, modify the code so as to generate (at least approximately) the total number of industries requested, although this may be difficult.

James is right that the UI text needs to be changed to reflect the fact that it's now the number of consumer industries that is being set, not the total number of industries.

In the future, I might update the help page for this window to explain why we are setting the number of consumer industries specifically. But there's no point starting that until the patch is committed and I know for certain what behaviour we are documenting. The current English help text actually refers to the number of chains, not the number of industries, so it's arguably a better description of the new behaviour than the old behaviour.

Thank you for finding the relevant place in the code. I'll see about getting a translation in for all the languages in the files. Unfortunately I am not a polyglot so I am thinking I will probably have to rely on Google Translate, comparing my results with whatever the translation for 'consumers' (as seen in the industry list screen) is to make sure I'm on the right track.

I wonder whether there are some crossed wires here, perhaps caused by the fact that the translation process was not documented from the developers' point of view until recently.

If I have understood the process correctly, there are two steps.

Firstly, in this case, change the base text you can see in the C++ source code to something reflects what the code is doing (from "No. of Factories" to e.g. "number_of_consumer_factories"). That's an integral part of the patch, so is definitely your (Martin) responsibility. The relevant line appears to be /gui/welt.cc:L209.

Secondly, add that base text to SimuTranslator, translate into languages that you write fluently, and add an explanatory note for translators into other languages. Translating the text into other languages is not your responsibility (and it's actually a bad idea; translations should always be added by expert users).  I already have SimuTranslator rights so I can easily carry out this step if that's helpful.

If you want to learn how to carry out the second step yourself, there is now one page explaining translations from a developer's point of view (material for translators is now on another page). I made some changes to it over Easter, so if you didn't find it helpful before, maybe try again.

Industry numbers

I have been generating a few test maps and they seemed to show the results that we want. Here are the results for 256x256 maps with "No. of industries" as 24, using just-in-time mode 4, in 1750, 1830, and 1930:



You can see that we get a much better balance. Particularly at later dates, I saw a few manufacturers with many suppliers and consumers, instead of a few manufacturers each massively overproducing for a single consumer.

I am trying to test bigger maps, but it's a slow process. But I thought these initial results might be useful if you are trying to calculate a fudge factor.

[jamespetts]

Thank you all. First of all, I am re-testing the network synchronisation with the latest changes to the code and it now appears to stay in sync stably, which is definitely a good start.

Secondly, thank you to Matthew for explaining the translation process: that is helpful. Thank you also to Matthew for the testing, which is also helpful.

So far, in terms of issues I've seen in the pakset, many of them can in fact be tracked down to unrealistic or arbitrary numbers, both in terms of low consumption and in some industries not producing enough. The planned addition of imports/exports would be nice to have but does not impact the overall balance super greatly, as from what I can tell even just adding more realism would help a lot. I'd post a writeup here (the short version is that agricultural yields don't reflect modern agriculture, some end consumers also have unrealistically low consumption, and supermarket consumption in particular could probably stand to be roughly halved) but I will probably save it for when I think the changes I have in mind are ready for a PR to the pakset.

That is interesting. The intention has always been to balance the pakset using real world data, but it is possible that some mistakes were made or data were unavailable.

Are you able to start a separate thread in the pakset subforum showing which industries that you have found to be wrongly calibrated and what the correct calibration is?

As a reminder, the calibration is done in the short time scale and assuming that every day has 16 active/productive hours - so, for example, a factory that produces 1,000 widgets a month in real life would be calibrated as follows: assume 5 days/week, giving 260 active days/year; 12,000 widgets per year / 260 = 46.15 widgets/active day, assume 16 active hours per day to get 2.88 widgets per active hour, * 6.4 (Simutrans-Extended hours/month) to get 18.46 (rounded down to 18 ) widgets per Simutrans month. Refineries, power stations and other industries that run constantly would be calibrated to 7 days/week and 24 hours/day instead of 5 and 16 active hours as above. 19th century factories should be calibrated to 6 days/week and farms to 7 days/week.

On another note, I have started looking at the code for savegame versions. All of the relevant code for adjusting industry densities based on game version appears to be based off major game versions. I am not sure how big of a deal incrementing that version number to 15 will be. In terms of approaches for savegame correction, I am leaning towards one of two approaches: either apply a fudge factor of say, dividing industry density proportion by 3-4, or recalculate the industry density and set that as the new target proportion. The first approach would probably cause some amount of new consumer infill, while the second might lean more towards retiring excess production. I will defer to you as to which approach you would prefer (same with the version increment).

We need to use a minor version for this as the 15 major version is reserved for the 15.x branch - but that is not difficult. You just need to increment the Extended revision in simversion.h and use loadsave_t::is_version_ex_less(14, x) where x is the number of the revision after incrementing to detect whether you need to run the recalibration of the industry density on loading.

Thank you again for your work on this - it is most helpful.

[martin509]

That is interesting. The intention has always been to balance the pakset using real world data, but it is possible that some mistakes were made or data were unavailable.

Are you able to start a separate thread in the pakset subforum showing which industries that you have found to be wrongly calibrated and what the correct calibration is?

As a reminder, the calibration is done in the short time scale and assuming that every day has 16 active/productive hours - so, for example, a factory that produces 1,000 widgets a month in real life would be calibrated as follows: assume 5 days/week, giving 260 active days/year; 12,000 widgets per year / 260 = 46.15 widgets/active day, assume 16 active hours per day to get 2.88 widgets per active hour, * 6.4 (Simutrans-Extended hours/month) to get 18.46 (rounded down to 18 ) widgets per Simutrans month. Refineries, power stations and other industries that run constantly would be calibrated to 7 days/week and 24 hours/day instead of 5 and 16 active hours as above. 19th century factories should be calibrated to 6 days/week and farms to 7 days/week.

Thank you for the note on industry calibration. I have been drafting a post in the pakset subforum on the topic but it has been taking a while - the short version of it is that some industries do not have any particular realism-based justification and as a result can either be freely adjusted without particular regard to real-life conditions (such as builders' yards and supermarkets) or can be made more realistic to good effect (such as petrol stations and car dealerships). In some cases some amount of fudging is necessary as late-game manufacturers are very large and can't otherwise feasibly be saturated without either a very large map or, I suppose, the planned import/export system.

On converting from real-time to simutrans time, this has explained some particular decisions I saw (and prompted a lot of rewriting what I already had for some industries) but also lead me into one specific conclusion, which is that the livestock-generating industries are very hobbled from producing in simutrans time. I have done some experiments with basing their production instead on real time, and am of the view that it plays much better without sacrificing a realistic feel.

We need to use a minor version for this as the 15 major version is reserved for the 15.x branch - but that is not difficult. You just need to increment the Extended revision in simversion.h and use loadsave_t::is_version_ex_less(14, x) where x is the number of the revision after incrementing to detect whether you need to run the recalibration of the industry density on loading.

Noted!

[martin509]

I have been drafting a post in the pakset subforum on the topic but it has been taking a while [...]

Here it is!

[jamespetts]

I have now tested whether this merges cleanly with the ex-15 branch, and I can confirm that it does (the merged version for testing is on the ex-15-new-industry-generation branch).

All that this needs now is the code for adapting older saved games (and testing whether that works in the ex-15 branch) and this should be ready to incorporate.

Thank you for your work on this so far.