Consumption-centric industry growth: the return

[martin509]

Anyways, all this to say - this project is not dead, and work continues! The savegame compatibility code should be on GitHub now for you to peruse.
- martin509, well over a year ago

Whoops!

Ultimately this ended up on the backburner, ~95% finished, and getting a new job sapped most of my free time to devote to this project. However, I kept adding additional tweaks and bug fixes, and am now very confident that the consumption-centric industry generation implementation is all but done. Eventually there was discussion about rolling this feature into the 15.0 milestone, but I ended up disengaging, not wanting to push back 15.0's release date given all the review it might have entailed if I threw in.

However:
- The novelty and fun factor didn't wear off for me - it became a must-have feature of Simutrans, something that made playing freight lines feasible at all, and also just.. generally very fun to explore all the links the system would generate on the way to making a balanced distribution of goods.
- Other players expressed interest in the patch, notably Matthew, and confirmed that they enjoyed playing with it as much as I did.
- Upon reopening the github, I realized that there was a theoretically mergable port of my patch into the 15.0 branch sitting around that James had made.
- Extended 15.0 development was revived a few weeks ago, thanks to the integration of an AI bot working round the clock on bug fixes and performance improvements on both branches.

So as a result I decided to finish the final stretch and make a PR:

https://github.com/jamespetts/simutrans-extended/pull/686

To sum up, since the last time I had worked on everything:

- Savegame backwards compatibility is fully functioning
- Industry generation now accounts for competing suppliers when scaling upstream supply to downstream consumption (I can go into more detail but the main thing is it was a very complicated and fiddly load-balancing algorithm)
- Various bugs ironed out

As far as what's up for review, the biggest thing I know I want to run by others' opinions is how savegame compatibility will be handled. Currently, upgrading a 14.x savegame to this new industry growth system causes a massive burst of infill of consumer industries in order to resolve bottlenecks. I can go into much more detail as to how this algorithm works in particular, but essentially it tries to find a middle ground between keeping the world unchanged and completely saturating every industry chain with consumers. While this was a natural thing to implement at the time for myself, I realize not everyone might desire this kind of step change in their ongoing games, so I would like to hear opinions.

In addition, the matter of translating the "number of consumer industries" is still unresolved. Whatever is the cleanest resolution to this that needs the fewest number of people involved is probably the most desirable, because getting someone to translate a single line of text for every single supported language could be messy.

Finally, the last issue I see is the potential for pak-side compatibility. At the time, I noted that some consumer industries had arbitrarily-set consumption rates, particularly in the modern eras, and this especially caused massive unwieldy spam of hundreds of farms to feed supermarkets that consumed unrealistically gigantic  amounts of goods (iirc 160 crates/month per good). This resulted in a side quest to rebalance industry consumption figures to be somewhat realistic in pak128-Ex. I now have this more or less fully implemented on a local copy of pak128-Ex, touching most of the consumer industries in the pakset and some of the producers, and can make a PR to pak128-Ex if requested, but obviously it is not in scope of discussion here beyond noting that I would really prefer if both changes were merged together.

Anyways, as I am only just picking this project back up again, I would like to know if this whole feature will be feasible for 15.0 integration at all. While there aren't any merge conflicts with the branch and the pace of development has picked up recently, if it's still too significant an effort to review, I 100% understand leaving things as they are.

Thanks,

- Martin

[jamespetts]

Thank you for that: that is very helpful. I have been reviewing this and have merged it into a sub-branch of ex-15 for testing, being ex-15-industry-growth-test.

Given that you have done the work of doing this, that it does not affect the planned ex-15 work, that it addresses longstanding problems and that AI review has greatly reduced the workload of integrating new code, I intend to merge this into ex-15 before release.

Before I merge this into the main ex-15, however, please just check that none of the bug fixes and other changes identified/implemented b the AI are against your intention. 

Thank you again.

[martin509]

Thank you James. I will be going over the AI's review notes in the branch soon and will make a post addressing them, and will test the branch it made on my end. I will also see about getting a PR up for the pakset-level changes.

[jamespetts]

Thank you James. I will be going over the AI's review notes in the branch soon and will make a post addressing them, and will test the branch it made on my end. I will also see about getting a PR up for the pakset-level changes.

Splendid, thank you. 

As to the pakset level changes, very, very great care is needed there to ensure that everything is fully consistent not only internally but also with the scaling system. 

[martin509]

A small update: I have managed to get compiled releases out on the Github pages for both the code-side rework and the pakset rework. If anyone reading has been casually interested in this project and want to try it out for yourself but didn't have the ability to compile the game, now is a good opportunity! All you need to do is copy simutrans-extended.exe and the Pak128.britain-ex folder into your Simutrans game folder.

https://github.com/martin509/simutrans-extended/releases/tag/Nightly

https://github.com/martin509/simutrans-pak128.britain/releases/tag/Nightly

The code-side rework uses a copy of the AI review branch James made, though it doesn't auto-update so I will have to manually recompile if further changes are made. I wasn't able to figure out how to get the nightly run to compile pakset and all into one folder like the main repository does, but it's not needed.

I'll be putting up a PR and thread explaining the pakset-level changes later. Not quite everything is annotated with math for how the scaling works out, but most of it is.

[Milko]

Splendid, thank you.
As to the pakset level changes, very, very great care is needed there to ensure that everything is fully consistent not only internally but also with the scaling system.

Hi James, great to see you back on the forum

Do we run the risk that, in the future, a pakset might have to be split into two versions, one for version 14 and one for version 15?

Giuseppe

[martin509]

Hi James, great to see you back on the forum

Do we run the risk that, in the future, a pakset might have to be split into two versions, one for version 14 and one for version 15?

Giuseppe

As far as the changes being considered for here, there is no chance of them breaking compatibility with version 14 as I am using existing properties of industries to rebalance them.

I would have to do a more in-depth review of the version 15 pakset to know for sure, but I don't believe any properties have been removed from the objects, only added, which means that a version 14 game engine would just ignore the new properties added and load the pakset normally.

[jamespetts]

Hi James, great to see you back on the forum

Do we run the risk that, in the future, a pakset might have to be split into two versions, one for version 14 and one for version 15?

Giuseppe

Version 15 will in any case require a pakset with significant additional parameters than 14 given its additional features (fuel cost, staff cost, etc.): just as with previous version changes, the plan is simply to continue development towards 15 and, when that is released, that will simply be the latest version and the pakset will be for it. People will always be free to download the sources from an earlier checkpoint on Github and compile them, of course, but I do not plan to support legacy versions once the new version of 15 is fully released. 

And thank you for the welcome back!

[martin509]

Some notes from concerns raised by the AI's review in industry-generation-rework.md. I will ignore the notes where the AI is positive or neutral and focus on questions and concerns:

1. Producers in proportion per distribution_weight regresses: "producers are removed from density accounting entirely, so the weight no longer governs them. Whether demand-derived producer counts are the intended replacement is an open decision"

This is inaccurate. Previously, producers were already always being filled in to absolute maximum capacity upstream regardless. Distribution weight for producers only affected choices between multiple producers of the same good when filling in capacity (e.g. if livestock production is being filled in, distribution weight affects the proportion of sheep farms to cattle farms). This is unchanged, the main change is that producers are now filled in proportionally to downstream consumption. Distribution weight of consumers drives the overall makeup of the economy and which goods chains are most prominent, while distribution weight of producers and manufacturers drives the exact path the economy takes to produce those goods.

2. Balance-critical prerequisite "Towns generated to serve industries" (project-roadmap): "Worsens exposure: free producers plus map-wide placement generate more rural upstream industry, while the feature that would give those industries a workforce is unimplemented"

This is false. This patch generates *much less* rural upstream industry in most situations. As an example: In the previous system, let's say the game spawns a pub. The pub needs a beer brewery to feed it. The pub needs 10 crates of beer a month, but the brewery produces 100, and takes in 200t of grain to do so. Under the old system, the map would then spawn in enough grain farms to produce 200t of grain, 90% of which goes unused. Under the new system, the map only spawns in enough farms to produce enough grain to produce 10 crates of beer, which results in fewer upstream industries in general. Despite the fact there was a hard limit on all industries in total during mapgen, this limit was ignored when filling in goods chains, so producer industries were effectively free no matter what. As a result, this patch cannot run into a situation where it spawns more rural industries than normal (although, obviously, a map set to generate 12 *consumer* industries will generate much more than a map set to generate 12 industries).

On to questions, though obviously I'm not the maintainer and don't have final say:

Q: "Should producer industries remain governed by distribution_weight, or are demand-derived producer counts the intended replacement?"
A: See point 1. Producer industries weren't really governed by distribution-weight to begin with. Backfilling production based on the overall distribution of consumption is, IMO, more realistic and more easily comprehended than trying to replicate the entire makeup of the economy in one go.

Q: Should the basis change be accompanied by pakset-side recalibration of distribution_weight, and how are pakset authors told the weights' meaning changed?
A: Pakset recalibration is already a project, but also this patch brings the generated industries overall closer to current intent of distribution_weight. See point 2 for an example of why.

Q: Savegame basis: bump EX_SAVE_MINOR now and again when town growth lands, or make the persisted representation survive the second change (basis tag, or derive at load and persist nothing)?
A: Defer to the maintainer on this but as there are effectively no 15.0.0 saves in existence beyond a few developer debug saves, I am of the view that EX_SAVE_MINOR should be kept at 0.

Q: Should the durable upstream-balancing half be split from the consumer half that town growth will displace?
A: Not sure this is a valid question. Currently in game, if one founds a town and increases its population over and over using debug tools, it actually will spawn in new consumer industries alongside monuments and etc. I will need to read the code more closely but as it stands I think that feature already at least partially exists.

Q: Does recalc_idp()'s dependence on factory production figures settle before it runs at load? Finance history and fab_list are loaded first, but fabrik_t::get_monthly_production may depend on state settled later; an in-process server reload would mask the difference → sync-and-determinism. Not verified.
A: As far as I can tell the value returned by get_monthly_production() does not change over time for a given industry, so it should be fine for synchronization. Worth checking.

Q: Is the reduced industry churn acceptable for the whole period until ports/import-export exist, given that feature is unscheduled?
A: Defer to maintainer, but as a player I massively prefer eliminating churn that happens due to poor industry generation. It isn't fun and doesn't logically correlate with how real life works. If we decide churn is a desirable feature later on, then I think it should be implemented as a direct, intentional feature.

Q: Should the tolerance bands be simuconf/pakset parameters before merge (see the ports row above)?
A: The +/- 12.5% tolerance bands were copied from existing code that handled non-just-in-time goods. I am not opposed to breaking this out into a simuconf parameter as a way to make the industry distribution more chaotic but am not particularly attached, either. This game already has a lot of obscure simuconf parameters no one uses that this would be buried with.

Again, the maintainer obviously has final say on these questions, so these are just my opinions as the guy who wrote the code and has played a good few games with it enabled.