Potential Paradigm change for supply chains

[PJMack]

I was not going to announce this until I was a little further along as I was not entirely sure this is going to work, be worth the effort or even have time to complete it.  Due to the number of forum threads on industries, I think it may be worth mentioning.

In a separate branch, I have been working on a project to have industry links contain a monthly contract to be used for distribution of goods and balancing supply and demand.  For distribution, goods would be shipped at the rate specified in the contract to stabilize good departures.  Max intransit values would be used as an extreme cap to verify that not too many goods have been shipped due to a sudden lead time reduction or loss of consumption rate, however instead of stopping shipments entirely, contracts would be reduced according to the excess en route.  Contracts would only be assigned such that the total contracts for a supplier cannot exceed what the supplier can produce, allowing the actual surpluses and shortages to be quickly calculated, rather than the heuristic methods currently in use.

This would be a major undertaking and require redoing the step function as well as nearly all of the factor builder functions for increasing industry density and appending supply chains.

So far, I have the code for initializing contract arrays, displaying them in the GUI, and renegotiation every month.  I am trying to keep everything backwards compatible so that just_in_time zero through four could still be used.  I have pushed the branch to https://github.com/PJMack/simutrans-extended/tree/factory_link_rebalance

[wlindley]

Seems sensible.  Also either party to a contract could be "unhappy" enough, weighted against an "affiliation level," to switch to a competitor.  And manually-created industries would perhaps even choose new links likewise... altho achieving a balance with some consumer shops selling a few high-volume items while other shops specialize in (higher-priced) specialty items (from further away) seems difficult

[jamespetts]

Thank you for looking into this: this looks interesting, but I am not entirely clear on how you envisage that this would work.

Can you elaborate a little on how this would work differently to what we already have now? Do you envisage that this would mean that any industry could supply any other industry of a compatible type, and/or that these supply contracts may come to an end at some point even if both industries remain on the map?

How precisely would this new abstraction layer of a contract work, and what exactly would it represent? Would you envisage the contract rate being displayed to the player in some way?

Thank you again for your work on this.

[PJMack]

The reason I started this was due to two issues I have been having with industries.  The first is that shipments leave factories in rather large batches every several months rather than a constant stream.  The second is that due to industry upgrades and closures, maps can become unbalanced in such that some suppliers are connected to too many consumers while others are connected to too few.

What I envision is that with each industry link, there is a contract as to how many goods to ship per month from supplier to consumer, and the supplier would ship them evenly thought the month.  For example if colliery X has a contract to ship 12 tons to coal merchant Y, It would release 1 ton destined to that coal merchant every 1/6th 1/12th of a month that it has coal.  If colliery X also 60t ships to ironworks Z, it would also release 1 ton ever 1/60th of a month to the ironworks.  Each contract would be displayed to the player, as well as the total number of contracts per each good type, allowing players to know how many goods need to be transported each month along a particular route even if both ends have multiple cross connections.

As it is now, each industry would only be linked to a few compatible industries.  I am still ironing out the details to how contracts are negotiated, but what I have coded at the moment is to check monthly if the output has too many contracts and reduce them evenly amounts the existing links, then increase or reduce contracts for input if they have too few or too many contracts respectively.  This takes into account customer history for end consumers and would also be reduced inversely proportional to any overflows of consumer stock and maximum intransit value. 

In the event that a consumer cannot get enough goods per month using existing links (possibly after a few tries, I have not decided yet), then the consumer would try to find and link to a supplier(s) with enough spare production capacity.  Factories with missing links would also use the spare production capacity to pick the best supplier to link to, rather than simply picking the most recently built one.  Again, I am still thinking about the details.

Currently, when it is time for the map to using, a set of functions in the factory builder creates a new chain using heuristically found supply and demand information.  I plan on replacing the heuristic algorithms with ones using the total contracts to determine the actual supply and demand quickly and more accurately. 

[jimishol]

if colliery X has a contract to ship 12 tons to coal merchant Y, It would release 1 ton destined to that coal merchant every 1/6th of a month that it has coal.

Do you assume a contract as 12tons per 2 months? I cannot explain the 1/6th of a month.

As player, i assume that consumer wants max delivery equal to his storage. An example: Let demand be 12tons/month and storage is 8t. Then consumer demands, the less, 3 deliveries, of 8tons each (3*8=24), every 2 months. He would not accept or it will not be good if player delivers fewer than 3 times per 2 months. It is bad if he delivers 12tones once a month, as 12t is larger than his storage. But would be totally acceptable if player delivers 6 deliveries, of 4tons each, every 2 months (or equivalently 3 deliveries of 4 tones once every month). I do not know how just_in_time and maximum_intransit_percentage, in configuration, justify some lead time for deliveries but may be it is a good time to fix this too. The game is not possible to know how many convoys a player will use to service the demand. The more convoys he will use the lesser quantities will be delivered but with lesser delivery times. In the example above, player may chose to deliver 1ton every 1/12 of the month and that should be acceptable. If the purpose of the above configuration's variables is to protect the player from overstocking due to a possible luck of knowledge to plan accordingly, they will mathematically fail in all cases except when it happens the players to use the exact number of convoys that will create the lead time that the game assumes. The lead time of deliveries is a missing information and, as it depends on player's choices, it cannot be assumed, as by the expected duration of a trip (or twice that duration as the lesser of two evils) for example. An idea would be to record the last 3 or 4 deliveries for quantities and time, so as to take the average quantity per time. That can be compared to the demand and if it is greater then the overstock will happen for sure and source production might be shut off or low down.

I wish success in your efforts. Thank you.

[PJMack]

The "1/6th" was an error, it should have said "1/12th".  I corrected the above post.  I apologize for the confusion.

The maximum storage for a customer actually has nothing to do with consumption and little to do with output of the supplier, depending on the just_in_time setting.  The lead time is the time it takes for a good to get from supplier to consumer.  It is calculated during path-finding (using the Floyd-Warshall Algorithm) and does take into account actually past transit times (though I am not too familiar with the details).  What the current system does is use the lead time and the consumption rate for the consumer to calculate how many goods should be en route to that consumer.  Even if the consumption rate or lead time decreases, the arrival rate would still be relatively steady and equal to the rate of consumption, so the player would only need to be concerned about making sure there are enough convoy capacity to meet the rate of consumption.  What this new system addresses is that even though the arrival rate would be steady, the departure rate from the supplier is rather sporadic and unpredictable.  For the new system, goods would depart at a steady rate which is clear to the player.  The rate would only change if there is a change in demand.

[jimishol]

You are right. Consumption has nothing to do with storage. I only wanted to point that our goal is to satisfy the maximum of consumption we can, without alerting just_in_time about an overstock ahead.

From simuconf.tab
# This setting determines the behavior of industry demand.
# At 4, industries will demand goods whenever their internal storage does not have sufficient stock to last until the next delivery is estimated to arrive if the goods were to leave the producing factory now.
just_in_time = 4
# How much amount in transport is sent before further distribution stops
# This is only enabled when "just_in_time" is enabled (i.e. set to a value > 0)
# This number is scaled to the ratio of the time that it takes the industry to
# consume its stock to the average lead time for new deliveries. Values of
# slightly over 100% are recommended.
maximum_intransit_percentage = 110

Concerning the output input balancing, we see that as stock is crucial to just_in_time such the storage has the same significance as just_in_time. Unfortunately, I have not yet a confirmation if the description above is literal or not. I take it literal and, as such, is fundamental wrong in principle. It does not fix overstocking from bad players' schedules, if a player takes into account what the description says, it creates overstock on its own and can cause unbalance by motivating developers to alter balanced values or invent new ones, making the game more complex than it need to be, in tries to fix unbalanced phenomena they observe caused by just_in_time=4 at least.

The parameters of the problem is the number of convoys in service, the quantity of goods each convoy carry, the demand and the lead time (that includes convoy's velocity and the distance it travels). It is clear, and in some posts or wiki is mentioned likewise, that just_in_time  tries to control and prevent overstocking by controlling the amount of goods in transit. It is fundamental wrong because we do not care about  the amount of goods in transit but we care only not to overstock storage. That is, by using the description, we care about "industries to demand goods whenever their internal storage does not have sufficient stock to last until the next delivery is estimated to arrive" and not "whenever their internal storage does not have sufficient stock to last until the next delivery is estimated to arrive if the goods were to leave the producing factory now."  The missing information that makes the method wrong is the number of convoys. Floyd-Warshall Algorithm and lead time do not include that information. Low speed convoys have to be in great numbers and goods in transit could be rightfully huge, while just_in_time will falsely be alerted. The stock could be small and  just_in_time will never be alerted when she should.

Say lead time as "L". Each convoy between its arrivals will be in travel "2*L" time because it needs "L" to return to source and "L" time to come for delivery.  If "n" convoys" moves simultaneously, in "2*L" time they need to deliver "2*L*demand" goods. But "n" convoys, if each carries "q" goods, carry "n*q" goods. So, for balance we need "n*q=2*L*demand". Half of them will be empty in their way to return to the source. So, goods in transit will be "n*q/2". But that equals to "2*L*demand/2=L*demand". We ended at the official description of  just_in_time=4. It is exactly the right answer about the quantity of goods in transit, but is fundamental wrong to claim we achieved balance. Why? Because by eliminating the "n" parameter, we answered the wrong question.
Go back to where, for balance, we need to be valid that "n*q=2*L*demand". If we spread equally in the time of "2*L" the "n" convoys, each arrival will happen in "2*L/n" time. Each convoy carries and delivers "q" goods, so for balance, that is to not overstock, we need "q=storage". That is "q/demand=storage/demand=2*L/n". In other words, the "q=storage" goods, to be consume just in the time of "2*L/n". Or in official words " industries will demand goods whenever their internal storage does not have sufficient stock to last until the next delivery is estimated to arrive." (dot, full stop here).

We know "L". I see your work as good opportunity, by recording few recent arrivals, to know "2*L/n" too, so as to compare it with "storage/demand" for input – output balance.

maximum_intransit_percentage equals to "storage/demand/L". Official description considers it falsely 1 or for safety 1.1. I, and hope we, see it should be equal to "2/n" and may be for safety equal to "2.2/n" where "n" will be estimated by your work. We see that only when players use in each line exclusively 2 convoys the  just_in_time=4 works as it should.
Thank you.

[PJMack]

I have gotten to the point where this branch is playable. 

By default, the entire contract system is disabled.  To enable it, use the settings window (keyboard shortcut "i") to change just_in_time to 5, then close the window.  That will initialize all contracts to zero.  The contracts will be renegotiated at the start of the next month, and may take a few months to stabilize.  Setting just_in_time back to 4 (or any value other than 5) will immediately *delete* all contracts, but should allow the game to be saved so that it can be opened on the jamespetts/master branch. 

As stated previously, the basis of the system is that each link between industries has a contract for each how many goods per month need to flow between them, and that many goods per month are shipped when available.

Each industry audits its contracts at the start of each month and renegotiates them if necessary.  I put a detailed description of the algorithm below.  I am not entirely sure I am completely happy with it from both a performance and game-play balance perspective.  On an optimized build, I am getting lag at the start of each month, however the lag does not persist on the same map for the debugging build!?  Only with long term game-play could balance be properly evaluated, and that can be a matter of opinion. 

The algorithm for contract negotiation is as follows.  For each industry, it is has production capabilities and production contracts for more than its production capacity, it will evenly reduce the contracts to its linked consumers with the aim of having the total contracts being equal to its production capacity.  If that producing industry's total contracts is less than one eighth of its production capacity, it solicits contracts to its linked consumers.  If the industry consumes goods, it calculates its expected consumption using past consumer data (for consumers), current staffing levels, current production contracts (for manufacturers), and current amount of goods in transit and in stock (which only reduces, not halts, the expected consumption).  If the total contracts is less than the expected consumption or more than 10 percent of expected consumption, it tries to adjust the contracts to existing links in two rounds, one concentrating on active and staffed producers, then all producers.  Such adjustments are done evenly with slight weight towards more nearby producers.  If that fails to produce a total number of contracts above the expected consumption, a third round is performed more aggressively such that it saturates the supply of each producer, starting with the nearest, until the total exceeds the expected consumption.  If it still cannot meet the expected consumption with existing links, it then will attempt to connect to other suppliers to fill the void.

[PJMack]

I have repaired the issue with the stats.

I also looked into the issues with that lagging, and found the culprit to be adding fields.  When testing in the debug build, few industry farm industry upgrades were occurring, however I switched to testing in the optimized build just as many field industries became obsolete.  There are essentially two issue, the first being that instead of building all fields then recalculating all the parameters, the recalculations are done for each field added (e.g. a few hundred more times more than necessary).  The slowest part of those calculations appears to be the karte_t::update_weight_of_building_in_world_list function which involved linearly searching through a weighted vector twice.  The second issue is that the weighted vector probably should be replaced something more efficient.  Neither of these are unique to this fork and branch and are present on jamespetts/master.

Edit: removing fields for industry upgrade has the same issue.

[PJMack]

I have turned this branch into a PR.  Although I would have liked to test this for longer, something  has come up (a milestone in life) which precludes me from continuing.  It does appear to be relatively balanced and ready though.  As game-play balance is subjective I would recommend extended testing to get a second or third opinion.  Note that the use of contracts is only enabled when just_in_time is set to 5 (it is 4 by default).

In addition to the features noted in previous posts, I also added one where factories that cannot find contracts for 12 straight months or are unproductive for 60 straight months (5 years) will close.  A cap of 16 factories per month can close for that reason, with priorities given to the ones unproductive and contract-less for the longest time.  I also increased the cap number of new industry chains per month from 4 to 8.  Older industry growth functions sometimes would create too many or too few industries of a particular type, and this would speed up the correction for games that started with those functions.

The fix for in inefficiencies in fields are in commit #07f1bdadf784a0632bade60bf07183c33d5a49cd, so if it is decided that contracts are not ready for the master, that commit should probably be cherry picked.

[Ranran(retired)]

In addition to the features noted in previous posts, I also added one where factories that cannot find contracts for 12 straight months or are unproductive for 60 straight months (5 years) will close.

In my opinion this feature is destructive and detracts from the gaming experience.
After all, simutrans and simutrans-extended are simulation games, and no matter how much simutrans extended tries to imitate the real economy, it is not an economic simulator.

For example, if a player tries to play a large map alone, buildings that fail to bring the industry to normal operation within 5 years will be shut down one after another.
It will take a lot of time to earn enough money to build a vast network across the map.
Also, I don't think it's possible to keep the factory running normally for a while to avoid the industry chain closing condition when the player finally establishes the chain connection.
Players who finally open a freight route in year 4 may see all their investment go to waste the following year due to industry closures. (It can take a long time to get goods to remote end consumers, and when it closes, the industrial chain breaks and transportation plans are ruined.)
Rail routes to remote coal mines are almost always no longer valuable once the mines are closed.
Therefore, for players to engage in freight transport involves even greater risks than now, and I suspect that everyone will avoid the freight transport business. When I was playing on the Bridgewater Brunel server, some people said that the extended industry chain was broken and didn't work and they didn't want to do it. And most players were devoted to passenger transport. Not sure how much better this is now.


Anyway, the reason why it does not work, in my opinion, is because the industrial construction installation does not correctly simulate reality.

For example, coal mines are placed where coal is buried. It was in great demand in the old days. So when it is found, people plan to transport it. Western countries wanted to supply coal in a land far away from their home country, so they made Japan build a coal mine and transport it. Coal mine construction is planned in consideration of coal demand, demand, and transportation routes.
Historically speaking, coal mines were often created along with the means of transportation. Transportation will begin as soon as the routes and industrial chains are completed. But in simutrans mines are not set up that way.
A worker town may be created nearby. I think james has plans to implement this feature.
In reality, jobless people may migrate from other lands if there are jobs available. People without jobs will not migrate in simutrans extended. They aspire to be transported or move on their own. If that doesn't happen they want to be unemployed.
Thus, I think, industries occur when there is some prospect of securing workers. Otherwise they are very stupid.
The simutrans industry does not take risks. So they may be doing stupid and reckless construction.

In the real world, industry closures are generally given advance notice to interested parties. But extended doesn't do that and abruptly closes. This is an act that is very annoying to the shipping company and does not provide any guarantees if it is done. In the case of online play, the player may be absent, and the network must continue to transport empty for a while and pay unnecessary costs. I think that is also the reason why players tend to avoid freight transportation.

There is a big contradiction with the real world like that, so I don't think it works well as a whole even if you imitate the real world only partially. Anyway my point is I think the conditions are still insufficient to make this feature work. Even if it is one of the options, it cannot be the target of the choice, but rather a trap that traps the player.
For example, there is a private wagon in James' implementation plan. It may also be a required feature.

[PJMack]

I have found in gameplay that this feature is more constructive than destructive as it removes nonviable industries to allow new ones to take their place.  It also helps to stabilize the supply and demand of goods which is critical for the contracts to work properly.  Any production resets time unproductive counter to zero.  If one were to make a rail route from a coal mine to the distribution network after four years, production would be enabled as soon as the first train leaves the depot.  Unlike with the previous just-in-time algorithms, shipment and production with contracts are stable and there will not be long periods of idle time between periods of maximum production.

You may be right though on the beginning map, however it is important to note that there is a cap to 16 industry closures per month.  This means that the larger the number of industries in the map, the less likely a particular industry would close if all industries were created at the same time.  For a map with 1600 industries eligible for closure, it would take 8 years to close them all.

The constants in the code probably could be adjusted or added to the settings menu (though adjusting settings in a branch is almost a guaranteed merge conflict).  One could also allow setting the unproductive time on a per industry basis in the pakset.

[jamespetts]

Thank you for this and apologies for the delay in looking into this. I have now incorporated this. I note Ranran's comments about industries closing down; however, since this is an optional feature, and one which I have not enabled by default in Pak128.Britain-Ex, can be avoided by the user in any event.

One thing that I do wonder is whether it is worth having a separate setting to disable this even if just_in_time = 5 is enabled; I should be grateful for any feedback on that issue.