The International Simutrans Forum

Development => Patches & Projects => Topic started by: DrSuperGood on July 06, 2014, 01:41:01 AM

Title: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 06, 2014, 01:41:01 AM
Currently Simutrans uses a very simple supply and demand mechanic referred to as "Just In Time" (JIT). Like all supply and demand mechanics, the idea it is to limit the amount of goods that go to certain industries to add challenge (cannot send goods just anywhere of the appropriate type) and realism (it makes no sense a that a tiny industry can process a huge amount of goods). However many people suffer a lot of gripe with the way this system works as it can be difficult to use and annoying under certain circumstances. The purpose of this post is to explain the existing system, discuss the operation and flaws with it and then suggest a possible new JIT system that tries to mitigate these flaws.

The current system is highly similar to water filling up a bucket. Each consuming industry has a storage capacity which mirrors some arbitrary height in a bucket that has potentially infinite volume (well it is limited to underlying integer types used). There is also a rate of consumption which can be viewed as a hole in the bottom of the bucket allowing water to escape at a certain rate. Supplying industries can be viewed as taps that fill up the bucket by producing goods destined to the consumer. When the industry holds less goods than its storage capacity, the bucket is running empty, it signals all the suppliers to start generating goods, the taps are opened and water flows into the bucket to fill it. When a consumer's current storage exceeds its storage capacity then it is considered full and the suppliers are stopped (the taps are closed).

However a big part of simutrans is the movement of goods, which requires time. This means that there is an "in transit" delay between when suppliers start producing (the taps opening) and the consumer receiving the goods (the bucket filling with water) which can be thought of as a long pipe the water has to travel through from the tap to reach the bucket. This also means that when the suppliers are closed due to the consumer being full (bucket is full so taps are closed) there is a delay as some goods are still in-transit (water is still in the pipes going to the bucket and will continue to drain into the bucket even after the tap is closed until the pipes are empty). This volume of goods in-transit (water in the pipes) is recorded and a maximum bound can be applied based on game settings and the storage capacity of the consumer.

This system seems to work on the surface. Place 2 connected industries near each other in pak64 (eg a coal mine and a coal power station) and connect them together with ample flow rate and the coal power station will operate all the time. However what if they were further apart? Since the coal mine produces at maximum rate a large in transit amount will be generated before the supplier stops due to over supply (very long pipe problem) resulting in large bursts of goods and extended periods where the supplier sits idle, potentially even for months. Now what if they were very far apart (eg 2000 tiles)? Now the maximum storage is no longer sufficient to keep the industry working between the time it starts running empty and the good arriving (the water pipes are so long that by the time the water arrives at the bucket, the bucket is already empty). What happens if they are yet further apart? Eventually the in transit amount is so large that the setting defined maximum will take affect (the taps will be turned off when the bucket is empty because too much water is in the pipes). In both long distance cases I refer to as the "bursty supply" problem since supply production and industry consumption go in huge bursts with a very low frequency (long period).

To avoid the burst supply problem there are currently three approaches. The most obvious is to "not care" and "let it happen" which places a lot of strain on transportation networks to cope with large bursts and results in inefficient consumers (the consumers are inactive for long periods, or have a low "duty cycle"). An improvement over this is to find the closest supplier and make a direct fast link with the consumer (make one pipe short) so that the time before goods arrive is less or even if consumption is slower to allow time for other goods to arrive (some water arriving sooner is a lot better than all water arriving later) which results in slightly improved consumer duty cycle but still does not help badly placed chains where even fast direct routes take too long for the consumer. The final approach, the "flow", is to carefully hand pick suppliers and create artificial bottlenecks (narrower pipes) so that just enough goods continuously reach the supplier (the bucket is always empty but flow rate in from the pipes approximates the flow rate out the bucket) which can result in very high duty cycle over any distance at the cost of extra route design and some micro management. All of these produce sub-optimal results as none achieve perfect duty cycle (100% consumption).

This means that the current system forces a maximum distance between suppliers and consumers where by it is possible to achieve perfect duty cycling of a consumer. This is based on the average speed that the goods can be moved from the supplier to the consumer and the amount of time that the consumer takes to fully empty the current storage from the maximum storage amount (if water can reach the bucket before the bucket is empty, all is well). This means it is not really a problem for a pak128 coal power station as it can run for many moths before storage empties (the distance restriction is huge) but is a huge problem for a pak64 bookshop which, when working optimally, can empty its storage about 10 times a month. Obviously maximum in transit amount also applies a restriction but for any reasonable multiplier this would place a restriction on the maximum efficient long haulage distance after which an extra duty cycle penalty is applied (turning the taps off when the bucket is empty will always be bad for average out flow).

So what fundamentally does a JIT system have to do? Using the above system and various game play elements as a source you can extract the following requirements.
- The maximum average amount of goods sent to a consumer is the consumers rate of consumption. This is done currently by turning off the suppliers when storage is full resulting in a sort of goods buffer that oscillates very badly, potentially falling apart over long distances.
- The storage capacity of an industry dictates its transport requirements. Supplying a huge 12 tile train of books to a tiny book shop is not ideal while supplying a 12 tile coal train to a coal power station is perfectly fine. This is not really implemented well as it makes fussy industries impossible to supply in a reasonably sized map.
- Some limitation on goods in transit is required to prevent "shipping to nowhere" problems resulting from bottlenecks and no restrictions to transfer capacities. This is currently done by the maximum in transit limit however this also places a maximum efficient transport distance so is not ideal.
- The most money should be earned when a consumer has a perfect duty cycle. If a consumer is consuming the most it can, you will be able to transport the most to it and the amount you earn is based on the amount you transport. This is a result of the JIT system, even fulfilled by the current system although not perfectly.
- The system has to be simple. Experimental solves this problem differently with complicated mechanics that no one fully understands without detailed technical knowledge. The current system is simple, easy to learn so should the new system be simple and easy to learn.

So how could JIT be improved? Well lets look at the flow approach described above and analyse it why it produces near perfect duty cycle over any distance up to the in transit maximum limit. Instead of suppliers working at full (taps open) and then turning off when consumer storage is full (taps close when bucket is full), a bottleneck is applied (narrowing of the pipes) meaning that the suppliers will be forced to produce at near the same rate as the consumer consumes the goods. This means that goods are always being produced and always moving in a continuous cycle. It even comes with a whole lot of benefits such as less platform space required, easier to design networks (no designing to cope with maximum flow rates) and more realistic looks. The problem is currently the bottleneck will result in "overcrowded" stations and also still never allow the industry to run at perfect duty cycle. Now instead of manual bottlenecks having to be used for a flow approach why not make the JIT system do it for you? If it did all would be solved and this is where to start with the new JIT system.

Based on the first point I raised, the maximum average amount of goods suppliers can produce towards a consumer is based on the consumption rate. This means that for every unit a consumer uses, a supplier should produce a unit to replace it. Since suppliers produce in batches of 10 units and finding an available supplier is not immediate lets define a "credit" buffer for each consumer. This buffer is incremented when resources are consumed by the consumed amount. When it exceeds the minimum supply amount (10?) all suppliers are turned on towards the consumer. Every time goods are sent from a supplier towards the consumer, the credit buffer decrements by the amount sent. When the credit buffer reaches 0 the supply of goods is shut off towards the consumer. You can even allow a slight over run into negative credit since eventually it will go positive over time. Obviously this value must be persistent so will have to be saved to the industry (it is not deterministic from other state information so makes up its own state). The entire system can now be viewed as taps that are rapidly opened and closed to give the exact same flow in as flow out the hole in the bucket.

The fundamentals of such a JIT system really are that simple complying with the final point. We only need a few more restrictions to implement the others points of JIT requirements.

A penalty is applied when the current storage of a consumer exceeds the maximum storage by "dropping credits". This is when only part of the amount consumed is added to the credits to generate new supply. Mechanically you can think of it as skipping opening the odd tap for a brief second so that the water level in the bucket drops a little. I would recommend making this an option setting with a default of either 50% or 75% re-credit so that the drop is gradual. This solves the second feature by applying a penalty (skipping goods, lost shipment opportunities) if a convoy with larger than storage capacity is used to supply the consumer. It also allows for in transit reductions as a result of line upgrades to fewer but faster convoys.

Obviously a consumer starts empty so the goods must come from some where. This would come from "gaining credits" when the consumer is inactive yet connected to suppliers. When gaining credits only a fraction of the consumption rate is credited so that the industry can start. One can view this mechanically as the taps starting being turned on too little and the amount flowing out observed and used as feedback to slowly maximize throughput. This also solves the third requirement as a bottlenecked supply may be unlimited (you never have to deliver the goods and could pile them up to stupid numbers in a transfer) but will be less profitable than properly shipping the goods to the consumer as you are only getting a fraction of the goods per unit time compared with a perfect duty cycle industry (bottlenecks mean less than 100% consumption rate and this is not an issue if any form of over crowding transfer restriction is in place). About 50% of the consumption rate sounds a reasonable amount but again this could be a game setting users can configure.

There is also the ability to detect a insufficient supply condition. If the credit is constantly growing (>>0) then there are not enough suppliers for the consumer. In this case it should cap out at some amount (credit at 25% of the storage maximum?) and obviously issue some graphic notification before then (credit at 12.5% of the storage maximum?).

There is a special case where by a consumer is a supplier as well and requires multiple goods in different ratios to produce products. To prevent over supply of certain goods there needs to be coupling across all their consumed goods (different from general consumers which they can all behave separately). When goods are consumed they are credited at the appropriate ratios based on the industry (mirroring their consumption rate). In the case of under supply of any good then all of them fall to gaining credits mode (to stop oversupplying of the others). Dropping credit mode behaves normally and separately for each good except that if a good is both in gaining credits mode (from another good being undersupplied) and dropping credit mode (from being over supplied) both penalties should be applies so that flows of that good are stemmed even further (possibly coming to a stop in the case of both being 50%).

Notice that there is now no importance placed to the in transit amount. It could potentially go to whatever the game allows it to. This means that even a stupidly huge map would still allow a flow of goods to a consumer to reach perfect duty cycle from a supplier the furthest possible distance away.

So how would this system behave in game? Here is an outline of the stages a simple coal/iron to steel mill to builder's yard supply chain may go through. Assume all were connected from a paused game state to be instantly and sufficiently connected.
1. Builder yard tries to consume steel in gaining credit mode. There is none at the still mill so slowly starts building credits up at 50%. Eventually goes to "under supplied" state and then the credit buffer maxes up at some value.
2. Steel mill storage is empty so it is operating at 100%. Since there are no goods it enters gaining credit mode for all goods.
3. Both coal and iron ore mine start sending goods slowly. The ratio of them being sent mirrors the steel mill need.
4. The coal mine was closer than the iron ore mine so starts arriving first however since none is being consumed it still remains in gaining state.
5. Coal shipments fill up the steel mill forcing it to enter dropping credit state so coal production comes to a stop.
6. Iron Ore finally starts to arrive, the steel mill beings producing steel and re-crediting is happing at 100% of iron ore but at 50% for coal due to still being in dropping credit state.
7. Iron Ore eventually runs out and so it falls back to gaining credit mode, however this does not last long as already enough iron ore has been introduced into the system to keep the steel mill at 50% production rate with any more from gaining mode raising it slowly to 100%.
8. The steel mill produces far too much steel for a single builders yard so the builders yard soon finds its credit buffer depleted.
9. The builders yard starts receiving steel fast due to the credit buffer but this helps bring consumption to 100% balance very quickly.
10. The steel mill is spending a lot of its time with supply storage full so inactive. As a result both iron ore and coal credits are being generated slower so shipments start to become less frequent.
11. The steel mill briefly enters a stage of dropping credits for both coal and iron ore again due to the consumption rate reduction and in transit amount causing it to fill up current storage (the in transit amount has dropped).
12. All industries sit at a equilibrium. The coal and iron ore mines produce just enough for the steel mill to produce just enough for the builders yard to keep consuming steel at perfect duty cycle. If you look at month graphs all appear to be constant with only some minor granularity variances based on shipping restrictions.

The builder yard could get connected to power, or have a good commuter/mail service to improve consumption but the steel mill will eventually adjust to compensate. If 5 other builders yards were connected to it there may be a under supply problem but the steel mill will still work at 100% and all the yards will all be under supplied state with maxed out credit buffer. The distance between any of the supply chain members does not mater.

I do apologize for the length of this post but it was required in order to give enough explanation behind why such a revision was necessary and how this could benefit the game. I strongly feel such a system would benefit game play of standard (Experimental already has a flow based system in place, although implemented entirely differently in a way not suitable for standard). I have met at least a few people on like who have said the industry model should be revised so after giving much thought to the mater I came up with the above suggestion. Obviously I might have missed things or not factored in certain mechanical limitations of the game but I did try and keep it as simple as possible to avoid complicated coding.
Title: Re: Suggestion for JIT mechanics revision.
Post by: jamespetts on July 06, 2014, 11:40:12 AM
This is a very interesting suggestion, but may I ask: how had you envisaged detecting whether industries are connected? Do you mean connected by transport here, or connected in the sense of being suppliers and consumers of each other?
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 06, 2014, 02:19:01 PM
Quote
This is a very interesting suggestion, but may I ask: how had you envisaged detecting whether industries are connected? Do you mean connected by transport here, or connected in the sense of being suppliers and consumers of each other?
Hopefully however it is done currently would be sufficient (how goods find their way from suppliers to consumers at the moment?). The underlying idea is to change the frequency at which this goods flow is turned on and off such that the overall duty cycle can reach perfection (consumer working 100% of time) and that goods are semi-continuously arriving and being produced like a flow from suppliers to consumers. This mechanic already happens if in-transit is maxed out or that a supplier loading bay is capacity bound and is how the flow approach does it, although this would make it a lot easier and standardly used.

Currently how it looks is goods are being produced in huge bursts (low frequency, some duty cycle) resulting in a consumer receiving huge bursts with possibly down time between (low frequency, poor duty cycle). A flow approach turns this into small bursts from suppliers (high frequency, some better duty cycle) resulting in consumers constantly receiving goods at a similar rate to their consumption (high frequency, near perfect duty cycle). The major difference is that the high frequency of arrival means that there is no long period of down time (loss of duty cycle) while goods are in-transit (the length of pipe no longer maters as water is continuously flowing through it).

Among other things advantages of a in-built flow system include...
No overcrowding, currently required for a flow approach to limit good production rate.
Less transport infrastructure required if a non-flow approach is used (the pipes can be narrower like with the flow approach without causing water to pile up somewhere).
Fewer convoys that are virtually never idle (where the "flow" idea comes from).
More goods moved on average (consumer duty cycle is better in a lot of cases where perfect duty cycle was not able to be achieved).
More smooth transport graphs, there will be no multi-month (low frequency) oscillations in profit and amount transferred as goods flows turn on and off.
Title: Re: Suggestion for JIT mechanics revision.
Post by: jamespetts on July 06, 2014, 02:43:28 PM
The current system does not give as much data to factories as I think that your model, if I understand it correctly, requires. All that a factory does is check whether any stop within its catchment area can reach by transport a stop connected to the destination. The destination has no means of checking whether it is connected to the origin: all that happens is that goods arrive.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 06, 2014, 04:56:31 PM
There is still a system for turning off shipment towards a consumer when the current storage is full. This may be done by simply saying that the consumer is no longer a destination for goods (appears dis-connected to the supplier). This is why a credit buffer is used as the goods do not require immediate shipment from suppliers, just they need to be shipped in the near future (a transition which happens when current storage falls below maximum storage anyway). The end result is that suppliers would be shipping goods to consumers as currently except the consumer itself would toggle acceptable considerably faster than currently, based on the value of the credit buffer rather than the current storage. When a supplier produces a good it has to decrement the credit buffer, which should be similar to the current incrementing the in-transit amount. The only different between the in-transit amount and credit buffer is that you will have to save the credit buffer as it is not deterministic from other data (unlike the maximum in-transit amount which can be determined from the actual goods that are currently in-transit).

Consumers themselves do not care about what suppliers they receive from and what their status is. They only care about the status of the credit buffer which represents how many goods are still outstanding to be produced. If this is constantly growing it means that the industry is under-supplied or there is a collection bottleneck. The consumer adjusts the rate the credit buffer is refilled so that enough goods get sent, the suppliers simply need to send goods to consumers which have outstanding credit.

The gaining credit phase could be entirely removed but then people will not be given a big penalty for failing to ship goods efficiently. The dropping credits phase is required so that the current storage will tend towards maximum storage amount over time.

Suppliers that require consuming multiple goods are the only very complex part as to manage all 3 goods in parallel without causing a current storage overflow in the case of any supply problems requires additional logic. However even if the goods are all treated separately it should still tend towards within the storage amount as long as supply is eventually fixed.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on July 06, 2014, 07:36:26 PM
Please check out these two threads:
http://forum.simutrans.com/index.php?topic=12919.msg130045#msg130045
http://forum.simutrans.com/index.php?topic=13097

Your idea of credit buffer is very similar to my idea of industry demand counters. The discussion slowly died out, maybe because I was not able to explain it exactly (i'm not native english speaker). Perhaps you could get some inspiriation from those discussions, and move the idea forward. Perhaps it could be simplified a little so that the changes to simutrans core would be minimalised.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 13, 2014, 04:02:39 AM
I have started looking into making this change. I believe all that is required is changes to the factory class (fabrik_t).

Two new variables are required.
A signed 32 bit Q10 fixed point for the production buffer.
A boolean for production demand.

The production buffer is incremented at the scaled production rate or half if current storage is greater than maximum storage.
If it is greater than 0 then production demand is set for that industry tick. This is required to allow multiple suppliers a chance to send to the factory. When sending they decrement the production buffer by the amount sent.
If the production buffer is less than or equal to 0 on a factory tick then acceptance is turned off, suppliers will no longer send to the factory.

A final step is needed to detect buffer overflow (production buffer exceeds maximum storage, some other value could be used but this seems generic enough). In the case factories that are both consumers and suppliers will need to drop (skip) buffer from all their consumed goods in the appropriate ratios.

Seems easy enough and nothing mind explodingly complicated.

That said fabrik_t is in urgent need of attention...

I have so far found the following faults...
1. Fixed points are used without a type indicating that the number stored is in fixed point form. This leads to a lot of potential for errors and certainly makes the class unreadable (it took me hours to understand all the fixed point variables and what fractional size they were). I am trying to introduce named types that are just renamed normal integers so that programmers knows what type each value is stored as and what operations are needed to change it between types. This is not a long term solution, a special fixed point class should be used to decouple the fixed point arithmetic operations from the class implementation but that is not of high priority.
2. Many constants that should be in the header file were randomly in the implementation file. Some are being moved as nameless enums to the header for consistency and convenience.
3. A minor inconsistency should be fixed when a factory exceeds the hard-coded maximum storage amount (15,000? Or so the constant says). Current behaviour is to drop the entire shipment arriving never actually allowing it to reach 15,000. Now it will only drop what cannot fit up to exactly 15,000. It may be a good idea to re-evaluate this constant as it looks like some factories in Pak128 (oil refinery) are dangerously close to exceeding it with maximum in storage amount (14,600 in one of fifty's servers). I am sure a larger, power of two aligned constant might be a good idea, or maybe an environment set constant that can be assigned different defaults depending on pak. An alternative could be to make it a multiple of maximum storage.
4. Incorrect fixed point multiplication causes a potential overflow hazard with consumer only consumers with huge production rate, large delta time or huge production scalars. A 32 bit unsigned is used to hold the product result of two 32 bit unsigned fixed points which are then downsized to a 32 bit unsigned fixed point of a different precision. This intermediate stage must be done using a 64 bit unsigned to prevent the potential loss of 8 bits of data due to overflow.

And that is as far as I have managed to get so far...
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on July 13, 2014, 08:22:54 AM
1. Fixed points are used without a type indicating that the number stored is in fixed point form. This leads to a lot of potential for errors and certainly makes the class unreadable (it took me hours to understand all the fixed point variables and what fractional size they were). I am trying to introduce named types that are just renamed normal integers so that programmers knows what type each value is stored as and what operations are needed to change it between types. This is not a long term solution, a special fixed point class should be used to decouple the fixed point arithmetic operations from the class implementation but that is not of high priority.

While a good idea, I think it will be even more confusing when there is a fixed point type, but lots of the code (everything but fabrik_t) do fixed point arithmetic without it. Introducing a fixed point data type should also rather be a separate patch. I think that makes it easier to review, althought that's mostly prissi's problem.

There are several fixed point "types" in Simutrans, many of which are scaled decimally rather than binarily.

2. Many constants that should be in the header file were randomly in the implementation file. Some are being moved as nameless enums to the header for consistency and convenience.

I like enums to have names, so you know what they are for. If they are unrelated constrants, #define will do, or even better, a C++ const (global, perhaps static, or in the class, depending on it's scope).
Title: Re: Suggestion for JIT mechanics revision.
Post by: jamespetts on July 13, 2014, 09:57:29 AM
Incidentally, how do you envisage this working differently to the current Simutrans-Experimental system?
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 13, 2014, 01:18:12 PM
Quote
While a good idea, I think it will be even more confusing when there is a fixed point type, but lots of the code (everything but fabrik_t) do fixed point arithmetic without it. Introducing a fixed point data type should also rather be a separate patch. I think that makes it easier to review, althought that's mostly prissi's problem.
I have also been adding comments explaining where everything is coming from. It took me hours to work out what is going on and I could swear that I found 3 or so errors in precision logic (it works by luck/only for certain constants, if they were changed it would break badly).

Quote
There are several fixed point "types" in Simutrans, many of which are scaled decimally rather than binarily.
fabrik_t needs a binary fixed point for performance. You must remember that these factory calculations are done a lot so one wants to avoid adding stuff that is unnecessarily slow. Fixed points are simple enough, but it appears some people did not quite understand them when writing fabrik_t.

Quote
I like enums to have names, so you know what they are for. If they are unrelated constrants, #define will do, or even better, a C++ const (global, perhaps static, or in the class, depending on it's scope).
Nameless enums are useful for defining constants as they are compiler inlined and produce more exact errors with some compilers. The "#define" macro should never ever be used for constants in c++ as it is not really type checked (it literally is just a textual copy and paste algorithm, many IDEs will fail to pickup a type mismatch with them and errors they generate at compile time can not be obvious, also they have certain problems relating to scopes and things), nameless enums achieve the same result as define for such constants are considerably safer and easier to use (they represent a value of a certain type, unlike define which just represents a piece of code). The use of constant global/statics is actually inefficient unless you need to take the address of the constant for some reason since it allocates memory for the constant and reads that instead of in-lining the constant into instructions. That said constants should optimize out with a properly set compiler and are certainly a lot better than define but still nameless enums have the advantage that they do not explicitly define a storage space for the value.


Quote
Incidentally, how do you envisage this working differently to the current Simutrans-Experimental system?
You compute max in transit based on real line metrics (which are totally rubbish atm... see my complaint thread about in transit times) giving a limit to the amount that can exist at any given time. The suggested approach has no such limit but instead just applies a limit on good generation (well goods being sent from suppliers) so that they are generated constantly instead of in bursts.
Title: Re: Suggestion for JIT mechanics revision.
Post by: jamespetts on July 13, 2014, 02:02:10 PM
You compute max in transit based on real line metrics (which are totally rubbish atm... see my complaint thread about in transit times) giving a limit to the amount that can exist at any given time. The suggested approach has no such limit but instead just applies a limit on good generation (well goods being sent from suppliers) so that they are generated constantly instead of in bursts.

I see; and how would this differ in practice once the bug to which you refer has been fixed?
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 13, 2014, 02:27:24 PM
Quote
I see; and how would this differ in practice once the bug to which you refer has been fixed?
In experimental the production is limited to a maximum in transit (more like in existence as it is both the current storage and in transit amount but where the value came from was your max in transit ammount). This means for small industries serviced by large transports (such as a huge 1050 ton carrying ship to a tiny butcher outlet) you cannot full load (100%) the ship as the maximum allowed in transit will mostly be less than 1050 tons, the amount the ship requires. With the system proposed for standard you could as eventually enough will enter existence to send a full shipment however since the maximum storage is less than this amount you will incur a production penalty after it arrives (it will be operating near half efficiency on average). In experimental that same 100% loaded ship will never leave as enough goods will never come into existence. One can argue that such a ship should never leave as it is nonsense to supply a 20 tons per month shop with 1050 tons but since standard is meant to be easier I do not think adding such logical restrictions is a good idea (a penalty of inefficacy is much more understandable).

The reason for the difference is that standard should be easier to understand (less fussy) than experimental and that standard does not have route times (you use these to calculate maximum in transit). The disadvantage of the system is bottlenecks not really being resolved (you could get to multiple hundred thousand in transit, obviously not a problem with transfer restrictions).
Title: Re: Suggestion for JIT mechanics revision.
Post by: jamespetts on July 13, 2014, 02:46:45 PM
That is a very helpful summary: thank you. Yes, it is easier to work with a system in which there is notionally infinite warehouse storage at every consumer industry, but less realistic.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on July 13, 2014, 03:41:39 PM
Nameless enums are useful for defining constants as they are compiler inlined and produce more exact errors with some compilers. The "#define" macro should never ever be used for constants in c++ as it is not really type checked (it literally is just a textual copy and paste algorithm, many IDEs will fail to pickup a type mismatch with them and errors they generate at compile time can not be obvious, also they have certain problems relating to scopes and things), nameless enums achieve the same result as define for such constants are considerably safer and easier to use (they represent a value of a certain type, unlike define which just represents a piece of code). The use of constant global/statics is actually inefficient unless you need to take the address of the constant for some reason since it allocates memory for the constant and reads that instead of in-lining the constant into instructions. That said constants should optimize out with a properly set compiler and are certainly a lot better than define but still nameless enums have the advantage that they do not explicitly define a storage space for the value.

#define is bad, I agree to that. Enums are for creating data types that can only have specific values. Collecting unrelated values in an enum is just as bad as storing fixed point values in regular ints in my opinion, while an enum for each constant will look cluttered. const is for creating constants, and any compiler worth using will inline them. static const can be put in headers.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 13, 2014, 04:34:10 PM
Quote
Enums are for creating data types that can only have specific values
Anonymous enums are for creating constants since you do not define a type. The constants declared in an anonymous enum become attached to the parent namespace, as if they were declared constant values in it. The main advantage of anonymous enums is that it forces complete compile time inlining unlike actual constant values which may have their value inlined (optimizing compiler) but may also not or may still keep a copy of the constant allocated somewhere in memory. The obvious dis advantage is that unlike const declared variables, you cannot take the address of an enum constant which means they are not suitable for any form of pointer related use (such as passing as a function argument or doing some hacky bit manipulation).

The main problem is that currently fabrik_t actually uses all 3 approaches! It has definitions, anonymous enums and constant globals. Migrating to anonymous enums would certainly be more consistent but that may be another task.

Quote
Yes, it is easier to work with a system in which there is notionally infinite warehouse storage at every consumer industry, but less realistic.
Actually the system does factor this in. It only produces exactly as much as the industry consumes and if current storage exceeds maximum storage it then throttles back production so that current storage will eventually fall bellow maximum storage. The idea being that the actual amount in transit does not mater at all (if it is cross an enormous map it could be several hundred thousand!) but the rate at which it leaves and arrives does. Will it work? Well only time can tell.

Your system also allows temporary exceeding of maximum storage limit and I am still struggling to get industries working at 100% (especially ones with small consumptions and storages).
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on July 13, 2014, 05:08:15 PM
I still expect an enum to enumerate related values. The anonymous enums I've seen in Simutrans does so. The reason why they aren't named and used as a type is because there is no reliable way to force a certain size, and Simutrans tries to pack data as densly as possible, at least in certain structures where performance really matters.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Leartin on July 13, 2014, 07:51:50 PM
So I get this is about a problem with long routes over giant maps, probably by train. But if it's done, it's probably the same even for the shortest possible routes...

Let's say I want to transport coal to a nearby power plant by truck. Usually, I'd do it like this: Guess a fitting amount of trucks, and watch them completing the route once. At the time the first truck is back at the coal mine, the mine should be just about to fill up again. If it isn't, there are too many trucks on the line, and at some point, they will have to wait at the freightyard for coal to be produced. If it filled up too early, I can add another truck.
I think that's nothing new to you all and pretty much the common way to set up a route like this.

With that change in place however (if I understood it correctly), since the production builds up slowly, I can't grasp the needed amount of vehicles that quickly, or ever. On the other hand, I don't have to, since it's done automatically. What I'd have to do is send in too many trucks, wait till the equilibirum is reached by the game itself (although there is no way to know when that happened, so - just wait "a while") and later just look for those which have to wait for load and delete them.

Basically, the "pipe" where you as the player had to choose the right thickness before would be given by the game. Doesn't that take away part of the challange?
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 13, 2014, 11:03:35 PM
Quote
Basically, the "pipe" where you as the player had to choose the right thickness before would be given by the game. Doesn't that take away part of the challange?
No because it forced you to either manually install bottlenecks to keep back the flood of production (overcrowding is not good) or face a bursty production model or inefficient consumers. With a bookshop 500 tiles away consuming >2,000 units per month and 2 or 3 printing factories (as 1 cannot supply the bookshop alone) I would like to see you balance lines at all without some form of bottleneck from overcrowding. I have seen a case like this a few times in multiplayer just because of bad luck with the RNG, the result is bookshops seldom run anywhere near 100% of the time.

With the proposed model you will always get goods going towards industries all the time at the rate at which they are consumed. This prevents station overcrowding, eliminates the need to cope with huge volumes of traffic to avoid overcrowding and avoids all problems associated with "max in transit".

Basically instead of a consumer ordering all it can get its hands on until it suddenly gets flooded, or your stops are flooded and then stopping, they will instead order on demand as they consume. This is much more like a just in time model as it is ordering replacements as fast as it uses goods and not relying on storage (which is the key of a just in time supply chain). It is also more realistic to some extent as you do not get coal power stations ordering from 20 mines across the map as much as they can produce until some arbitrary number is reached and then finding that the power station is buried under a pile of coal as it all arrives very quickly.

The focus is shifted more from fighting bad supply and consumer mechanics (trying to get them to work efficiently) to actually making a fast and efficient transport network that can get the goods from one or more sources to the consumer in a reliable way.

The other solution (which appears partly implemented? Or maybe it was a WIP that was never finished?) was to raise the maximum storage based on average distance from consumer to supplier. This will be nowhere near as effective as a constant stream of goods and will possibly suffer in early years (low transport speed) and be too generous in later years (high transport speed).
Title: Re: Suggestion for JIT mechanics revision.
Post by: Leartin on July 14, 2014, 12:31:51 AM
Why would I be forced to install bottlenecks?
I install lines which have the correct capacity. I can do this by making sure that each consumer does not get more then he can use, which means that a new delivery has to arrive exactly when the internal storage gets empty. It can't be done with 100% accuracy, but usually there is enough capacity so that a bit more does not hurt. It would probably fill up slowly over the years, but then there are always some disturbances so it would get less. This is quite possible, given that in-transit and internal storage are big enough. The only overcrowded stations are those next to the supplier, and I don't care for these.*

I can see that there is a problem with too long routes, but it is only a problem with too long routes. Some people like to play small maps, where the problems you have can't possibly come up, since in-transit and storage is always big enough.



With the proposed model, I just don't have to think about the fluctuations anymore. I can just go yolo mode and throw in as many as I want. Currently, this behavior will result in too many goods in the consumers storage, shutting the whole line down. The problem is not even that the consumer might not get new goods soon enough, I think it's more problematic that there are a bunch of vehicles standing around doing nothing. You know, you don't earn money for having a maxed out consumer, you earn money for delivering goods. But with your model, there will be the same number of vehicles waiting for goods at the producer at any given time. I kick out all waiting vehicles but one - there you go, route optimized.

Also, the amount of goods in-transit does matter, if you want to be realistic (and you used realism as a point) - companies have a budget. This is what max in-transit represents - If there is some consumer on your map which wants to have a good a factory somewhere else produces, but the consumer is too small to have the budget to ensure a steady stream, you just shouldn't connect them. That's realistic: The consumer just can't afford so many goods to be in transit. And even so, you may still build a route that's balanced in a way that there is always the max in-transit used. It might not be enough to let the consumer run on 100 percent, but so what? It's effectively the same as the consumer just having a lower productivity, it does not matter to a transportation agency (especially with end consumers like the bookshop)

But that's just my opinion on the matter, play styles differ, and I'm sure for you your proposed model would make the game more interesting. I just wanted to say that might not be the case for every player.



*) the overcrowding of a station with goods produced from the factory right next to it is a really stupid thing, since the goods could just be stored in the internal storage of the factory in many cases, and the "station is overcrowded" message gives the wrong impression that the player is supposed to transport everything away the factory produces, which is obviously not true. Thus, making it so a factory won't ever overcrowd the station, only fill it, would be welcomed.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on July 14, 2014, 07:10:42 AM
If there is some consumer on your map which wants to have a good a factory somewhere else produces, but the consumer is too small to have the budget to ensure a steady stream, you just shouldn't connect them.

That would easily become a boring "game" of doing nothing, then. At least as far as goods is concerned.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Leartin on July 14, 2014, 12:19:25 PM
That would easily become a boring "game" of doing nothing, then. At least as far as goods is concerned.

I strongly suspect that must be pakset dependant then. I just made a test route over 2000x2000 tiles at 65 km/h, and it worked just fine. Sure, that was because the storage of the consumer was big enough, but why wouldn't you allow for a big storage/in-transit? If every industry in a given pakset has so low limits that the game becomes boring, maybe it's the paksets fault?
But you just used one sentence without context, so just as a reminder what came next:
Quote
you may still build a route that's balanced in a way that there is always the max in-transit used. It might not be enough to let the consumer run on 100 percent, but so what?

See, if there are not enough suppliers on the map to let an industry run at 100%, is that a problem, or does it just mean you transport what's there? Or if an industry produces two goods, one which there is no consumer for, do you care and demand that there must be a consumer for it, or do you just transport what they want you to transport? The max in-transit is just like that, it limits the amount you can transport. It does so in a more interesting way, since it might encourace you to choose faster transportation methods.


I just don't like the praised "effectiveness", as it seems like a button "set as many vehicles on the route as needed automatically". It would force a change of playing style even for players who never had any trouble with that stuff, maybe because of map size, maybe because of different usage of storage in different paksets. The 'other solution' ("raise the maximum storage based on average distance from consumer to supplier") seems much more promising to me, and the change of speeds over time is already reflected in the table for speedbonuses which could be reused to change the maximum in_transit over time, thus not being too generous later on.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on July 14, 2014, 02:30:25 PM
I dropped what you call the context, because it is irrelevant to me. I'm not bothered by how the consumer operates, but by the number of crowded station messages I get when the supplier suddenly starts dumping its accumulated storage. As for whether this is a pak set problem, an unoptimally tuned in-transit limit or an error in the algorithm as a whole, I have no idea.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Leartin on July 14, 2014, 03:54:38 PM
I dropped what you call the context, because it is irrelevant to me. I'm not bothered by how the consumer operates, but by the number of crowded station messages I get when the supplier suddenly starts dumping its accumulated storage.

For which there is a 'simple' solution, as I already proclaimed: An industry next to a station should simply not be able to overcrowd the station, but only fill it up instead. Since the initial producer does not need any goods, it's irrelevant that the goods-slot is full anyway, as it's just there waiting to be transported away.
For the second industry in the chain, it seems problematic, since that one also receives goods. However, it's you who chooses how many goods are produced there, simply by sending the right amount of raw material. You'd look at the end consumers needs and let the intermediate industry produce the right amount by sending it the right amount of raw materials. This is not a simple task, but it's part of the challenge.

If you don't want to balance it, just don't play with "just_in_time". The gameplay is the same as the proposed model: Transport away everything an industry produces. If any station overcrowds at any time, add vehicles to the route transporting stuff away from it. The only difference is that the proposed system lowers the productivity of intermediate industry, thus less stuff to carry around.


I hope now it's more understandable why I prefer the current method, and why I don't want to lose it. You all may now continue to discuss and later implement something else, but please don't get rid of the current behaviour.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on July 14, 2014, 04:39:01 PM
As a hired transportation company, it should not be up to me to choose how much to transport. It's the industries that should decide how much to send, and then complain to me if I don't keep up.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 14, 2014, 06:09:56 PM
Quote
Why would I be forced to install bottlenecks?
Quote
The only overcrowded stations are those next to the supplier, and I don't care for these.*
That is a contradiction. You first say you do not need a bottleneck (no overcrowded stations) and next say you do (overcrowded at source).

The fact is over crowded stations are a bad model. It represents you as a transport company being incapable of dealing with the required transport demand. That coal mine has more orders just you cannot cope with them. Obviously a big question is if it should have so many orders in the first place.

Quote
Some people like to play small maps, where the problems you have can't possibly come up, since in-transit and storage is always big enough.
Some people like to play big maps (especially in multiplayer servers where space can be quite limited) where in-transit is a huge problem. On one of Prissi's servers (now closed) one player specialized in coal shipments except had such an inefficient network that all coal power stations were maxed out in-transit (transfer overcrowding with no transfer restrictions) so even if you tried to make them more efficient it was not possible.

Quote
I kick out all waiting vehicles but one - there you go, route optimized.
Until you add another mine to your transport network, or have to deal with 1:n or n:1 connections, but then you can repeat until all is optimized. The big difference is now you will have many lines running in the same time it took you to optimize a single one.

Also I thought this was meant to be a transport simulator, not a supply/demand logistics simulator. The industries are meant to place their orders, and you just have to deal with the shipping, not also manage their entire supply chain including how much to send them.

Quote
Also, the amount of goods in-transit does matter, if you want to be realistic (and you used realism as a point) - companies have a budget. This is what max in-transit represents - If there is some consumer on your map which wants to have a good a factory somewhere else produces, but the consumer is too small to have the budget to ensure a steady stream, you just shouldn't connect them. That's realistic: The consumer just can't afford so many goods to be in transit. And even so, you may still build a route that's balanced in a way that there is always the max in-transit used. It might not be enough to let the consumer run on 100 percent, but so what? It's effectively the same as the consumer just having a lower productivity, it does not matter to a transportation agency (especially with end consumers like the bookshop)
Not true. In real life it depends on payment structure (when the goods arrive or depart) and all large shops can get enough credit to buy huge amounts of goods even when in transit. One can assume that there are no small shops in simutrans as in reality a city commercial district is nothing but shops yet a huge city in simutrans is lucky to have even  a few shops. Thus one should think about these being major shops (bulk books inc runs all bookshops), not some family run corner store. Even if it was a small shop some kind of wind-up mechanic could be added.

If you want to limit max in-transit I advise you play Experimental. Its system does just that and instead of using maximum storage and current storage to determine when to ship it instead sums current and in-transit and then compares if it is less than max in-transit. Max in-transit is computed using measured line statistics (slightly broken atm, hopefully fixed soon) so bottlenecking transport is very bad and making transport more efficient will reduce the amount to prevent excess.

Quote
*) the overcrowding of a station with goods produced from the factory right next to it is a really stupid thing, since the goods could just be stored in the internal storage of the factory in many cases, and the "station is overcrowded" message gives the wrong impression that the player is supposed to transport everything away the factory produces, which is obviously not true. Thus, making it so a factory won't ever overcrowd the station, only fill it, would be welcomed.
This is what the suggestion addresses. Now the goods will remain in the factory storage with only as much leaving it as the consumer requires.

Quote
I strongly suspect that must be pakset dependant then. I just made a test route over 2000x2000 tiles at 65 km/h, and it worked just fine. Sure, that was because the storage of the consumer was big enough, but why wouldn't you allow for a big storage/in-transit? If every industry in a given pakset has so low limits that the game becomes boring, maybe it's the paksets fault?
This is not an issue in pak128 where you can move goods like paper at 900 km/h and the storage is in multiple thousand units.
This is a problem for pak64 (+food) where a marketplace only allows 64-128 units to be stored and consumes several thousand a month.

Bad pak design? Probably, I certainly would not have set them so small with the current mechanics but still they are set like that. One could also say the current mechanics are broken (why should a tiny market need to store 5000 tons of produce?). Without a mechanic change there is still a map size limit. With pak64 I would say this is in the order of 100-200ish before some industries become impossible to operate at full efficiency while in pak128 its probably in the 1,000 range.

Quote
The 'other solution' ("raise the maximum storage based on average distance from consumer to supplier") seems much more promising to me, and the change of speeds over time is already reflected in the table for speedbonuses which could be reused to change the maximum in_transit over time, thus not being too generous later on.
Except it will never really work. You can easily use a plane in pak128 to ship cargo in maps >10,000 in size since the planes move at >900 km/h. However suddenly you need to move oil the same distance in an island situation and you are limited to 30km/h odd of ships. It is just not possible to compute a "1 size fits all" solution to max in-transit in that situation. Experimental does this obviously but it has a lot more information available such as actual route times from source to destination based on good approximations and real measured data. Why not add such a system into standard? It is excessively complicated and already in in experimental. Standard is meant to be easy to use and not have many complicated game mechanics. If you want complicated you should try experimental where your passenger pickup depends on service quality but even there you can just time-table a perfect line (they made it easier). Personally I have moved to Experimental more and more recently, just I still keep a presence on carious standard servers.


Quote
If you don't want to balance it, just don't play with "just_in_time". The gameplay is the same as the proposed model: Transport away everything an industry produces. If any station overcrowds at any time, add vehicles to the route transporting stuff away from it. The only difference is that the proposed system lowers the productivity of intermediate industry, thus less stuff to carry around.
Except then you are transporting 100% of the production of a site to single factory. This is OpenTTD style play which gets quite boring and easy over time. The entire point is to limit consumption so to maximize profit you need to maximize the amount consumed. The suggested approach would apply a penalty for exceeding the maximum capacity of an industry. The maximum capacity then becomes a buffer to allow for fluctuations in transport mechanics. Especially in a case like where you have multiple scrapyards to multiple incinerators this buffer may be required as some times more goods may arrive than at other times. This is a more realistic approximation of just in time as the factory storage is purely used to ride out until the next shipment arrives, which should be consistent to some extent as goods are being replaced as they are consumed.

Quote
I hope now it's more understandable why I prefer the current method, and why I don't want to lose it. You all may now continue to discuss and later implement something else, but please don't get rid of the current behaviour.
I understand it is important to you which is why I am trying to implement it around the existing code for now. It could be given a new name like "Just in Time v2".
Title: Re: Suggestion for JIT mechanics revision.
Post by: Leartin on July 14, 2014, 09:22:53 PM
I thought this was meant to be a transport simulator, not a supply/demand logistics simulator. The industries are meant to place their orders, and you just have to deal with the shipping, not also manage their entire supply chain including how much to send them.
Okay, this seems to be where the real trouble starts. As I said before, I like that supply/demand logistics aspect. If you want to get rid of it, you want to get rid of a part of the game I like.
You say it's because Simutrans is not a logistics simulator, I was curious and looked up that term. Interestingly, there is a "Warehouse and Logistics Simulator", which is pretty much about forklift driving. Thus, I agree that Simutrans is not a logistics simulator. :)
However, it's not a construction simulator either, yet you can build stuff. I'd suggest not to rely on those terms too much when it comes to which features should or shouldn't be in a game.

I think if it was a straight transportation simulation, the whole system of supply and demand would not need to exist. Actually, you'd have to assume that the goods get to their goal anyway, even if you don't do anything. If there was no other option, clearly the companies would just use their own vehicles before they all get bankrupt for not selling or not even producing anything at all, so your job is to deliver the goods cheaper then they could themselves.
I stop here, since you know where this is going. Somewhere where it's not Simutrans anymore, maybe a more OTTD-like direction. But don't worry, I don't want Simutrans to be like that. I want it to be more then a transportation simulation, it should have supply&demand. But, you know, since we hold a transportation monopoly anyway and no industry could even exist without us, since we are pretty much controling the world, why would it be far fetched to assume that industry would want us to do the logistics stuff as well as the transportation stuff, solely concentrating on producing their goods? ;)

Also, with everything Simutrans has gone through over the years, I'm not sure if anything is farfetched for the future. While right now it's a transportation simulation, it's almost like you'd just need to add buttons which build new factories, the connect-factories button in a normel toolbar and a pakset that is balanced accordingly to get something like an Industry Giant clone. Or add some citizen demands to create a citybuilding game. You could add a polution map and give all objects you build some kind of pollution generation/absorbence, and you have an environment simulator. This is not what Simutrans was intended to be, but I think it's well possible. Nothing the core developers would work on, though ;)

Quote
This is what the suggestion addresses. Now the goods will remain in the factory storage with only as much leaving it as the consumer requires.
I guess I was not thinking it through. The moment the factory tries to put it's goods in the freightyard is the moment they get a destination. Which means even if you wanted to turn off the message in these freightyards, you'd need to tamper with distribution and demand anyway. Upon realizing this, It's easier to understand why my idea to "just do that" is a bit stupid.

Quote
Standard is meant to be easy to use and not have many complicated game mechanics. If you want complicated you should try experimental
Really? I mean, I know that Experimental does have more complicated mechanics, but does this really mean Standard can't get any and needs to be simplified? I always was under the impression that Experimental was a fork meant to include stuff that was not deemed suitable for Standard, mostly due to performance issues. Which would mean that the way Standard works wouldn't be dependant on Experimental (except for interchanging parts of code if it fits both).
"complicated game mechanics" is by the way not the same as complex. For example, "Go" has really simple game mechanics, yet is more complex then chess, which has far more rules *caugh* I mean, game mechanics.
It would be sad if something can't be in Standard "because Experimental has it, go to Experimental if you want it" (if there is another reason why it isn't in Standard, it's a different story.)
Also, isn't OTTD already the easy version of Simutrans? (all this does not belong in this thread though... so, if anyone feels like answering, just PM me, I don't bite)

Quote
I understand it is important to you which is why I am trying to implement it around the existing code for now. It could be given a new name like "Just in Time v2".
That's pretty much what I wanted, even though I don't like the "for now". *gg*
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on July 15, 2014, 04:34:50 AM
The curse of Simutrans. Everyone sees a different game in it.
Title: Re: Suggestion for JIT mechanics revision.
Post by: jamespetts on July 15, 2014, 09:43:57 AM
The curse of Simutrans. Everyone sees a different game in it.

That's why we have Experimental.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 15, 2014, 06:14:31 PM
Well I have successfully implemented this into simutrans standard and so far it appears to work absolutely amazingly. Even middle industries that are under used consume just enough!

It still has problems with pak64 + food market places as in 2050 the smallest unit of goods you can ship is 40 odd while the canned food storage is only 35 units. This points towards a balancing issue with that building (it should maybe have a larger storage factor). There is no problem with beer though as that has a good storage factor for a packed good (many trucks). However I doubt you would operate them anywhere near as efficiently without the new JIT2 system!

I will make a build for Windows shortly so you people can test it out for yourselves. It will not save the JIT2 data properly due to no save version allocation, however it will be backwards compatible with existing saves (hopefully).

EDIT

I have done tests and all seems to be working well. I need to just add an extra safety to it and then it is ready for people to test. I will provide a Windows build and the modified files which you can just place in a folder with any nightly build and it will operate. Existing saves do work but saving is incorrect until a version is allocated. I tested it on Fifty's Christmas Server and even the first server I played on and it ran fine and many industries showed great improvement including overall reduction of in-transit amount and improvement in stability of operation. Most noticeable was how a waste power station in pak64 was able to operate at full efficiency for once.

There appears to be a problem with the max in-transit graph of some factories. I am not sure if this is a nightly problem (since I used latest source) or it I messed it up. The amount in transit in the factory info panel is displaying properly however (is correct) so I am not sure where the error is coming from but I doubt it was something I did.

Download Here From DropBox (too big to upload to post) (https://www.dropbox.com/s/a5iksle1tvt2qpe/Simutrans%20JIT2.zip?dl=1).
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on July 17, 2014, 10:50:34 PM
The small values for the market was set deliberately to avoid supplying it by trains and rahter sue traucks from a central station.

I am not sure how to get a central storage facility right with you system, but I still have to playtest it (I am on a conference until Sunday).
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 17, 2014, 11:58:59 PM
Quote
The small values for the market was set deliberately to avoid supplying it by trains and rahter sue traucks from a central station.
Yeh that makes sense. The problem is that it is so small that the smallest truck available end game (2050ish) is 40 units when a market in the session in question only had capacity for 35 units. This meant it is impossible to deliver to without exceeding the storage. This was fine in the standard system as there was no real penalty for exceeding the storage but with this system it means you do suffer a consumption penalty of a bit. However it does mean that you can always supply the market with some efficiency, no mater how distant it is from the sources of its food and no mater how well the passenger/mail service is to it.

Quote
I am not sure how to get a central storage facility right with you system, but I still have to playtest it (I am on a conference until Sunday).
By this I assume you refer to a central processing facility where a lot of cargo runs through from many sources and bundled together to move to consumers. It works great for some industries. Even if many suppliers exist on average the flow rate to the consumer will keep it working 100% of the time. Others do struggle to some extent (small capacity ones with lots of suppliers) but on average it can achieve >90% operation, however this is distant independent unlike currently where you could struggle to operate some small capacity consumers over long distances.

Observations from this system being retroactively applied to well in progress "end save server games" was promising. Soon after running with the new industry model current storage levels drop to below the maximum levels and seldom exceed them (exception being for industries being supplied with convoys larger than their storage, eg a printer works by a huge goods train). In-transit amounts mostly drop (will explain later) to a much more reasonable level. Both storage and in-transit stabilize quickly and suffer very little variance with respect to time.

The first problem noticed was related to transfer bottlenecks. If there is no transfer restrictions, it is now possible to have central transfers (many to one to many goods flow) or accidental inverted flow stations (sending chemicals and books to a distribution station resulting in printer ink requiring inverted transfer is the best example) or even just a gathering station (many to one for long distance shipment) to run up millions of units of goods in storage. Over a year on one of the test games an improperly configured oil network ran up over 100,000 units of oil at some random point (I guess oil pooled there before, just now it made more of a difference since no excess was being sent). However if this is really a problem is debatable as if someone observes this happening they can always correct it, and it used to happen anyway but was just limited in severity by maximum in-transit.

The other problem was related to overloaded suppliers with few linked consumers. In fifty's current pak128 game I tested it on I noticed that there was a shift in consumption from some suppliers to others. Specifically some fish producers had a shift in consumption with some places starting to operate at full capacity while others dropped in load. Fish was being moved around by plane so there was no issue distance or time. This means that there may be a possible fairness issue with the implementation (could be resolved with adjustments to distribution behaviour?) however this is quite minor, the percentage load change was probably under 10% across all fish producing places with some hitting maximum while others produced less.

Overall I am quite satisfied by the results and a bit sad few people have taken interest in it.

I have updated the file just now to Simutrans JIT2 Demo (https://www.dropbox.com/s/a5iksle1tvt2qpe/Simutrans%20JIT2.zip?dl=1). Middle industries (both consumer and producer) now manage input more fairly in the case of insufficient supply with one of the supplies being overfull. I also fixed a bug where by the demand counter would start out at in-transit amount instead of 0 (which should be the default if loading old saves).

There should be noted that the implementation differed from the original idea in these ways.
1. Demand counters are refunded in the case of demand maxing out for one of the input goods with a middle level industry. They are refunded in a scaled way so that demand for all input goods is scaled back to the supply of the weakest link (the one with insufficient capacity available). This works amazingly well at preventing over-ordering of input goods in the case of an insufficiently supplied middle industry.
2. Consumers that are powered but with insufficient supply to use power will order as if they are fully powered. Since they are not actually consuming power and there exists no relationship to get the power net attached to them it is impossible to make a better approximation so it assumes that the power grid will sufficiently supply the producer. Obviously in the case of an overloaded power network this may result in some over-ordering but that should only really be the case for poorly managed power networks.
3. There is a maximum limit on demand beyond that of the maximum storage limit of the factory. Currently this has been set to the maximum limit on current capacity which makes sense. It will prefer to max out at the maximum storage amount but this limit was added in case of stupidly large maximum storage amounts that exceed the current storage maximum.

Possible ideas for future work...
1. Add some limit to stop storage preventing excessive build up of goods (eg twice maximum storage of destination factory). This will resolve the bottleneck transfer problem climbing to stupidly large numbers. Some form of no transfer mechanic could be made where by goods cannot transfer over a stop which has too many goods already destined for the same location. It should also be noted that due to the flow mechanic (no bursts) players may want to use transfer restrictions more often as you seldom need huge amounts in storage as there is no huge burst of production anymore.
2. Change the warning colours of industries to reflect demand in some way. Not sure on exact behaviour but it is now possible to detect insufficient supply by large numbers of demand (since sufficient demand should be negative most of the time). Possibly new colours could be added to represent supply shortages. In any case green should still be the colour to aim for.
3. Revise middle industry production mechanics. I am sure there are possible optimizations to be performed with this system there. Also do not think the current consumption production model is completely fair as I do not think it factors how much product was actually made when consuming inputs (the input consumed ratios should represent the amount of product consumed if all products are produced at maximum rate, currently I believe it only reflects the amount needed for the factory to produce any or all products with most efficiency being if all products are made and least efficiency being if only 1 product is made).
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on July 18, 2014, 05:27:09 AM
The small values for the market was set deliberately to avoid supplying it by trains and rahter sue traucks from a central station.

How was that supposed to work? Nothing prohibits you from dumping a trainload of goods into the consumer, even if it's way beyond it's storage capacity. Having a central station doesn't change when goods is sent from supplier either, so the restriction affects that solution just the same. For me, the prime motivator for using trucks, either from a intermediate station or directly, is to avoid demolishing several city blocks.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 18, 2014, 02:23:56 PM
Quote
How was that supposed to work?
I am guessing that generating a train load of goods before it runs out of storage required too much time for anything more than short distances, especially at early days with slow transports. This meant you were either forced to set up a flow to the market or suffer great periods of no operation. Flows cannot work if they exceed maximum storage with the current JIT mechanics as that stops the flow of goods temporarily. With the JIT2 system however this truly is the case as exceeding storage capacity results in a supply production penalty (default of 50%) however the values of the market storage are so tiny that it is impossible to supply one of the goods late game as no truck holds less than 40 units (max storage was 35 in this case).

So anyone else have feedback for the JIT2 system? If you use Linux you should be able to build with it simply by swapping out the header and class file provided with the zip.

EDIT

I built this using VS2010 so you will need "pthreadVC2.dll" not the standard "pthreadGC2.dll" that comes with most releases. You also will need compatible visual C++ runtime libraries. This is kind of annoying...
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on July 25, 2014, 05:51:10 PM
I really hate to double post but I am concerned if I edit my last post the change might not be noticed.

Has anyone tried this? I understand that the code modifications might not be acceptable but this was done more for an idea or "proof of concept" than an actual submission worthy change. However I do think it gives standard Simutrans a different way to play that mirrors experimental to some extent without the complexity and problems.

Feedback on problems and suggestions might also help.
Title: Re: Suggestion for JIT mechanics revision.
Post by: jamespetts on July 25, 2014, 06:17:41 PM
Double posting is permitted if the second post is more than 24 hours after the first; I have not had time to try this, I am afraid, as I have been busy with trying to find a house, but I should be interested in others' experiences if they have tried it.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on August 11, 2014, 08:57:38 PM
Hi DrSuperGood,

finally I've got time to read this thread carefully and try out you patches. (Compiling on Linux now). I'm really glad you did the coding. I have a few suggestions. Your system is really very close to mine - i think we should join forces and take the best of them. I' try to describe an improved demand counter algorithm:

1. I suggest separate demand counters (and switch), for each input good.
2. decision when to increment the buffer:
- if the factory is producing and input is not over limit - increment by amount consumed
- if the factory is producing and input is over limit - increment by 50% of consumed amount (+1 I really like this - thats big improvement comapared to my original proposal)
- if the factory is not producing and input is empty - increment by possibly consumed amount (with or without power/pax/mail boost)
- if the factory is not producing and input is not empty - do not increment (this means that factory cannot produce because its consumers are overcrowded, or that it does not have all needed input goods available)

3. further improvements could be made in the algorithm when produced goods get their destination. The current behavior is that the goods go to the factory that has the least cargo waiting on the station. I recoomend sending it to the one with highest demand buffer.

4. Demand buffer should be initalized at its max value, equal to max input storage. I think it is sensible for real world factory to begin by ordering full storage...

Compilation is done - I got this warning, if you are interested:

===> CXX simfab.cc
g++ -O3 -minline-all-stringops -DNDEBUG -DREVISION="7287M" -Wall -W -Wcast-qual -Wpointer-arith -Wcast-align  -I/usr/include/SDL -D_GNU_SOURCE=1 -D_REENTRANT -DCOLOUR_DEPTH=16 -c -MMD -o build/default/simfab.o simfab.cc
simfab.cc: In function 'sint64 convert_goods(sint64)':
simfab.cc:65:95: warning: suggest parentheses around '-' inside '<<' [-Wparentheses]
simfab.cc: In member function 'void fabrik_t::step(uint32)':
simfab.cc:1393:77: warning: suggest parentheses around '-' in operand of '&' [-Wparentheses]


Now I'm going to open my game and see the improvements.




Leartin: your approach of establishing the link with exact capacity is fine as long as you have dedicated links between factories, especially trucks. But it completely fails if you deliver goods for multiple factories in one train. Then you need more capacity but cannot choose which goods you are loading. This system could help in such cases.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on August 11, 2014, 09:13:38 PM
Quote
2. decision when to increment the buffer:
- if the factory is producing and input is not over limit - increment by amount consumed
- if the factory is producing and input is over limit - increment by 50% of consumed amount (+1 I really like this - thats big improvement comapared to my original proposal)
- if the factory is not producing and input is empty - increment by possibly consumed amount (with or without power/pax/mail boost)
- if the factory is not producing and input is not empty - do not increment (this means that factory cannot produce because its consumers are overcrowded, or that it does not have all needed input goods available)

Essentially this would increase the demand at all times. The case of a factory far away which suddenly becames overvrowded (which why the in transit limitation was introduced) will reoccur when there is demand with an empty factory.

And I would like to try your patches. Where are they. The Simutrans JIT2.zip I cannot download, my provider tells me this is malware ... Please is there a zip with only the patch?
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on August 11, 2014, 11:19:01 PM
Essentially this would increase the demand at all times. The case of a factory far away which suddenly becames overvrowded (which why the in transit limitation was introduced) will reoccur when there is demand with an empty factory.
If you really want you can ovecrowd a factory. In-transit limit won't stop you either. But this system is IMHO much more helpful.

Quote
And I would like to try your patches. Where are they. The Simutrans JIT2.zip I cannot download, my provider tells me this is malware ... Please is there a zip with only the patch?
My testing is OK with simple chains, let's see tomoorow on a more complex game. Sources attached.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on August 12, 2014, 02:51:31 AM
Quote
1. I suggest separate demand counters (and switch), for each input good.
That is what I did? Each input and output has a demand counter and each input uses their demand counter separately. The reason outputs also have one that is never used or visible is because of poor design decisions (inputs and outputs should extend a production parent type, instead they share members for both of which some are never used).

Quote
- if the factory is producing and input is not over limit - increment by amount consumed
A special case is needed on the tick that consumes the last of the product in storage. This was not modelled as the current factory tick system is rather poorly structured.

Quote
- if the factory is producing and input is over limit - increment by 50% of consumed amount (+1 I really like this - thats big improvement comapared to my original proposal)
Percentage is adjustable. The only problem is no mater what you do it will still run at at-least 50% efficiency. It would be nice if people really messed up that it could reach lower percentages maybe (a hard mode?). However for such a loose definition as Simutrans industry this is the best one can do.

Quote
- if the factory is not producing and input is empty - increment by possibly consumed amount (with or without power/pax/mail boost)
The one I made goes through extensive effort to make sure to order the correct amount. This is especially problematic with power as if the factory has a transformer and is not actually producing anything you have to assume that it has sufficient power and order as if it did.

Quote
- if the factory is not producing and input is not empty - do not increment (this means that factory cannot produce because its consumers are overcrowded, or that it does not have all needed input goods available)
Not as easy as it may sound. Factories will keep producing even without a supplier at a diminishing rate until their output storage is full. Also you still need to order all goods that are consumed in equal ratios unless an overflow is applied. I also implemented this to some extent with the first rule, ordering as consumed, and a special refund case where if the order amount is too large it will scale back the other (under supplied situation).

Quote
3. further improvements could be made in the algorithm when produced goods get their destination. The current behavior is that the goods go to the factory that has the least cargo waiting on the station. I recoomend sending it to the one with highest demand buffer.
It currently chooses them in a round robin fashion out of all the destinations accepting orders. Orders are accepted for the entire duration of a tick to assure fairness (all suppliers given a chance to fulfil order) There is a bias towards some factories as a result of internal order and just how the numbers play out however it should make little difference except under extreme circumstances (there exists a good destination allocation pattern such that all consumers have their orders fulfilled yet this algorithm does not find it).

Quote
4. Demand buffer should be initalized at its max value, equal to max input storage. I think it is sensible for real world factory to begin by ordering full storage...
Initialization does need improvement. I just choose 0 as it is a clean value to test with. The buffer also needs to be saved and loaded.

Quote
===> CXX simfab.cc
g++ -O3 -minline-all-stringops -DNDEBUG -DREVISION="7287M" -Wall -W -Wcast-qual -Wpointer-arith -Wcast-align  -I/usr/include/SDL -D_GNU_SOURCE=1 -D_REENTRANT -DCOLOUR_DEPTH=16 -c -MMD -o build/default/simfab.o simfab.cc
simfab.cc: In function 'sint64 convert_goods(sint64)':
simfab.cc:65:95: warning: suggest parentheses around '-' inside '<<' [-Wparentheses]
simfab.cc: In member function 'void fabrik_t::step(uint32)':
simfab.cc:1393:77: warning: suggest parentheses around '-' in operand of '&' [-Wparentheses]
G++ pedantic errors. The code is valid and takes full advantage of the operator precedence of C++ (as defined by the standard) however G++ thinks it needs bracers as it could be I am an idiot and did not read operator precedence so is not executing in the intended order. Visual C++ compiler I used to write the code did not throw this warning which is why I never fixed it. I did not build with G++ so was not aware of it.

Quote
And I would like to try your patches. Where are they. The Simutrans JIT2.zip I cannot download, my provider tells me this is malware ... Please is there a zip with only the patch?
I have no idea how to generate a patch with the git system. I kind of downloaded the source code and hacked away on 2 files which I included in the zip. Unless dropbox somehow infected the zip it should not contain malware (your ISP probably is thinking that the 2 windows executables included are malware which is sad). Anyway I have no idea why I did not think to attach the code files directly here in the first place (it was only these two code files, which appear to be located in the root folder of the project).

In my opinion the entire industry tick function needs to be rewritten. There are too much hacky calculations applied everywhere that it is hard to get a proper industry production and consumption model working.

Of specific note is with factories that produce many output products the amount of raw material consumed is based on the most produced output and not the actual amount produced (especially case if one output is full). Also the outputs are all tied synchronously to the most produced output and do not scale production down in parallel as storage fills up (having a oil refinery using 100% gasoline and barely touching printer ink will result in the printer ink being restored at full production rate despite the output storage never getting near empty). In any case the result is that more product is consumed than is produced which usually means free money (more raw product to ship at a profit).

Same example, Oil Refinery with Gasoline used 100% and Printer Ink used 0-1% (hardly at all).
What is happening now is that it will be consuming oil at its full declared production rate. This is the same amount as if it was producing only Gasoline at 100% and if it was also producing Printer Ink at 100%.
What should happen is that it consumes oil at a reduced rate. This is because only part of the output is working at full power (lost production) so it consumes less.

There are many ways this could be implemented. Simplest is averaging production across all outputs (above example this would give 50% production) however some pak balancers may want different ratios (Printer Ink may use 2 times as much oil to produce as Gasoline so affects consumption rate more).

This is also the case with power where an industry will either use all or nothing. Power used should be based on the amount of work a factory does. This would make power grids a lot more stable as everything will settle to a largely stable base load which more accurately reflect a power fraction (out of maximum power demand).

There needs to be certain quantities defined.
Maximum Production: The absolute maximum amount of effort the factory can output. This is based on things like base production, pax, mail and power fulfilment.
Desired production: The amount of productive effort that is desired based on maximum production. This factors in production scaling down for each output with appropriate weightings and determines the amount to order (base order rate is always at the desired production rate scaled for the various inputs).
Actual production: The amount of production effort actually produced as a fraction of desired production. This is used with scaling and maximum production to determine the amount of product produced.

Maximum production factors in how much of the power demand was fulfilled. Actual production is used to scale power order for the next tick (so it uses only as much power as production done to get full power boost). Power plants need to work differently where their desired production is scaled based on the amount of power consumed (???? not possible ATM) and actual production is overwritten when storage runs empty to represent the amount of power produced from the remaining resources otherwise it is always 1. This would change game dynamics a lot (for the better?) as you can no longer simply supply power stations and expect them to generate electricity for nobody. Although they always appear to generate maximum power when working they will only consume based on what is used so they will reach 100% use when overloaded. Such a change as this fits quite well with the entire flow mechanics idea and JIT systems and even turns power into an industry chain (you need power consumers for power to work).

There should also be different industry flags for pakset authors to change how industries behave.
Synchronous Consumer: This is basically the current factory model, all inputs are consumed synchronously with all stopping if any storage runs empty.
Asynchronous Consumer: This is basically the current end consumers, all inputs are consumed asynchronously in parallel with overall production being lowered if at least one is empty (but not stopped).
Synchronous Producer: Like with the consumer, it is forced to produce synchronously. Factories kind of do this already but there would be stricter limits with a bottleneck occurring if output storage of 1 product is full.
Asynchronous Producer: Basically it fills the output storages as required with maximum production obtained when all outputs are empty all the time.
Power station: Produces power as described above. As long as it is working it will output maximum power but consumption is limited to how much power was used.

So what would these lose flags allow?
Something silly like a hamburger joint might have separate inputs for buns and patties yet require they are ordered synchronously as they cannot server a burger without both. This is currently not possible as the outlet would work at 50% production if 1 of the goods was lacking completely.
Something like a nuclear powerstation could be made to be a Synchronous Producer and Power Plant, meaning that you have to remove the atomic waste from it before it can generate more power.

The overall idea is to generalise behaviour away from hard coded implementations into mix and match flags/attributes you can assign an industry. You could even define other kinds of flags such as...
Shared Demand: All inputs share a common demand (scaled appropriately).
Interchangeable Input: Each input alone can now run the factory at full production if sufficiently supplied (intended with shared demand for a factory which can use 1 of many raw materials to make a product). If not shared demand present then it would order based on least demand.
etc...

Such flags would be a great extension but not directly needed for a production revision.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on August 12, 2014, 09:25:50 AM
The power depends on the production done. You can consumer more or less power. It requires setting on the dat files to use this, otherwise it will just use the default settings for backward compatibility.

Power plants not consuming anything without a consumer would force people to built powerlines. That is somehow against the spirit of simutrans; o.e. you are not forced to do stuff. You can happily do only freight transport, or only passengers, and you can use to ignore power plants.

Obviously one could introduce a setting to force power plants to consumer at reduced rates (10% or so) if no power is demanded.

The suggestion to make the consumption dependent on how much is produced is certainly good. Consuming base on actual total produce instead to a fixed number sound good. However, I am not sure how much it would change in the end. Very few producer produce more than two.

The reason is that Simutrans industry model is "broken" rather in the sense it does not really guess very well what industries are needed next. Producer with too many output are very hard to balance so all outputs can be used.

There have been quite few attempts to work a that out over the years. But in the end the whole options (crossconnect all/decidated connections/production increases by power/very complex factories in pak128) make this almost impossible to improve beyond what is there now. (Which is not far from what was done in the very beginning.)

I do not think the synchronous end consumer will happen that much that you have to worry about it. Simutrans is already conplex, and there can be too much complexity for too little gain.

Imho most gain would be a patch to base the consumption on the actual output if there is more than one good (implementation like by consuming per each good separately).
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on August 12, 2014, 06:40:28 PM
Quote
Power plants not consuming anything without a consumer would force people to built powerlines. That is somehow against the spirit of simutrans; o.e. you are not forced to do stuff. You can happily do only freight transport, or only passengers, and you can use to ignore power plants.
Not entirely true. It would turn power stations from being an end consumer (they will take whatever you can ship to them) to a factory (with production that requires consumption). This would mean that power becomes another part of the industry chain.

I personally think this would add to the game as currently power stations like coal and oil are the first industries people supply as they are simple no step source to destination routes. With the new dynamic they would need to first run other industries moderately well before there is enough power demand for the power stations to become profitable. Basically power becomes part of the chain kind of like you need enough people connected for a lot of passengers or you need to offer commuter services for consumers like bookshops for them to have decent demand. Basically all other chains have similar "forces" so why not power? Only difference is that suddenly people actually use power infrastructure outside of a few specific purposes.

Obviously power system generally would need improvement. This includes pay being based on distance (so transferring power across the map to that remote factory is not profitable by itself) and the ability to share power networks (with some fairness in pay). Until then power stations can just be consumers like they are currently now with desired production always being 100%.

Quote
Obviously one could introduce a setting to force power plants to consumer at reduced rates (10% or so) if no power is demanded.
Having something like that would be good to keep them ticking over. The key thing is that they should only produce at full if they are nearly fully used so that people are encouraged to actually use power. Currently power is so easy to max efficiency and in pak64 a single oil power station can earn you several million a year with a simple 2 station train line where as something like books requires you to grow the shop's city, offer it good commuter services, set up all kinds of factories so finally you get a good throughput of products to it. Power is just too easy in my opinion which is why in every online game you will find such power stations are always built up unless it happens to be very fresh (no one noticed it yet) or bugged (it cannot be supplied, like that pak128 oil-fired power station on Fifty's server due to max in-transit computation error).

Quote
The suggestion to make the consumption dependent on how much is produced is certainly good. Consuming base on actual total produce instead to a fixed number sound good. However, I am not sure how much it would change in the end. Very few producer produce more than two.
The order in which the factory production is computed could be changed to accommodate it. Looping through all outputs to first work out the desired production (which factors this in) before looping through the inputs to work out exactly how much can be produced with it then finally looping through all outputs to actually produce the product. This is kind of the inverse of what happens at the moment.

Quote
The reason is that Simutrans industry model is "broken" rather in the sense it does not really guess very well what industries are needed next. Producer with too many output are very hard to balance so all outputs can be used.

There have been quite few attempts to work a that out over the years. But in the end the whole options (crossconnect all/decidated connections/production increases by power/very complex factories in pak128) make this almost impossible to improve beyond what is there now. (Which is not far from what was done in the very beginning.)
This all comes down to economic models. I would say the best approach for this is to view each factory/source/destination as their maximum production (all bonuses applied) and then work with some rules. All end consumers should always have sufficient supply (more production of their consumed goods than needed). Middle factories operate slightly less efficiently with insufficient demand for their total production. Raw materials have a lot of extra production capabilities. This forms a pyramid with the top being the bottleneck on productions and the base being very wide and capable of support holes (you may not need that extra steel mill, or may not need to connect that iron ore mine at all with the end consumers still working all the time). Fine tuning could be done with individual factories raising or lowering base production (which appears to be in the code but not use?) However this is more a discussion for an economic model revision and not a factory mechanics revision.

Quote
I do not think the synchronous end consumer will happen that much that you have to worry about it. Simutrans is already conplex, and there can be too much complexity for too little gain.
The idea is that it adds no complexity to the code as you run it past that step anyway as you need that mechanics for factories. Currently all the mechanics are separated (with end consumers treated by a different branch of the code) which results in a lot of procedural coupling in the end.

Quote
Imho most gain would be a patch to base the consumption on the actual output if there is more than one good (implementation like by consuming per each good separately).
Except that starts to add the problem of [inputs * outputs] worth of operations which scales very badly (some strange pakset might have a factory with 20 inputs and 20 outputs, the time would start to become significant if many such factories existed).
I think running extra passes through all inputs/outputs would scale better.

Basically a pipeline like follows.
1. For all outputs, compute the desired production (scaled for their current storage level so that only as much work is done as required).
2. For all inputs, compute the amount of production that can occur based on input. This is a fraction of work done.
3. For all outputs, apply the production scaled based on intended amount for that output scaled by the fraction from 2.
4. Consume all inputs based on the fraction from 2.

I mentioned 2 and 4 separately as they would need separate passes unless you take advantage of negative storage amounts (allow them but stop when negative). This would still keep average production as precise but instead simplifies the logic since a factory production is boolean (on/off) and not (on but at some fraction due to insufficient inputs at this tick). Maybe simplicity is the best thing in this case.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on August 12, 2014, 08:33:30 PM
It would turn power stations from being an end consumer (they will take whatever you can ship to them) to a factory (with production that requires consumption). This would mean that power becomes another part of the industry chain.

It would be a different kind of link in the chain then, because all other links are mandatory, whereas power distribution is just extra fluff for those that are into it.
Title: Re: Suggestion for JIT mechanics revision.
Post by: hreintke on August 12, 2014, 09:00:31 PM
Quote
It would be a different kind of link in the chain then, because all other links are mandatory, whereas power distribution is just extra fluff for those that are into it.

You could generalize this and to "optional inputgoods" and/or "outputgoods recipies" and get very flexible chain definitions.

Discussed a little in this thread : http://forum.simutrans.com/index.php?topic=11865

Herman
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on August 12, 2014, 09:37:12 PM
That is what I did? Each input and output has a demand counter and each input uses their demand counter separately.
Ah, fine. I did not read the code yet, only the forum posts, which made me think there is only one counter per factory.

Quote
Initialization does need improvement. I just choose 0 as it is a clean value to test with. The buffer also needs to be saved and loaded.
I gave that another thought and 0 seems to be a good value. Well supplied factories will have demand near zero, while demand of  undersupplied factories will grow to the maximum. For many cases saving/loading is not very important. Notable exception would be a factory link where convoys have higher capacity than stations - typical for ships.

The rest is getting a bit out of topic, but I want to say something to it.

Factories with multiple outputs: if we take oil refinery as an example, you cannot separate the outputs. Oil refinery splits the oil to fractions, so it can make only X liters of gasoline from Y liters of crude oil. It does not matter if it uses other fractions to make ink, plastics, chemicals, or not. So you can think that it is just throwing away the extra production that nobody wants. Similar situation is with sawmill (pak128.britain), you cannot produce planks without producing woodchips. But you can just throw them away.

Of course there are other situations where it should be the other way round. Orchard can produce apples and cider, but you should get either a box of apples, or barrel of cider, not both. However this could be solved at pak level - let orchard produce only apples, and have another factory that turns apples into cider. And you could choose to deliver apples to the cider factory or directly to the market.

Similar thing applies for input goods. I'd like to have e.g. a cannery that can accept either meat or fish or fruit or vegetables (for foodstuffs), and either steel or glass (for package). Currently I would have to supply all of that, but I would like it to produce if at least one foodstuff and one packaging material is suplied. Yes again this could be done by having 4 (or 8) different canneries...

Well instead of sync/ansync groups or models it could be generalised by having a logical formula for inputs and outputs. Something like this: cannery input = ( meat || fish || fruit || vegetables ) && ( steel || glass ); sawmill output = ( planks && chips ), orchard output = ( fruit XOR cider ). Current behavior is && everywhere, except for end cunsumers which use ||

Furhter idea could be that some goods are only a booster (fertilizer for farms, grain for feeding livestock, coal instead of electricity for glass, cement and other factories)

Power plants - thermal power plants (coal, oil, nuclear) cannot easily adjust their output power. They operate best at full power. They cannot slow down and use less fuel if there is lower power consumption. The extra power is just "thrown out of the window". (Its more complex but my english is not that good to explain it). So I'm against binding power plants fuel consumption to electricity consumption. And I really like the fact that there are these simple chains, especially at the beginning of the game, so that you can get some steady income to fund building the infrastructure for more complex chains. If this is ever implemented it shoud be made optional, and the power plant should start full production when something gets connected.

The other suggestion that power boost (consumption) should not be on/off but proportional to production is fine. I would base it on  fullness of input storage - if it is near empty, boost a little. if it is almost full, boost almost 100%. If there are more inputs use the minimum.

A note for Leartin: please join the testing. You have opposed JIT2 on the ground that you are comfortable with the current JIT and do not want it broken. Please try JIT2, and see yourself if it breaks yor way of playing. I think it requires the same attention from player in regard to choosing the right amount and capacity of convoys. It just makes the flow of them more smooth and regular.  I have seen it during my test. I have a link where 2 trucks are not enough, and 3 trucks are a bit too much. With old JIT the consumer was slowly overcrowded, and then all 3 trucks piled up at the producer, blocking the road, and traffic around. Not for long but still enough to delay other deliveries. This does not happen with JIT2, as the trucks are loaded gradually and come in regular intervals.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Leartin on August 12, 2014, 11:07:48 PM
A note for Leartin: please join the testing. You have opposed JIT2 on the ground that you are comfortable with the current JIT and do not want it broken. Please try JIT2, and see yourself if it breaks yor way of playing. I think it requires the same attention from player in regard to choosing the right amount and capacity of convoys. It just makes the flow of them more smooth and regular.  I have seen it during my test. I have a link where 2 trucks are not enough, and 3 trucks are a bit too much. With old JIT the consumer was slowly overcrowded, and then all 3 trucks piled up at the producer, blocking the road, and traffic around. Not for long but still enough to delay other deliveries. This does not happen with JIT2, as the trucks are loaded gradually and come in regular intervals.

I don't want to test it since I'm sure it does not break my way of playing, only my expierience. The time you are happy to save is pretty much the time I lose in gameplay value with no gain from it, since the intended gain is what I don't like about it. As long as I have the option to play the current JIT, I'll play that. Once I can't anymore, it does not matter if I like it or not, I'd need to adapt anyway. What I want and what JIT2 does are just two completely different things, so I'd much prefer you go on and make JIT2 the best it can be without interference of an opposer.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on August 13, 2014, 04:25:20 AM
Quote
Power plants - thermal power plants (coal, oil, nuclear) cannot easily adjust their output power. They operate best at full power. They cannot slow down and use less fuel if there is lower power consumption. The extra power is just "thrown out of the window". (Its more complex but my english is not that good to explain it).
Not entirely correct. Power stations are seldom 1 reactor, often with several. Power output can be adjusted by varying the number of reactors online at a given time. In the case of a coal powerstation this would mean shutting off many of the fires/boilers resulting in less steam being generated.

If you are talking reality, a power station not connected to a load cannot even generate electricity as it would blow the generators apart. Economically without generating electricity there is no reason to even have the power station consuming anything at all. Thus my suggesting making sense.  That said, there could be production tiers with power stations (20%, 40%, etc) so that 100% utilization is not required, only enough to reach the 100% tier.

Quote
And I really like the fact that there are these simple chains, especially at the beginning of the game, so that you can get some steady income to fund building the infrastructure for more complex chains. If this is ever implemented it shoud be made optional, and the power plant should start full production when something gets connected.
That is the problem... They are so simple and profitable you really do not want anything but power. Instead of "hay a new bookstore has opened" you say "awww it is not an oil-fired power station". There should not be such easy money to be made. Power is almost always the first industry to be supplied and I often see 3 players fighting over power stations and even complaining and being rude if they cannot get one all to themselves.

Quote
The other suggestion that power boost (consumption) should not be on/off but proportional to production is fine. I would base it on  fullness of input storage - if it is near empty, boost a little. if it is almost full, boost almost 100%. If there are more inputs use the minimum.
Storage has nothing to do with how efficiently something can run. It is there to act as a buffer between shipments, not some strange production process. The amount of power consumed should be based on the amount of work done as in the end a factory not producing anything is not going to take much power. This would in turn be based on the fullness of output storage since output storages define the production rate. Currently there appears to be logic for some kind of cut-off where output is above 75% it stops using power but until then it consumes at full power rate. Since a flow based system depends on the storages being high for continuous production, this could be a problem so by making it based on the amount produced you will find that your entire power network reaches an equilibrium and remains approximately stable around it, which would be a huge improvement over now where in large networks I have had them fluctuate by ~60% from having tons of spare power to being heavily overloaded.

Quote
Factories with multiple outputs: if we take oil refinery as an example, you cannot separate the outputs. Oil refinery splits the oil to fractions, so it can make only X liters of gasoline from Y liters of crude oil. It does not matter if it uses other fractions to make ink, plastics, chemicals, or not. So you can think that it is just throwing away the extra production that nobody wants. Similar situation is with sawmill (pak128.britain), you cannot produce planks without producing woodchips. But you can just throw them away.
Except this is not what happens in real life. Oil refineries try and use naturally occurring compounds to make a substance but if there is not enough they can synthesise from other factions through various processes. This obviously is not viable for petrol or diesel but for more specialist chemicals it probably is. Also in the case of a sawmill, the woodchips can be converted into kinds of artificial planks (which are commonly used everywhere in this day and age).

Anyway, subtract all the "desired features" that people are coming up with now and see what I produced. Personally it appeared a viable way to play with no glaring faults when I tested it. Bottlenecks are a problem but since you now have flows you can enable routing restrictions to deal with those.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on August 13, 2014, 05:29:47 AM
If you are talking reality, a power station not connected to a load cannot even generate electricity as it would blow the generators apart. Economically without generating electricity there is no reason to even have the power station consuming anything at all. Thus my suggesting making sense.

Power stations in Simutrans are always supplying something: all the houses and shops. Since Simutrans isn't Sim City, this isn't simulated, it is just assumed to be there.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on August 17, 2014, 10:44:29 PM
Quote
Quote
The reason is that Simutrans industry model is "broken" rather in the sense it does not really guess very well what industries are needed next. Producer with too many output are very hard to balance so all outputs can be used.

There have been quite few attempts to work a that out over the years. But in the end the whole options (crossconnect all/decidated connections/production increases by power/very complex factories in pak128) make this almost impossible to improve beyond what is there now. (Which is not far from what was done in the very beginning.)
This all comes down to economic models. I would say the best approach for this is to view each factory/source/destination as their maximum production (all bonuses applied) and then work with some rules. All end consumers should always have sufficient supply (more production of their consumed goods than needed). Middle factories operate slightly less efficiently with insufficient demand for their total production. Raw materials have a lot of extra production capabilities. This forms a pyramid with the top being the bottleneck on productions and the base being very wide and capable of support holes (you may not need that extra steel mill, or may not need to connect that iron ore mine at all with the end consumers still working all the time). Fine tuning could be done with individual factories raising or lowering base production (which appears to be in the code but not use?) However this is more a discussion for an economic model revision and not a factory mechanics revision.

Since you want some choice, it means that more than one producer is producing for more than one consumer. How to balance this out is non-trivial. Furthermore there are certain goods designed to be in short supply, like coal in pak64. So it is a strategic decision either power the steel mill or the coal power plant. That is very though to get right.
Title: Re: Suggestion for JIT mechanics revision.
Post by: TurfIt on August 24, 2014, 11:25:06 PM
Some threads you may have missed as these subjects were brought up here again:
  Limit on storing incoming goods in factories (http://forum.simutrans.com/index.php?topic=8943.0)
  enable factory_max_input_capacity to be set from config files (http://forum.simutrans.com/index.php?topic=9090.0)
  Electrical issues (http://forum.simutrans.com/index.php?topic=10798.0)


As for the proposed JITv2, I keep flip flopping if it's good or not... Having all the suppliers dump their entire output stores into their attached stations (and the transport network) when a consumer starts up doesn't really make much sense. So throttling this to the rate of consumption is good.

However, one of the stated goals of JITv2 is to 'solve' the too far distance between supplier and consumer causing production stoppages at the consumer. I fail to see how this algorithm does so. The critical distance is simply doubled instead.

e.g. Take a factory that consumes 1000 units/month with a 1000 unit input storage located exactly 1 month travel away from its supplier who has infinite production.
Under the existing JIT system, once the input becomes full, the supplier stops sending. convois already enroute continue their deliveries resulting in possibly a quite large overstock. Consumption eventually results in the storage failing below full, and the supplier resuming shipments. The first convoi departs and arrives just as the storage empties. All's good.
Now move the consumer to be 6 weeks away from its supplier, and the storage sits empty for 2 weeks until the first convoi shows up. Enter JITv2 and the supplier continues to send at a 50% rate once the storage is full. Hence some convois continue arriving, and the consumer never runs out fully.
However, now lets move the consumer to be 2 months away. At the point the consumers storage falls below full resuming the 100% shipment rate, the entire transport system is full of convois delivering at the 50% rate. It will take two months before the first 100% convoi arrives, there's one month stock in storage, and one month stock in the 50% convois. Hence the first 100% convoi arrives just as the storage empties. Back to the All's good.
Move the consumer even farther, and there will be times the consumer runs out. Exactly the same as JITv1 (excepting the consumer produces at 50% rather than 0% during this time).


Ordering based on full electric boost but current pax/mail boost make no sense to me. Why the distinction? IMO just order based on the current consumption rate.

For intermediate factories, why link the input stores together with the 'lost order' mechanic? IMO the inputs should be independent. The linking is done naturally by the consumption ratios when the factory produces.
With the linked orders, it's possible to starve a factory on one input even when it's well inside the JITv1 critical distance. As a test, I setup a steel mill, and started supplying it with coal. Once the iron demand hit max, coal orders ceased which didn't make much sense. I then oversupplied it with iron. Once the iron became overfull, both iron and coal stopped ordering. After the convois in the pipeline emptied out, there was 100/150 coal, and 900/500 iron in the factory. Once steel started flowing and production resumed, the coal stock was exhausted before the first convoi could arrive. If instead the coal had continued ordering until it was also full, then the full stock of 150 would have prevented the shortage. This cycle continued as the steel consumption rate cycled. Sometime the steelmill would be left with only 20/150 coal, sometimes it would also be overful. Really depended on the exact timing of the convois. In the 20/150 case, the shortage also caused a steel shortage downstream. In this case, JITv2 caused production stoppages that would not have occurred with JITv1.

Having .accepting apply for an entire step cycle is good. Every supplier then has a chance to supply instead of just the first ones stepped. This should be back applied to JITv1, and the max_intransit limit so certain factories aren't preferred.

Speaking of max_intransit, JITv2 is completely ignoring this. The whole purpose of it (IMHO) is to prevent players just dumping goods into a station outside the factory and bypassing the JIT setting. So, IMO JITv2 should respect this too. Of course the opposite occurs too with malicious players maxing out intransit and shutting down factory chains, but then multiplayer games do require active moderation to have any hope of smooth working anyways...

And, WTF happened with max_instransit - it was based a percentage of the input storage which make sense. Now it's based on the sum of a percentage of the outputs of the suppliers? huh?
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on August 25, 2014, 01:01:59 AM
Quote
However, one of the stated goals of JITv2 is to 'solve' the too far distance between supplier and consumer causing production stoppages at the consumer. I fail to see how this algorithm does so. The critical distance is simply doubled instead.
It works because goods turn into flows. If you have a chain of trucks from a coal mine to power station 100 units away or 1,000 units away or even 100,000 units away there is no difference in power station operation since as fast as it consumes the coal more coal is being ordered. Instead of waiting for storage to become less than maximum storage then ordering everything it can, it instead orders a constant trickle from all suppliers which in turn results in a constant trickle into its storage.

That said the algorithm is not perfect...
1. Transfer bottlenecks without any transfer restrictions can grow potentially infinitely. Having 2,000,000 units of oil into an oil transfer is completely possible (however unlikely if the company manager knows the slightest what he is doing).
2. If an in-transit bottleneck was to occur which results in a storage overflow it is possible that the same problem with burst production consumption can occur. However it is limited at worst case to the overfull order fraction (50% in my demo) no mater how big the distances involved are.

Quote
Move the consumer even farther, and there will be times the consumer runs out. Exactly the same as JITv1 (excepting the consumer produces at 50% rather than 0% during this time).
Yes except that is kind of the point. It will only run at 50% use as a penalty for you overflowing its storage (this also serves to "burn off" backlog in case of changes of in-transit amounts). However if you deliver in appropriate shipment sizes it will never overflow the storage since it orders exactly as much as it consumes. Since it keeps ordering at 100% even when empty a buffer in the storage will build up to compensate for shipment difficulties.

There are problems with existing pakset balance such as marketplace in pak64 with food addon. The max storage of the consumer can be so small that it is not possible to supply it end game without causing it to overflow (there are no non-obsolete convoys capable of shipping that little). However this becomes a pakset issue and is easily resolved by raising it to 100-200 units max in storage. You still cannot supply it with 12 long trains without it overflowing however you can have 1-2 trucks arriving close together to account for transport dynamics.

Quote
Ordering based on full electric boost but current pax/mail boost make no sense to me. Why the distinction? IMO just order based on the current consumption rate.
Because when not consuming anything the power boost disappears (bad design of power boost). As such it would be ordering at the mail/pax boost rate and not the power boost rate despite it being capable of consuming at the power boost rate because the consumer is sufficiently supplied with power (just not currently reflected on the production rate due to it not producing anything). Since it is not using any power and you cannot get the attached power net (no relationship in that direction) we cannot determine actually what power boost it is going to receive so we have to assume 100%. When it is producing then we simply use the current production rate since now power is being applied.

What power should do is treat each factory as a load and vary the power used depending on work done. Under this definition an idle factory connected to power will get 100% power boost so we can directly order the amount. However the current definition is binary and the power boost vanishes completely when idle so we would order too little if there was no compensation.  In theory it would eventually correct itself without this compensation but it could take in the order of years to reach 100% production so it is far better to just make the assumption (if connected to any transformer and not producing we are running at 100% power boost) and over order if necessary (which can be fixed by under ordering later) than to sit idle trying to slowly build up the order.

Quote
For intermediate factories, why link the input stores together with the 'lost order' mechanic? IMO the inputs should be independent. The linking is done naturally by the consumption ratios when the factory produces.
Because in the case of a supply bottleneck you would end up overflowing the other inputs with no real limit. Yes if the bottleneck is fixed it will eventually stabilize thanks to the overflow order fraction (50% in demonstration) however until then and under bottleneck conditions nothing would stop the order storages reach the hard-coded maximum of 15,000 units and still keep ordering. For this reason if the order buffer overflows causing orders to be lost then it also loses orders for other goods so that the supply bottleneck is transferred to the orders of the other inputs.

Quote
Once the iron became overfull, both iron and coal stopped ordering. After the convois in the pipeline emptied out, there was 100/150 coal, and 900/500 iron in the factory. Once steel started flowing and production resumed, the coal stock was exhausted before the first convoi could arrive. If instead the coal had continued ordering until it was also full, then the full stock of 150 would have prevented the shortage. This cycle continued as the steel consumption rate cycled. Sometime the steelmill would be left with only 20/150 coal, sometimes it would also be overful. Really depended on the exact timing of the convois. In the 20/150 case, the shortage also caused a steel shortage downstream. In this case, JITv2 caused production stoppages that would not have occurred with JITv1.
Middle industries are particularly difficult since their actual rate of consumption is based on their linked consumers. For this reason additional stalls are required. The rate of ordering is based on the rate of production. Since the output storage is getting full the rate of production falls resulting in less being ordered. Eventually ordering stops as output storage approaches full (it might not physically stop but the units are so small it might as well be considered stopped). However once the steel mill gets going eventually orders out reach an equilibrium where the production rate of the steel mill sufficiently supplies the linked consumers. Under this circumstance it is now possible to know how much has to be ordered so the correct amount will start to follow into the steel mill.

You are correct that distance will cause some temporary issues. The steel mill may run out of input and so start ordering at 100% (more than is required). This may eventually cause an overflow (if the mill is not used 100%) which may under order for a while. However due to constant orders it should eventually reach an equilibrium. When the steel mill works at 100% this is not an issue.
Quote
Having .accepting apply for an entire step cycle is good. Every supplier then has a chance to supply instead of just the first ones stepped. This should be back applied to JITv1, and the max_intransit limit so certain factories aren't preferred.
JIT1 does not suffer supply preference. All factories will try and supply the consumer with a load if it is accepting and less than maximum in-transit. In the case of 2 accepting factories it uses a round-robin selector to be fair. Since maximum in-transit is so large it is virtually impossible for it to be filled only in 1 tick, the amount of bias left would be only for the final tick where it may stop before visiting all factories however this deviation is minimal and under loaded conditions the bottleneck will be the rate of production for suppliers anyway.

Quote
Speaking of max_intransit, JITv2 is completely ignoring this. The whole purpose of it (IMHO) is to prevent players just dumping goods into a station outside the factory and bypassing the JIT setting. So, IMO JITv2 should respect this too. Of course the opposite occurs too with malicious players maxing out intransit and shutting down factory chains, but then multiplayer games do require active moderation to have any hope of smooth working anyways...
That is not within the scope of JIT2 and instead is part of the game's transfer mechanics. However since JIT2 makes flows of goods you can now much more easily use transfer restrictions like no routing if overcrowded or drop at overcrowded. The reason for this is that the amount of goods at any stop in your network will now be pretty constant so with some storage planning it is easy to get transfers not to overcrowd. If you look at in-transit graphs of well established games for industries without a transfer bottleneck you will see how instead of huge spikes it flat lines at a very low level (the optimum in-transit, virtually nothing waiting and mostly actually in-transit on convoys). This is perfect for use with transfer restrictions that are otherwise discouraged as no one can handle a burst of >50k tons of coal to a power station in a huge coal network.

Also with the recent rise of max in-transit from being a multiple of storage to a multiple of output for every supplier this has become a major problem even with max in-transit restrictions. I mean on fifty's nightly I obtained a massive 200,000 units of oil in a transfer (as oil refineries had max in-transit of well over 150,000 units). Upon deep analysis I decided that maximum in-transit is kind of a useless mechanic. All it does is soft apply a maximum map size and eventually stop people shipping more once their transfer has over 150,000 units in it (or more for big maps). At this stage we might as well let them keep shipping as the fact is they will not be maximizing profits and if they ever collapse it (fix the bottleneck) they will suffer a 50% production penalty for several years. Obviously they could use the cargo bounce exploit and things to make infinite money from the cargo but that is kind of cheating anyway and they can even do it with still shipping to the end destination without a bottleneck.

A solution? Well anything would need to modify the cargo router to detect a bottleneck or upon arrival to a transfer feed back to consumer of a bottleneck. What to do with that information I am not sure. However all this is kind of pointless when there are already transfer restrictions which are perfectly suited for JIT2. In the end you should not be able to transfer more into a stop than it can hold, that is another problem unrelated to the industry model.

Quote
And, WTF happened with max_instransit - it was based a percentage of the input storage which make sense. Now it's based on the sum of a percentage of the outputs of the suppliers? huh?
That was not me (some other developer made this change). I think it was to address the soft map size issue that max in-transit applies where with enough suppliers far enough away you could get it that 100% of the maximum in-transit amount was in actively moving convoys which end up insufficiently supplying the consumer. In the end maximum in-transit makes little sense and is purely a work around for poor production and transfer mechanics which JIT2 when applied to transfer restrictions such as no route over overcrowded solves.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on August 25, 2014, 03:23:33 PM
Hi,

just a few comments from testing. I had not so much time to play, but anyway. I have a map (San francisco bay) with long distances between factories and low capacities, so I am using mostly trucks. I have a china shop whose input capacity i less than the truck's, so I have to run at 80% load to keep the shop running near 100%. It is really JIT, as the truck usually arrives just at the moment when last stock is being sold. Thus a traffic congestion can interrupt the production, but otherwise it is OK. The trucks supplying clay to pottery run nicely spread over the whole distance, clay pit is supplying the right amount of clay, always only one truck waiting for the load.  :thumbsup:

However I struggle with fishery. I have 12 boats supplying fishing port, and 4 trucks from fishing port to fishmonger. Input capacity at both is only sligtly larger than the capacity of boat (truck), so it is easy to overfill them. It seems to take forever to find the equilibrium, so I had to help it a little - stopping the truck just before unloading and waiting for the shop to consume enough fish to prevent overload. Anyway it is still much better than old JIT, which would easily leave the fishing port or shop out of supply for months. Maybe the 50% penalty for overcrowded input is too harsh in this case.

TurfIT: max_in_transit is IMHO really bad solution to JIT problems. You need different in_transit percentage for different distance between factories. Experimental tries to solve it by scaling max_intransit by measuring the delivery time, however JIT2 does not need it at all. Thats why it is ignored.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on August 25, 2014, 11:15:51 PM
Quote
Maybe the 50% penalty for overcrowded input is too harsh in this case.
It is needed to "burn off excess" as otherwise the storage will never decrease unless suppliers become overloaded or shipments stop (eg if a supplying company bankrupts).

I agree it is not optimal however what are the alternatives? I could "discard" excess incurring you a financial penalty based on distance from source however that could possible be abused. Alternatively discarding could occur with bonus income if shipments fall under the storage amount (shipping combos, with a maximum income multiplyer after enough is shipped correctly and resetting if discarding occurs).

However I still think the current idea of 50% order penalty is by far the most intuitive and is also very easy to implement. A smaller amount could be used but then there is no incentive to not overfill the buildings. It is obvious that some pak sets will need some balancing. I would advise raising storage for such industries as you described to accommodate for the new system. Anything in the 100-200 range would be suitable for trucks for most industry types but not for trains or trains of longer lengths.

Starting next week I will look into a second attempt at this, hopefully one that will be more publish friendly.

A possible optimization and simplification is to get rid of partial consumption (retro-active with the current JIT as well). Currently if input is insufficient it will only produce and consume what it can. This requires a lot of logic to handle for something which makes no difference in the end. Instead the number of "empty" inputs could be cached in a variable (that can even be computed at load time! so not extra data saved). For factories if this is above 0 then production halts. For consumers if it is above 0 then it is known as consuming less. What this does mean is that you could get negative storage (-0.xxx of an input as an example) however this makes no difference in the big picture as still the same amount gets produced on average and the amount produced from overshoot is as good as nothing (1 tick of full production at most, which gets deducted from the next shipment anyway). For the UI values below 0 could show 0. This at least would take advantage of the fact signed integers are used for storage amount. It could also allow for some considerable optimizations to industry ticks since a factory with empty cache being greater than 0 can return almost immediately and no extra loop through inputs is required to determine the minimum production possible as it would be all or nothing.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on August 26, 2014, 06:36:06 AM
I must say that jit2 is IMHO really good as is. I just didnt have much time to test thoroughly.

I have one idea about input storage, but I am afraid it will interfere badly with jit2. It is so that factories will never accept more cargo than their input, leaving the excess stuff on the station. Thus you could increase the input store by adding extensions to stations. But this will just shift the overcrowding from factory to station and will need no-route-via-overcrowded station.

Hmmm. Or an option to build storage extensions directly to factories so that player can extend the storage as needed.

Sent using recycled electrons.

Title: Re: Suggestion for JIT mechanics revision.
Post by: scamps on August 26, 2014, 09:42:40 AM
Any chance for a lazy player to get a windows executable?
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on August 26, 2014, 10:12:20 AM
I can offer linux executable...

Sent using recycled electrons.

Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on August 26, 2014, 02:24:11 PM
Quote
Any chance for a lazy player to get a windows executable?
I attached one in my post. If it works for you is another question (some people said it did not work). I do apologize if this is the case but I have little idea how to build perfectly portable windows executable and am using compilers and libraries that are about 4-5 years old. It works on my computer as that is how I did the testing.

You will need the visual C compatible pthread dll, this was something I forgot to bundle with (normal simutrans is built with G++ so uses the GCC compatible pthread dll).
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on September 09, 2014, 07:30:13 PM
I have played with JIT2 for a while now, and I am very satisfied with it. It helps a lot specially in cases where there is more goods in supply than demanded.

I would only suggest two improvements:

- display the demand counter in factory info
- make it a config file option, so that player can choose: JIT, JIT2, freeplay

I recommend accepting JIT2 into simutrans, as it solves frequent beginners question:

- overcrowded stations at source - if there is an oversupply, a player may try to buy more and more convoys and build more infrastructure to haul all the available cargo, just to find out that after a while, the destination is overloaded and not accepting any more, and all his expensive vehicles sit idle until the overstock is used. However when playing with JIT2, source stations are overcrowded only if the destination is willing to accept that without getting overloaded. (Or at least not overloaded for too long).

It also helps in other situations - recently discussed problem with supplying a coal plant and steel mill with one train carrying both coal and ore. With jit2, all cargo is loaded in proportion to demand of consumer, so that they are supplied more evenly. Without JIT2, you could often get one consumer overloaded and the other one starving.

This is even more visible if the consumers are not at the same station, and the train has to drop off part of the cargo on one station, then continue (half-loaded) to second station, and then return for full load. Without JIT2, you often end up supplying only the closest factory until it is overloaded, while the second is idle. Thus you usually end up having dedicated lines for both industries, but that needs more investment in vehicles and infrastructure. And you can't afford that in the beginning of the game.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 09, 2014, 10:51:47 PM
Quote
- display the demand counter in factory info
It actually does this! It replaces the maximum in-transit counter with the demand counters (since maximum in-transit is not used in JIT2). However this is only on by default for Pak64 with Pak128 using the traditional display.

Quote
- make it a config file option, so that player can choose: JIT, JIT2, freeplay
More difficult as this involves touching other files. However that is the next step.

I am going to try a second attempt at this quite soon and hopefully tag it with some other improvements (even possible optimizations).

For example processing industries could terminate early if last cycle ran into insufficient goods and no shipments arrived.
Another example is that instead of computing maximum production based on input always, it could assume maximum production and consumption and if consumption fails (storage less than 0) then apply a correction to production and consumption (production stop events are rare, most industry ticks will not have a stop event).
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on September 11, 2014, 07:13:18 PM
It actually does this! It replaces the maximum in-transit counter with the demand counters (since maximum in-transit is not used in JIT2). However this is only on by default for Pak64 with Pak128 using the traditional display.
How can I enable it for other paks? (especially pak128.Britain)
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 11, 2014, 09:57:27 PM
Quote
How can I enable it for other paks? (especially pak128.Britain)
It is a new feature to RC120 and later. To be honest I do not know but I would imagine it is some game configuration. The text is generated by a conditional statement.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on September 11, 2014, 10:14:47 PM
A configurable JIT model would indeed be very nice, as JIT2 has gotten some varied feedback.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on September 12, 2014, 05:06:11 AM
I'm a bit puzzled by why JIT2 behaves differently between pak sets. Or more correctly: how. The game doesn't really know what pak set is running. It just does things the way the pak set dictates, and it seems strange that pak sets already contain configuration parameters for JIT2.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 12, 2014, 05:32:11 AM
Quote
I'm a bit puzzled by why JIT2 behaves differently between pak sets. Or more correctly: how. The game doesn't really know what pak set is running. It just does things the way the pak set dictates, and it seems strange that pak sets already contain configuration parameters for JIT2.
It does not behave differently at all.

Between 112.3 and RC120 some configuration option was added to show extended factor storage information. The only difference is that the storage string also includes maximum in-transit amount in brackets next to the current in-transit amount. By default newer versions of Pak64 have this turned on. Pak128 on the other hand does not and still uses the classic non-extended display.

Since there are no maximum in-transit numbers in JIT2 (well there are but they are never used in the rough implementation I made, better implementation should not contain them as JIT2 does not need them) I replaced the string printer for extended display mode to include the demand buffer information.

In pak64 it will produce something that looks like this...

10/1987-20/140
the format is...
current storage/in-transit + demand/maximum storage

In-transit is kept because it still is useful. In fact it is now more useful than ever since in an efficient network this will indicate the capacity of your network when shipping that good. The demand buffer should mostly be negative for properly linked and supplied industries as positive demand means there is a lack of supply. It only makes sense that demand is shown next to in transit as it represents the amount that it wants to add to the in-transit volume.

That said the display is rather rubbish. There really ought to be a table for this information rather than some cryptic string generated from a formatted string printer.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on September 12, 2014, 02:38:58 PM
Sounds like it's the translation, rather than the pak set that controls it.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 12, 2014, 03:10:52 PM
Actually it appears it is based on "in-transit percentage".

Code: [Select]
if(  welt->get_settings().get_factory_maximum_intransit_percentage()  ) {

So any game which has a maximum in-transit percentage should display it in extended form. I do admit that was not very smart of me seeing how it does not even use maximum in transit...
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on September 14, 2014, 03:44:16 PM
Ah that's it. I played with max_in_trasnit = 0. After changing it to non-zero I got it displayed.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 15, 2014, 03:41:08 PM
I am having a second attempt at this, trying to make it an applicable patch.

So far I am changing the Just In Time flag to be a number.
0 -> Classic - I believe this results in no limit on good production.
1 -> JIT Classic - Does what JIT does currently with very few (minor, you will probably not even notice) mechanical variations.
2 -> JIT2 - Flow based industry model with demand buffers. Also includes improved industry logic.

Now this is not easy for the following reasons.
1. Many new fields. Forces incrementing the game version number (sorry).
2. Improving the industry logic while keeping the old logic practically intact requires a lot of care.
3. The entire tick function for factories has to be rewritten. The old one just was not structured well enough.
4. Migrating to edge events instead of polling requires changes in logic in other places in the code.

That said it will add the following features to JIT2.
1. Power consumption based on actual production. An industry working at 15% will only use 15% of the power it does at 100%. In JIT Classic it uses 100% power unless production falls below 25% in which case power bonus vanishes completely and no power is used.
2. Consumption is based on actual production. If a factory only produces 1 good at 50% and another at 25% then it will work at 37.5%. In JIT Classic consumption was based on the largest production rate output (50% in the example).
3. Improved production precision. The extra precision field has been improved from 10 bits to 16 bits just because the other bits were being discarded for no apparent reason. This will affect both industry models.

I am still a long way away from having it working sadly.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on September 15, 2014, 04:49:16 PM
2. Consumption is based on actual production. If a factory only produces 1 good at 50% and another at 25% then it will work at 37.5%. In JIT Classic consumption was based on the largest production rate output (50% in the example).

Whether this is right or wrong depends on the actual industry. Oil refineries are typical examples of an industry where the current behaviour is the right one. For one unit of crude oil, you get some amount of petrol and some amout of diesel. If you reduce the production of diesel, but keep petrol production the same, you will still need the same amount of crude oil. There will just be more waste. On the other hand, a factory producing hammers and crow bars will reduce iron consumption if production of either one is scaled back. Pak64 might arguably have the mutual-byproduct type factories. I don't know about the others.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 15, 2014, 05:13:48 PM
Quote
Whether this is right or wrong depends on the actual industry. Oil refineries are typical examples of an industry where the current behaviour is the right one. For one unit of crude oil, you get some amount of petrol and some amout of diesel. If you reduce the production of diesel, but keep petrol production the same, you will still need the same amount of crude oil. There will just be more waste. On the other hand, a factory producing hammers and crow bars will reduce iron consumption if production of either one is scaled back. Pak64 might arguably have the mutual-byproduct type factories. I don't know about the others.
Not entirely true. Cracking (http://en.wikipedia.org/wiki/Cracking_(chemistry)) is used to break longer/heavier hydrocarbons down into smaller more useful ones. Without this most oil would be useless heavy fractions. Thanks to cracking they can efficiently convert these less useful heavy fractions into more useful lighter fractions such as jet fuel and diesel.

It also economically makes more sense than them throwing away large amounts of their consumption.

From a game balance point of view it will help ease the oil demands of pak64 oil refineries which have been notoriously hard to supply under high demand due to them consuming 9-16 thousand units of oil a month, the most of any industry. Since supplying them is not a problem with flow model it is needed to trim profits a bit. It also might not be the most realistic but from a gaming perspective it probably will make more sense to people.

"So I am pulling more out of the refinery and it is using more oil? Yeh that makes sense"

Ultimately a later revision should allow adjustable weights and other features. An example could be that printer ink only uses like 20% of the input oil while the rest is used for chemicals.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on September 15, 2014, 05:46:03 PM
I just picked oil refineries because it seemed simple and pak64 has one. As for cracking and similar, it usually only works one way. I also think people are familiar with the concept that you can't produce more of something just because there is less demand for a byproduct. The chemicals/ink thing in pak64 is indeed strange, but I don't think it's something that need a global solution. It also feels like dumbing down Simutrans, where the game adapts to the player, rather than challenging the player. Not that Simutrans is perfect, with uneven goods flow you were originally trying to solve as one of the greatest culprits. (Capacity issues due to scale being the other great problem I face. One simply can not get enough horse-drawn coaches on the road the meet the demand.)
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on September 15, 2014, 07:11:22 PM
May I suggest to keep JIT2 changes separate from other industry changes? Please focus on one thing to do it right. I would really like to see JIT2 accepted into simutrans, and the other changes are not necessary for that.

Now to the industrial models:
- there are some industries, that can change the byproduct into main product, and some that cannot. E.g. - an orchard can grow X tons of apples every year. And then the owner can decide how many of them to sell directly, and how many will be used to produce cider. On the other hand, if you have a sheep farm with X sheep, you can produce wool and milk in proportion, but if there is no demand for one of them, you cannot produce more of the other. So I think it should be a configurable option for the pak designer to decide what happens if there is less demand for some of the products - whether the excess production is thrown away, or somehow converted to other products. And then there is also a question how effective the coversion is, and if it reduces the consumption of inputs, or boosts the producion of demanded output...

- electricity: In standard we should assume that there is some invosible and omnipresent low voltage (230 VAC) network used by houses and industries for their basic needs. Some industries, like farms or shops in some paksets do not have any electricity boost. Heavy industry should need a separate high voltage line, but I do not like scaling the power consumption by production, because the power consumption defines the production boost - you might get into nasty feedback loop. And JIT2 will make it even worse, as deliveries are too driven by consumption - low power boost, low production, low deliveries - you will never get to 100% boost. It might be sensible to not use electricity if the factory can satisfy the demand without power boost. But I'm not sure if that is feasible.

Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 15, 2014, 11:03:23 PM
Quote
It also feels like dumbing down Simutrans, where the game adapts to the player, rather than challenging the player.
I actually think it would make the game harder. Before loading printer ink at a chemical plant to 100% usage would mean the chemical plant would consume maximum oil. Not only does printer ink make a lot of money but also you have maximum profit from supplying the plant.

Printer Ink £££
Crude Oil £££

With this change unless you also use chemicals to maximum you will transport less crude oil.

Printer Ink £££
Crude Oil ££

It also means that using chemicals will increase the amount of oil you need to transport potentially meaning you need to link more oil to it. Before they only added more profit from the chemicals themselves.
Quote
May I suggest to keep JIT2 changes separate from other industry changes? Please focus on one thing to do it right. I would really like to see JIT2 accepted into simutrans, and the other changes are not necessary for that.
They kind of are to avoid the code becoming a complete mess. As it was I was doing a lot of hacky stuff to order the correct amounts for power boosted industries purely because power boost is not applied in the case of an industry not producing. Additionally without scaling power demand you will find industries consuming excessive power as they demand 100% all the time (they never fall out the 75% cutoff for reasonably used industries). The scaling power demand is to solve this problem and potentially make connecting all industries with power a very viable approach (will need to see).

Quote
- there are some industries, that can change the byproduct into main product, and some that cannot. E.g. - an orchard can grow X tons of apples every year. And then the owner can decide how many of them to sell directly, and how many will be used to produce cider. On the other hand, if you have a sheep farm with X sheep, you can produce wool and milk in proportion, but if there is no demand for one of them, you cannot produce more of the other. So I think it should be a configurable option for the pak designer to decide what happens if there is less demand for some of the products - whether the excess production is thrown away, or somehow converted to other products. And then there is also a question how effective the coversion is, and if it reduces the consumption of inputs, or boosts the producion of demanded output...
This would need to be added later. Currently it treats output production the same as old factory mechanics except for middle industries which it will consume less basing on what is actually produced. Basically an idea of "work done" is defined based on the total product made as opposed to which product type was produced the most.

Quote
- electricity: In standard we should assume that there is some invosible and omnipresent low voltage (230 VAC) network used by houses and industries for their basic needs. Some industries, like farms or shops in some paksets do not have any electricity boost. Heavy industry should need a separate high voltage line, but I do not like scaling the power consumption by production, because the power consumption defines the production boost - you might get into nasty feedback loop. And JIT2 will make it even worse, as deliveries are too driven by consumption - low power boost, low production, low deliveries - you will never get to 100% boost. It might be sensible to not use electricity if the factory can satisfy the demand without power boost. But I'm not sure if that is feasible.
Feedback loop is only a problem if there is insufficient power in the network. In which case the network size (amount of power produced and consumed) will determine the stiffness. I do agree that in the case of a huge power hungry factory connected to a tiny green energy facility like a solar farm you could end up with a feedback loop where by over-ordering occurs as a result of the power boost to find that when production actually starts the boost drops so too much is ordered, over-fulls the input storage etc.

Will it eventually stabilize? Not sure but chances are quite high that it will. Will it be an issue? Probably not since if you have a massive 4Mw network and attach a 200kw factory to it then that will cause at most a variance of 5% so the maximum oscillation might be 5% which should easily be overwritten by input storage amounts. That is if the network is lacking power as otherwise there will be no oscillation at all.

In case people do not understand. The power demand is based on the amount of work the factory does however the power bonus is based on how well the demand was fulfilled. An idle factory will consume 0 w of power so in the presence of a transformer have maximum power boost even if the transformer is not connected. However as soon as it starts producing its power boost will fall to 0% as the transformer net has no power to give. If this becomes a problem due to the full power bonus when ordering empty and no power bonus when producing (input oscillation) then you should not have been silly and attached a transformer to the place that has no driving power. However this will always be a problem even in the previous version of it you may have tried since I also make that assumption to try and order the correct amounts.

An improvement obviously would be to analyse the attached power net detecting if it even has power and assuming a boost around the power usage percentage. A network at 200% load will obviously give at best a 50% production boost due to it only ever supplying half the power. However the problem with this is that there is no direct connection between factories and their transformer so it is impossible to get such data to factor in. As such you can either use how well demand was fulfilled or assume that just because it has a transformer and is consuming no power it will have maximum power boost. Ugly but will probably work quite well in the end as long as people create reasonable power networks with sufficient power sources attached.

The main aim of this is to try and allow connecting more factories to a power network. In some servers I was on people complained to me that I was connecting too many industries and that only important ones should be connected. On the other hand I had a 6Mw network (about 20 coal power stations and multiple oil and green facilities) so I needed to assure sufficient load to at least cover some of the cost. However due to the previous industry model I noticed huge oscillations where power usage would dip to 2Mw (30% odd) and then overflow to 9Mw (overload of 50%). With this improved industry model and old power mechanics such a network would almost certainly be well over 50% overload as virtually everything would constantly be producing. On the other hand with this adapting power demand the network might be closer to 6Mw as many factories and suppliers were being under-utilized and so I would not need to disconnect as much (it would stabilize at the average in the old model when industries stopped producing from time to time and not the maximum). Due to limited oscillations it also allows you to plan better in power scarce maps as you can estimate quite well if you have sufficient power capacity in a large network for some new industries.
Title: Re: Suggestion for JIT mechanics revision.
Post by: TurfIt on September 15, 2014, 11:57:41 PM
I fear you're heading down a path that ends in wasted effort and disappointment. IMO the existing patch has some undesirable implementation details; Expanding the scope of the patch now is likely to make it too difficult to extract the good. I didn't mention before as it's the algorithm that needs nailing down first...


I'd suggest limiting the demand to 50% of the input storage. With it set to 100%, I see long distance end consumers getting into oscillations of overflowing/running out/overflowing/... A number of factors contribute to cause this - supplier distance, production rates, size of input store, capacity of convois, etc., but limiting the size of the initial surge reduces the likely hood greatly. 75% would be fine too.


Yes except that is kind of the point. It will only run at 50% use as a penalty for you overflowing its storage

If a 50% rate when overfull is a penalty for overflowing, then let's just set the penalty rate to 0%; Same as JITv1. Now, the need for the 'lost order' mechanic is gone, and intermediate factory inputs can work independently.
This 'lost order' creating a demand coupling between the inputs is the deal breaker with JITv2. Downstream factories end up starved despite there being sufficient supply available, and there's nothing the player can do about it; That's the deal breaker. With JITv1, the player is in control. It might require some micro-management to have the factories produce constantly, but it's possible. JITv2 - nope, the algorithm is in control. Again many factors involved, including the consumption rate of the end consumers. As you pointed out, as long as the end consumers use a high percentage of the intermediate factories output, things work ok. But have end consumers demand only 30% the output, and they end up at a mere 20% instead due to stalls caused by this coupling. A completely non-intuitive result.


I think the 50% penalty rate was also seen a a solution to the too small storage at a far distance problem, but as I detailed in my last post, it really just doubles the distance before poor behaviour starts. IMHO the real fix is as Vladki points out - let the player expand the storage. The easiest appears to be using the attached station capacity. Convois can unload to the station instead of the factory, and the factory then pull from the station. There would be a performance implication, but I suspect it would be negligible given factories already scan their stations to distribute production. The bigger problem is fairly dividing the station capacity to prevent deadlocks. Previous efforts to implement a warehousing ability encountered the same problem. Some sort of GUI control would be needed for the player to manually do this; I never found an acceptable automatic algorithm.

Combining the above with a modified avoid_overcrowding option (make it apply just to goods - way too much micromanagement is needed for pax/mail) would allow warehouse stockpiles closer to the factory further solving the distance problem. I tried this before, and it does work nicely is some cases. The same GUI to manually divide the storage as above might make it work for the rest.



1. Power consumption based on actual production. An industry working at 15% will only use 15% of the power it does at 100%. In JIT Classic it uses 100% power unless production falls below 25% in which case power bonus vanishes completely and no power is used.

A factory produces, or it does not. There is no working at 15%.
By production falling below 25%, I assume you mean output store above 75%... Cutting the boost is intentional here. If the factory is filling up, then it doesn't need to produce at the boosted rate, hence cutting power frees it up for another factory that can make better use of it.


2. Consumption is based on actual production. If a factory only produces 1 good at 50% and another at 25% then it will work at 37.5%. In JIT Classic consumption was based on the largest production rate output (50% in the example).

The JITv1 logic is correct - the defined consumption ratios must be respected, and if the outputs are independent, then there should be two separate factories. However, IMO the second good should be produced at 50% as well, with the excess over output capacity discarded. i.e. The lowest output storage determines the base rate all the goods are produced at. I've seen cases where a factory is sufficiently supplied to produce enough to fully supply its consumers, yet it fails due to the 'waste' caused by producing at different rates. Really doesn't make any logical sense.


3. Improved production precision. The extra precision field has been improved from 10 bits to 16 bits just because the other bits were being discarded for no apparent reason. This will affect both industry models.

Are 6 more bits needed? i.e. Are anomalies present due to using only 10? I expect some calculations may need to move from 32bit to 64 which should be avoided if possible. Last I checked, there were a couple places running close to the 32bit limit.


The scaling power demand is to solve this problem and potentially make connecting all industries with power a very viable approach (will need to see).

At least for pak64, power is purposely short. If you want to not think, and have the game play itself, there's already settings to control this.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 16, 2014, 01:07:42 AM
Quote
I'd suggest limiting the demand to 50% of the input storage. With it set to 100%, I see long distance end consumers getting into oscillations of overflowing/running out/overflowing/... A number of factors contribute to cause this - supplier distance, production rates, size of input store, capacity of convois, etc., but limiting the size of the initial surge reduces the likely hood greatly. 75% would be fine too.
I am not understanding what you are asking. Currently demand is 100% of the desired consumption rate (which is based on actual work done). If storage overflows it gets reduced to 50% until it drops below the storage amount (this basically burns off excess in the line). Line length does not mater as a reliable flow of goods should arrive as fast as they are consumed. The only time oscillation will happen is if boosts are not reliable such as from power or passengers/mail as those may cause an increase/decrease in production such that the buffer amount cannot cope with the backlogged flow amount. However this adds difficulty to the game as now you also need to make sure boosts are reasonably stable (which they usually are if done properly). In worst case you are still looking at >50% production utilization.

Quote
If a 50% rate when overfull is a penalty for overflowing, then let's just set the penalty rate to 0%; Same as JITv1. Now, the need for the 'lost order' mechanic is gone, and intermediate factory inputs can work independently.
This 'lost order' creating a demand coupling between the inputs is the deal breaker with JITv2. Downstream factories end up starved despite there being sufficient supply available, and there's nothing the player can do about it; That's the deal breaker. With JITv1, the player is in control. It might require some micro-management to have the factories produce constantly, but it's possible. JITv2 - nope, the algorithm is in control. Again many factors involved, including the consumption rate of the end consumers. As you pointed out, as long as the end consumers use a high percentage of the intermediate factories output, things work ok. But have end consumers demand only 30% the output, and they end up at a mere 20% instead due to stalls caused by this coupling. A completely non-intuitive result.
The coupling is there to stop over-ordering a specific product. With your suggestion what will happen is it falls back to JIT1 problem where by it will go in bursts if you ever overflow. If 1 input is under-supplied then the other two will overflow (resulting in a collapse of the supply chain) which will take a while to empty (due to the one being under-supplied) resulting in it eventually falling below maximum storage (restarting supply) to run out of storage before the supply chain restarts so now you have your under supplied product in storage but are lacking the other two products to continue. During this time the under supplied product could overflow resulting in lost orders of the under supplied product.

The coupling will not cause stalls because it orders exactly what it needs. If a factory takes 1 coal and 3 iron ore to make a steel then it will order exactly 3 iron ore for every 1 coal. If one of them is under supplied (too few iron ore pits connected) then it will throttle back coal automatically so it is ordered at the rate which iron ore is. The industry will never stall because of a lack of coal since it will still receive coal shipments sufficiently (just slower to match the rate of iron ore). If this is not the case then there must be some error introduced by the fixed point mathematics (at most a few fractional units out). Some oscillation might occur initially as it will over-order until the correct production rate is reached.

Quote
A factory produces, or it does not. There is no working at 15%.
If this was the case things would be so much easier! Everything becomes simple pulse-width modulation behaviour of producing/idle and ordering/idle. However alas there is a production rate reduction applied to outputs. If an output has more than 11.9990234375 units then it will suffer from a diminishing production rate based upon current storage over maximum storage. This was not my idea (it has been in simutrans for ages) and even surprised me when applying the first implementation. I could easily remove this.

However it will not solve the spike loads on power networks issue. There is still a chance all industries connected may produce all at once causing an overload just to stop producing immediately after and the network being hardly used.

Quote
By production falling below 25%, I assume you mean output store above 75%... Cutting the boost is intentional here. If the factory is filling up, then it doesn't need to produce at the boosted rate, hence cutting power frees it up for another factory that can make better use of it.
No I literally mean production falling below 25% based on what I said above. Outputs suffer from diminishing production rate based on the current storage over the maximum storage. When it is 75% full it will only be producing at 25% unless the output storage maximum is less than 11.9990234375 units which is the cut-on for diminishing returns. It only makes sense that power follows this curve to some degree since otherwise at high output percentages you are paying up to 4 times the power for the same amount of production.

Again, removing this diminishing curve would fix some issues but still cause unmanageable spike loads on your power network followed by idle times. Smoothing it with the diminished rate will average power usage allowing networks to be more utilized while still providing reliable power.

Quote
The JITv1 logic is correct - the defined consumption ratios must be respected, and if the outputs are independent, then there should be two separate factories. However, IMO the second good should be produced at 50% as well, with the excess over output capacity discarded. i.e. The lowest output storage determines the base rate all the goods are produced at. I've seen cases where a factory is sufficiently supplied to produce enough to fully supply its consumers, yet it fails due to the 'waste' caused by producing at different rates. Really doesn't make any logical sense.
Both outputs are treated separately and produce at the given output percentages. Imagine a factory producing 100 units of production per month. With output A having a 100% multiplier and output B a 200% multiplier then it will produce at most 100 units of A and 200 units of B per month (it will make both in parallel). You cannot pull out 201 units per month of B even if you use 0 units of A because it still is limited to 200 units of B per month. Currently the raw material consumption was entirely based on the output with the most utilization so using 100 A and 200 B per month used the same raw materials as just 100 A or 200 B (equivalently 200 B or 100 A being discarded).

The change I propose is that the amount of raw material is based purely on the amount produced. The factory will still produce the same maximums of A and B however the only difference is that if one is not fully used the amount of raw materials consumed will decrease. You can still load the factory to 100% by maxing out both outputs and it will still consume 100% of maximum possible raw materials however unless you max out both A and B products it will consume less than 100% of maximum raw materials even if either A or B product is maxed.

Few factories in Simutrans produce multiple products from raw materials. This is aimed mostly at ones like the pak64 oil refinery which has a massive 300% crude oil consumption rate and a massive production of well over 6,000 units per month when well supplied (18,000 units of crude oil). Before running such a facility at maximum efficiency was very difficult as gasoline and plastic or printer ink and chemicals were hard to fully consume or distribute efficiently and 16,000 units of oil with a storage of only 600 odd was near impossible to keep constantly supplied. However with JIT2 flows one could easily max out gasoline or plastic and then supply it fully with crude oil. The result would be a huge amount of money even though half the products are being un-used. With this change such industries will use less raw materials unless both outputs are maxed out (so the 16,000 units of crude oil will be considerably less until you use chemicals fully as well). If such a mechanic is not intended then I suggest making separate plants for printer ink, chemicals etc.

Quote
Are 6 more bits needed? i.e. Are anomalies present due to using only 10? I expect some calculations may need to move from 32bit to 64 which should be avoided if possible. Last I checked, there were a couple places running close to the 32bit limit.
They were just being discarded anyway. Also the remainder is only using 10 bits of a 32 bit value. As such those extra 6 bits could go there with practically no overhead (maybe even less overhead as its fewer operations). The actual computation is already in 64 bits as a q26 (8 base production rate, 8 production factor and 10 delta time) which gives a theoretical production of 274877906943 million entire units of product per tick, which will overflow the 2097151 unit maximum storage easily. Adding the extra 6 bits of precision to the calculation should make very little difference to anything. Only possible thing against it is that it could possible slow the multiplication operation down slightly depending on how it is implemented in hardware (functional unit multipliers can have dynamic cycle run lengths based on the physical length of numbers involved) however I doubt that will make much of a difference seeing how it is already a simulated 64 bit operation.

I am guessing those 6 bits were probably lost before there was a remainder value or before it migrated to a 64bit intermediate.

Quote
At least for pak64, power is purposely short. If you want to not think, and have the game play itself, there's already settings to control this.
The problem with pak64 is that power has not been set up on several industries or is set to silly values. Coal mines consume almost the entire production of a coal power station and are not even boosted enough to supply the coal power station. Timber Plantations (forests basically) consume a load of power for not even getting a boost (clearly they are meant to consume no power like pak128). However pak64 having poor power balance is not an issue.

The issue I am talking about only occurs in very large networks where you have 20+ power stations connected together power 50+ industries. Currently at times you will be below 50% utilization of your power network while at other times you will be overloaded because at some stages industries stop production together or start production together. This is made worse by middle industries consuming up to 4 times the power for the same production. Since everything is becoming flow based it only makes sense to make power flow based. The result is that the utilization of your power network becomes steady based on the utilization of all industries.

That steel mill that is producing barely around the 25% cut-off for power mark due to flows currently consumes the same power as that steel mill that has constantly empty output storage producing at 100%. With variable power consumption that 25% used steel mill will now steadily use 25% of the power reducing its power consumption by 75% while keeping the power per product the same as the fully utilized factory.

Another example is that oil refinery that is barely used. Instead of consuming full power for long periods just to produce 10% output product and then stop consuming resulting in unreliable power consumption. Instead it will consume just around 10% of the power reliably.

Together the result should be a power network that has very few power fluctuations. Because power fluctuates less and is used more efficiently you will be able to reliably connect more industries to the same network. However since this is not necessarily a good thing from a game perspective (unlimited power) this is easily solved by making power more scarce and lowering the power fraction in the game settings.
Title: Re: Suggestion for JIT mechanics revision.
Post by: ZéQuimTó on September 16, 2014, 02:17:14 AM
Sorry for my ignorance, but excuse me if i am wrong:

Couldn't your credit system be implemented the following way?
 - An industry chain is kept connected (i.e. "source" industries keep producing towards destination) while the "sink"'s station is not overcrowded.

This way, a user could just increase the capacity of the given station.... not only seems easier to implement, as well more "realistic".
By realistic, I mean: To connect two industries far apart, the destination industry should have a big storage capacity to keep working during the transit time...

Doesn't this solve the problem? Longer distance -> larger station capacity.

Am I missing something?

PS: DrSuperGood, are you from electronics? It seems so because you started by using the water bucket analogy (common in describing capacitors/integrators), then you mention feedback, and "problem stiffness"...just gessing :D
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 16, 2014, 03:22:53 AM
Quote
Am I missing something?
The problem that JIT2 is targeting is not so much the reliability of factories working (the bucket having water) but more the problem of the bursty in-transit amounts (the water and pipes going to the bucket).

Currently your source stops end up over-crowded with goods quickly after an industry starts demanding. These then are shipped in parallel from all suppliers towards the destination. If they require transfers then chances are the transfers will start to fill up and possibly overcrowd. The huge flow of goods will continue filling everything up until enough reaches the demanding station that its input storage is at least full or that the maximum in-transit limit is reached. In the case of a coal power station linked to 15 coal mines you could end up with its current storage reaching the maximum of 15,000 and coal literally being wasted (which you get paid for... cheating...). After all this all your coal convoys sit idle until another source asks to be drowned in coal. What is bad is that despite all this coal reaching the power station it might have run out of storage before the first shipment arrived however that is not the is only a little issue (as you said, bigger storages solve that).

However the main issue is that your network has just been flooded with over 15,000 units of coal just to supply a tiny coal power station that only uses less than 1k per month. If you look at in-transit graph of the coal power station you see huge periodic bursts where it reach insane numbers before falling back to reasonable quantities. During this this burst your network was pushed to its limit hauling coal, overfilling transfers, creating convoy traffic crowding junctions.

Now imagine you have an oil power station which is connected to 10 oil rigs. The same thing occurs with overfilling it with oil, creating huge amounts of convoy traffic and overcrowding transfers. What happens if both these periods happen to coincide together? Your transfers literally explodes with insane amounts of goods in it. Worse is all the convoy traffic is so much now that nothing is working efficiently and you are even struggling to ship both coal and oil to their end destinations. Eventually the madness stops when maximum in-transit limit is reached leaving your transfer with an insane amount of oil and coal to dissipate over the following months which is mostly wasted due to overflowing the current storage. The really stupid thing about all this is that none of this should happen as you have already over-engineered your network trying to cope with these bursts.

Currently the only solution to avoid over-engineering is to manually fine tune convoys so they are now moving almost 100% of the time and the target industry is being used almost 100% of the time. This is done by installing a bottleneck somewhere and using that to limit flow rate (either supplier pickup stop holding limit or maximum in-transit limit at a transfer). However this is very time consuming and impossible to scale up. Changes in traffic, convoys, need for multiple suppliers, etc all are a real pain to do.

The end result is that mostly people over-engineer their networks, using like 20 oil tankers to move the huge flows from oil rings, 4 parked trains in a station to get the oil from the coast a transfer and 8 dedicated trains at the transfer to move the oil to the power station all with the aim that you will cope with all the oil and get it efficiently to the consumer. The bigger the network the more trains you need and more storage capacity at transfers for them to not overcrowd (imagine crowding restrictions like no-route, they would be a real pain). This all costs money and you end up with trains sitting around, a huge tangle of train lines that still suffers traffic problems and massive stations the size of cities.

However as the flow approach showed, you actually do not need a lot of convoys, only enough to get as much as the consumer needs. You also do not need a lot in-transit, only enough to supply the convoys that are moving and some at transfers. Why must the game constantly flood you with goods from everywhere burying you in them?

This is where JIT2 comes in. Consumers no longer order everything they can, instead only what they need. This is done continuously meaning that goods slowly enter your network at a constant rate. This means that you are not buried under a nearly infinite tsunami of goods being thrown at you. Instead you get a nice continuous and easy to manage stream of goods. Traffic at your transfers will remain constant with convoys coming and going continuously (no big bursts of convoys). The amount in your transfers will remain small since as fast as they pour in, they are being poured out towards the consumer. The consumer is working efficiently as well since it never is without goods as they are replaced as fast as it consumes them. True Just in Time ordering!

The end result is those 20 oil tanker boats become 4, your oil dock only has 1 platform with 1 train. The transfer only has 1 train to the oil power station. The in-transit graph now shows a almost completely flat line hovering around 4,000 units, tiny compared with the >15,000 it was dealing with before. You save a huge amount of money on maintenance, have a much smaller transfer, have not covered half the map in rail lines and have only a fraction of the convoys and the oil power station is now working better than ever before. From the prototype you can see that it makes freight hauling a lot easier, more intuitive and run better. When applied to some pak64 server games in-transit amounts showed a considerable decrease in non-bottlenecked industries, and some less well supplied industries (the infamous mullekraftwerk) began to work considerably better as they had more supply available for them.

Quote
PS: DrSuperGood, are you from electronics? It seems so because you started by using the water bucket analogy (common in describing capacitors/integrators), then you mention feedback, and "problem stiffness"...just gessing
Kind of. Did about 50:50 with computer science.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on September 16, 2014, 09:02:22 PM
Turfit:
Quote
I'd suggest limiting the demand to 50% of the input storage.

Do you mean limiting the demand counter max to be 50% of input storage? If so, I think the JIT2 will do quite well with any value. Maybe it could be configurabe so that we can experiment with it. But during the game you'll find that the demand counter (shown i facotry info) is either jumping around zero (somewhere betweeen +10 and -10) for well supplied industries, or it is slowly growing towards max for undersupplied industries. Setting the max to be equal to input storage is meant to allow non producing factory to order full stock immediately. In my original proposal I wanted to send goods to the factory with highest demand counter, so that all of them get supplied in proportion to their production rate. However this is not implemented (and now I think it will provide only cosmetic improvements) and I think JIT2 will work just fine if the demand counter max is set to 1. If it is 1 - factory is accepting goods, when goods are sent, couter goes below zero, and as the factory demands more the counter will once again reach 1 and more goods are sent. No real need to go much higher. Ahh, I almost forgot - there IS one situation when the counter is needed to be more than 1. Consider long distance link with a few high capacity convoys but a low station capacity at supplier. The station will be overcrowded most of the time and the suppliers output will be (half-)full when the convoy arrives. Then it will load the cargo from station and also factory output store, but only according to the demand counter. If the demand counter max is set too low - it will slow down the loading, and starve the consumer. But this can be solved by increasing the source station capacity so that it does not overflow.

Next about the input overflow penalty - I think that it could be configurable too, and anything between 0 and 100% might work. Just think of it as a difficulty setting. Higher values mean smoother supply even if the factory's input is overcrowded, lower values might cause starving the factory, and force the player to use more smaller convoys instead of one big.

ZéQuimTó:
Quote
An industry chain is kept connected (i.e. "source" industries keep producing towards destination) while the "sink"'s station is not overcrowded. This way, a user could just increase the capacity of the given station.... not only seems easier to implement, as well more "realistic".
Yeah, that's what I suggested, but at the moment unloaded goods go directly to the factory, not to the station. Another patch would be needed for that. And I think it should be separate form JIT2. It will work fine with either old JIT or JIT2.

Factories with multiple outputs:
- please leave them as they are. It has nothing to do with JIT2. If you want a change in that, please make it a separate patch, preferably leaving the decision to the pak author. Option being - unwanted (by-)products are wasted (current behavior), or that the input consumption is reduced if some products are not being produced.

Factories with multiple inputs:
- binding the input demands as implemented now is good. If anybody has a demo game where it does harm, show it.

Power boost.
- I did not know that the factory production is gradually decreasing when output storage is getting full. In that case I agree with proportional reduction of power demand. And with complete power off at some level. But there will still be situations when the factory will do "pulse modulation": let's have a factory which has more supply than it can use without power, but less than it can use with power boost. When it gets some goods, it will "turn on" and use 100% power, quickly use all the goods and turn off and wait for another delivery.
- In real world, overloading the power grid is really bad, and can cause complete black-out. I think there should be some sort of warning (like overcrowded station), if the potential simultaneous consumption of all industires is higher than the supply. Or a warning when real overload happens, posibly with blackout like punishment - turn off everything for a while.
Title: Re: Suggestion for JIT mechanics revision.
Post by: ZéQuimTó on September 16, 2014, 09:39:18 PM
DrSuperGood, thanks for your explanation! Now I can fully appreciate your contribution :D

Maybe its too late now for suggesting alternatives, but wouldn't allowing the player to manually reduce production rate (select the production rate, where the cap would be the current maximum production rate) also solve that problem?
Instead of making simutrans doing all the work for us, this would require the player to identify that problem, and tune the factories to the right amount... don't know if this copes with simutrans philosophy...

I peeked at factory's code, and just saw too much german to try an one-liner :D
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on September 16, 2014, 09:54:31 PM
Some comments:

I think there are people that worry about overcrowding and want to have it and other who do not want it (and a third who do not care). Hence that part of JIT2 needs to be switchable, and will be warmly welcomed imho.

The other things (power consumption, separate consumption, more precisions, ... ) will likely go unnoticed, as they will affect only fue factories.

About power balance: The calculation is that only a certain minimum of power (the power promille) is satisfied. As such it is fully intended that pak64 coal mines consumer nearly all power. If the power consumption is lowered (i.e. most factories will never work at 100% and hence also never consumer 100%), that value need to be also weighted (maybe assuming mean consumption of 50% max. value).

Anyway, the electricity was never much balanced beyond testgames. Also I am not sure why the forest consumers power without boosting. I though this should have been connected. I would rather suspect there is something in the dat file missing.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 17, 2014, 12:06:48 AM
Quote
- please leave them as they are. It has nothing to do with JIT2. If you want a change in that, please make it a separate patch, preferably leaving the decision to the pak author. Option being - unwanted (by-)products are wasted (current behavior), or that the input consumption is reduced if some products are not being produced.
I will leave it on for the first test once it is done. It is easy enough to remove anyway (just remove some lines of code).
Quote
Maybe its too late now for suggesting alternatives, but wouldn't allowing the player to manually reduce production rate (select the production rate, where the cap would be the current maximum production rate) also solve that problem?
Who controls it however? (think multiplayer) Also do you really need to control it? If the game controls it automatically for you and gets it right there really is no need to control it.

What would stop people over-ordering? How to define goods flows? What about multiple sources? Demand counters that automatically fill are simple to implement and from earlier tests appear to work surprisingly well.

Quote
Instead of making simutrans doing all the work for us, this would require the player to identify that problem, and tune the factories to the right amount... don't know if this copes with simutrans philosophy...
Except Simutrans is not a factory management game, it is a transportation management game. You do not order people to take your busses to a certain place. You also do not order factories to buy goods. I do agree that factory management could eventually be added into the game where you define the production rate and have to fine appropriate sources etc however for now it is all about shipping goods and the aim of JIT2 is to make sure factories do not order goods for you to ship in a stupid and impractical way.

Quote
I peeked at factory's code, and just saw too much german to try an one-liner
The important part is mostly the tick method. Although variables are named in German you can still get what happens from the logic. That is if you have a good understanding of fixed point mathematics as the factory code depends heavily on it (menge (German for amount), the amount in an output/input is a q10 number as an example).

Quote
Hence that part of JIT2 needs to be switchable, and will be warmly welcomed imho.
I am designing the implementation around this. I have already converted JIT to a integer instead of boolean where 0 is the classic, 1 is JIT Classic and 2 is JIT2.

Quote
will likely go unnoticed, as they will affect only fue factories.
Most factories in pak128 and all in pak64 use power. Only some have power boosts. In smaller games or when players do not care about power it will go unnoticed however in games where players do care about power it will be a welcome convenience and make power management more easy.

Quote
About power balance: The calculation is that only a certain minimum of power (the power promille) is satisfied. As such it is fully intended that pak64 coal mines consumer nearly all power. If the power consumption is lowered (i.e. most factories will never work at 100% and hence also never consumer 100%), that value need to be also weighted (maybe assuming mean consumption of 50% max. value).
The problem with pak64 coal mines is not the power consumption, but the boost they get from full power bonus is less than the amount of coal burned in a coal power station to generate the power. This is a sort of illogical/impractical physics thing. Consuming that much power for a 50-100% boost would be perfectly fine as that is a ton of coal extra but they only get a 10% boost (this usually results in less coal than is used by the coal power station to power the mine). Now that they will use less than full power unless specifically fully used it will also solve the problem largely (since coal mines in pak64 seldom run at 100% utilization).

The changes to power is that it will base the power production bonus on the satisfaction of the demand (the sum of the remaining demand and the power supplied) rather than the base power demand (if working 100%). This means that as long as the factory is connected to a network less than 100% loaded it will receive a 100% boost even if it is drawing less than full power because it is not working at full capacity (it draws less power but that power is fully satisfied so 100% boost). This does result in a silly case where a factory connected to a standing transformer (no network attached) will appear to have 100% power boost until it tries to produce something which will cause it to drop immediately to 0%, however the amount of bonus production you could win from this is minimal and offset by the loss of production to setup (make the factory idle) so can be as good as ignored. A better solution could be a power demand buffer which remembers failed power demand and applies a penalty accordingly on the next tick to stop the previously mentioned fault (a tick with higher power boost than the factory got power for) however this is grabbing at fractional units.

Quote
Also I am not sure why the forest consumers power without boosting. I though this should have been connected. I would rather suspect there is something in the dat file missing.
I do not think forests should consume any power at all, especially the amounts they do. I do not know of any forests in real life that use artificial lighting to make trees for timber grow faster. Forest machinery also is usually fuel powered and clearly no wood processing is done (saw mills do that).
Title: Re: Suggestion for JIT mechanics revision.
Post by: ZéQuimTó on September 17, 2014, 03:59:57 AM
Who controls it however? (think multiplayer) Also do you really need to control it?
Good point. In that case, JIT2 imho is the way to go. Can't even see another way around.


Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on September 17, 2014, 04:57:42 AM
Sorry for my ignorance, but excuse me if i am wrong:

Couldn't your credit system be implemented the following way?
 - An industry chain is kept connected (i.e. "source" industries keep producing towards destination) while the "sink"'s station is not overcrowded.

This way, a user could just increase the capacity of the given station.... not only seems easier to implement, as well more "realistic".
By realistic, I mean: To connect two industries far apart, the destination industry should have a big storage capacity to keep working during the transit time...

Doesn't this solve the problem? Longer distance -> larger station capacity.

Am I missing something?

It might be realistic for the destination industry to have bigger input storage the further away the suppliers are. On the other hand, there has been a trend for industries to minimize their input storage. Just-in-time is a real world concept, not something Simutrans has come up with. I believe the main incentive is to save money on storage capacity, and to avoid ending up with huge amounts of raw material if production stops due to a drop pause in demand for the final product.

Furthermore, it's the consumer that primarily drives production of it's supplies, not the producer that keeps pushing. Realistically. (That the destination station doesn't get crowded has already been pointed out.)

And finally, I don't have the impression that it's the carriers reponsability to maintain a storage site for the destination customer. Not without getting paid for it.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on September 17, 2014, 08:18:46 PM
I think there are people that worry about overcrowding and want to have it and other who do not want it (and a third who do not care). Hence that part of JIT2 needs to be switchable, and will be warmly welcomed imho.

I'm not sure which overcrowding you have in mind - overcrowding facory input, or overcrowded supplier station ?

If you are talking about factory input, then adjustable overcowding penalty should solve it. Now the factory will order at 50% reduced rate if overcrowded, however if you don't care you might prefer it ordering at the same rate as if not overcrowded. On the other hand if you want the game to be harder, you might prefer stopping orders altogether until the extra cargo is consumed.

If you had station overcrowding in mind - then jit2 works just nicely - source station is overcrowded only if the destination is willing to accept more goods than you deliver.

Ters:
Quote
Furthermore, it's the consumer that primarily drives production of it's supplies, not the producer that keeps pushing. Realistically. (That the destination station doesn't get crowded has already been pointed out.)
Yeah, and that is exactly what JIT2 is about.
Title: Re: Suggestion for JIT mechanics revision.
Post by: ZéQuimTó on September 18, 2014, 12:52:47 AM
It might be realistic for the destination industry to have bigger input storage the further away the suppliers are. On the other hand, there has been a trend for industries to minimize their input storage. Just-in-time is a real world concept, not something Simutrans has come up with. I believe the main incentive is to save money on storage capacity, and to avoid ending up with huge amounts of raw material if production stops due to a drop pause in demand for the final product.

Furthermore, it's the consumer that primarily drives production of it's supplies, not the producer that keeps pushing. Realistically. (That the destination station doesn't get crowded has already been pointed out.)

And finally, I don't have the impression that it's the carriers reponsability to maintain a storage site for the destination customer. Not without getting paid for it.

I agree with all your points (I have already converted myself into a JIT2 apologist) .
Regarding the last point, I do understand your point, but I am not sure how it translates into simutrans. Let me get you and example, in real life the transportation company presents a bill to the consumer. However in simutrans the opposite happens (the consumer pays as much as the distance traveled + speed bonus)...As so, It transfers to us the responsability to make the network as efficient as possible. As so, If in real life a company would find more efficient to just pay for a few transfers from time to time, and pay the rental of warehouses to store surplus, in simutrans I guess that would "translate" into "we pay you exactly the same. If you want to earn more money, make your network more efficient.". And that could imply using extra warehouses..

The bottom line is (TLDR):
I really think it should be possible to add extra input capacity, just by increasing stop capacity. It is an intuitive behaviour (in the sense that a player would simply add more storage to a consumer if it is getting overcrowded), yet, does not work.

BTW (warning, offtopic). I confess I did not understand simutrans devel. policy: Do you use scattered patches in *zip format, or actual git pull-requests?


Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 18, 2014, 02:28:07 AM
Quote
Regarding the last point, I do understand your point, but I am not sure how it translates into simutrans. Let me get you and example, in real life the transportation company presents a bill to the consumer. However in simutrans the opposite happens (the consumer pays as much as the distance traveled + speed bonus)...As so, It transfers to us the responsability to make the network as efficient as possible. As so, If in real life a company would find more efficient to just pay for a few transfers from time to time, and pay the rental of warehouses to store surplus, in simutrans I guess that would "translate" into "we pay you exactly the same. If you want to earn more money, make your network more efficient.". And that could imply using extra warehouses..
The current pay system in simutrans itself is kind of messed up. I was trying to balance pak64 aircraft earlier (was planning to do pak128 if results looked good) and all I ended up with was a mess of ok convoys when released but losing money by the time they obsolete.

The problem? Speed bonuses are very inflexible when the only figure you have to control is the cost per tile of a convoy. To fight this the only solution is to make running costs for the future resulting in crazy profit when first available. The RvG R-1049 Super Constellation as an example makes a massive 0.18 profit per unit at full load when first available which considering passengers default is 0.15 makes them pretty insane (they pay off in 5.4 hours fully used, less than a month).

What I wanted as a test was to standardize all passenger aircraft to make 0.06 profit per unit at 100% utilization with a utilization of about 70% odd with a minimum pay-off time of 12 hours (pak64 is meant to be easy). As the aircraft approaches obsolescence it should decrease towards 0 profit and 100% utilization to force people to migrate. However after messing with numbers this seems pretty much impossible to do as running costs are not flexible enough. Convoys needs other cost parameters (eg cost/bonus per unit moved or end of life cost per tile) to balance them without resorting to silly speed bonus speed figures however that is something for another time.

Quote
Do you use scattered patches in *zip format, or actual git pull-requests?
I believe experimental uses GIT pull requests while standard uses SVN patch files.
Title: Re: Suggestion for JIT mechanics revision.
Post by: ZéQuimTó on September 18, 2014, 11:30:31 PM
Convoys needs other cost parameters (eg cost/bonus per unit moved or end of life cost per tile) to balance them without resorting to silly speed bonus speed figures however that is something for another time.

Just a silly idea (but with the same result): what about making convoys lose some cargo as their age increases? Something like a "lose cargo" probability on each tile travelled. Its something like an old rusty train dropping stuff on its way, or a leaking tank. Such probability could be some sort of nonlinear function based on convoy's age, since a brand-new train's car would need to get a little bit aged before starting to lose stuff.

The point is: it does not make much sense for consumers to pay less just because the cargo was shipped on a old train. Yet, if less cargo arrives, its a different story.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 19, 2014, 02:05:22 AM
Quote
Just a silly idea (but with the same result): what about making convoys lose some cargo as their age increases? Something like a "lose cargo" probability on each tile travelled. Its something like an old rusty train dropping stuff on its way, or a leaking tank. Such probability could be some sort of nonlinear function based on convoy's age, since a brand-new train's car would need to get a little bit aged before starting to lose stuff.

The point is: it does not make much sense for consumers to pay less just because the cargo was shipped on a old train. Yet, if less cargo arrives, its a different story.
Except that would make the problem even worse.

The problem only applies in the following situation.
1. The convoy carries a type of goods that benefits a lot from speed bonus.
2. The convoy has an extended life cycle, being available for a considerable number of years during which speed bonus changes considerably.
3. The convoy has a high utilization threshold. It needs to be mostly full to break even with running costs. This is what pak64 uses to trim profit for everything since it has low maintenance costs and what pak128 aircraft need.
4. The desired convoy profit is a small fraction of the goods value. Eg 0.06 per unit for a good with base 0.15 per unit.

As an example. An aircraft available in pak64 is available for 40 years. During this time speed bonus increases from 213.1578947 km/h to 930 km/h. What one would like is for it to be released making 0.06 ppu (profit per unit) at full utilization. By the end of its life it should be making 0.01 ppu (or even 0.00 ppu), certainly a time for the player to consider upgrading. Currently this is not possible as if you want it to make profit by the end of life then you need to make starting ppu too high while if you want correct starting ppu then end of life profit is a huge loss.

One current solution could be to release multiple versions of the same aircraft as time progresses but without some kind of auto replacement UI this would require too much micro management.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on September 19, 2014, 05:44:40 AM
The funny thing is that there are DC-3s still in service making profits. Unlike in Simutrans, modern jets with greater speed, and likely better comforts, have problems outcompeting them, even on passenger routes. There is a factor, or maybe several, that can keep a vehicle economically viable, beyond what Simutrans models through speed bonus.
Title: Re: Suggestion for JIT mechanics revision.
Post by: sdog on September 20, 2014, 01:16:34 AM
Not entirely correct. Power stations are seldom 1 reactor, often with several. Power output can be adjusted by varying the number of reactors online at a given time. In the case of a coal powerstation this would mean shutting off many of the fires/boilers resulting in less steam being generated.

If you are talking reality, a power station not connected to a load cannot even generate electricity as it would blow the generators apart. Economically without generating electricity there is no reason to even have the power station consuming anything at all. Thus my suggesting making sense.  That said, there could be production tiers with power stations (20%, 40%, etc) so that 100% utilization is not required, only enough to reach the 100% tier.
Experimental has a good solution for that, with cities being consumers as well. If the city closest to the power station would be automatically connected, it would serve as a sink, and keep the power station running with reduced output power.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 21, 2014, 01:19:27 AM
Trying to re-write and stream line the industry code is not straight forward. Specifically with all the production rate conversions for inputs and outputs. Each adds error often requiring bounds checking etc making things quite complicated.

However yesterday I was having a major think about the subject as I was starting to get stuck with the code and came up with a revolutionary idea. Instead of storing inputs and outputs as units of product (well they are q10 fraction units of product), you normalize them into factory units. This means the factory tick code does not care about the production factor of the various inputs and outputs, it is entirely unaware of them as they make no difference. Instead when adding or removing product (entry and exit of the system) you apply the production factor. Obviously graphing also needs the production factor applied to it however that is not so important if error occurs.

This makes the factory code super simple. Once you calculated the amount to consume in a middle factory you can literally iterate through all inputs and remove the same amount. Error is limited only to entry and exiting the system which occurs considerably less often and can probably be limited. Finding the limiting input is super simple as well, just a min of the storage amounts.

This is something I will certainly look into.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on September 21, 2014, 07:44:22 AM
One could go one step futher and say that no dependent code should care whether input and ouput amounts are stored as logistical units or factory units. It should be abstracted away into some class that provides the appropriate functions to do the basic operations needed. That way, there will also be less risk of scattering conversion formulas all over the place.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on September 21, 2014, 08:51:56 PM
It would still need some precision but otherwise this sounds clever.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on September 22, 2014, 12:18:35 AM
Another little trick I was thinking about would greatly improve the efficiency of computing the minimum production at a middle factory (input limited).

Currently it needs to find the minimum normalized input which is less than the intended production on every tick in case there is insufficient input for full production. The above suggested normalization would already greatly improve the efficiency of this as no normalization with production factor would occur. However a minimum search must still be performed which is O(n) and at least requires several memory lookups.

However since all inputs are consumed synchronously by the same normalized amount, once you find that minimum normalized input (limiting input) it will remain constant until new shipments arrive. This means that instead of computing minimum every tick you can compute minimum on shipment/load and cache the results in a uint8 (larger needed?). In theory ticks should occur orders of magnitude more often than shipments as such this would be a minor saving at the cost of a few bytes more memory.

Another saving idea is to keep track of "active" inputs, outputs and demand buffers. If all inputs are empty then there is no need to do anything directly related with consumption as there is clearly nothing to consume. Like wise if all outputs are full then there is no need to do anything with production as there is no room to produce anything more. Although minor this could add up to some savings for industries that are not in use, which can often be a considerable number.

All these result cache variables can be computed at events (receive, send etc) instead of every tick. They can also be computed when loading meaning no extra storage is necessary for them. The memory increase is insignificant as the number of fabrik_t objects is usually much less than 1,000 for most purposes. It also might read better.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on October 01, 2014, 02:35:28 PM
I am slowly getting close to a better implementation. I have totally rewritten industry tick and just have a few bugs to sort out as well as testing for classic support.

Changes include...
Input/output activity caching, so that complex input/output calculations are not performed every tick when not required.
Dynamic power consumption based on amount produced (currently buggy...)
All inputs and outputs are now normalized with conversion only occurring when goods are sent or received (currently graphs are also normalized so will show incorrect results for some industries, still working on a solution).
All ordering is now done in ticks. This is not giving me satisfactory results ATM because of unbalanced ordering from factories so I might change it to every delta t period (as that assures all factories have a chance to order).
With JIT2 it is no longer possible to exceed factory input storage. Doing so will result in wasted production with the amount of overflow being decremented from the demand buffer.
It will not alter the mechanics of existing games (well it does but you will not notice it the changes are so small). It is also not recommended to change the setting of an existing game due to it needing extra fields to be saved and the use of unions to prevent excess memory allocation.

EDIT:
Here is hopefully a working patch with JIT2. It modifies the version of the game for save support (might need correction there). It will not apply to existing saves, you will need to start a new game and during setup in settings choose just_in_time to be the value 2.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on October 03, 2014, 01:19:22 PM
As much as I hate to triple post I am worried people missed me updating the previous post.

The implementation of JIT2 provided is functional although still a few minor improvements remain. Some results could be cached at factory initialization/load instead of computed every tick and the UI for storage information might need improvement. That said it should be fully functional and appears to work perfectly with backwards compatibility for existing games and the ability to start new games using fully functional JIT2.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on October 09, 2014, 03:06:25 AM
So has anyone managed to test out the patch yet? Does anyone need a binary build for windows?

I would like to push this forward towards something that can be integrated into the main branch. Due to the size of the code change it is important people test it for bugs as I might not have spotted them all.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Junna on October 09, 2014, 04:34:42 AM
Though I don't play standard, I think you'd probably get more luck if you did upload some binaries? I'd guess most people up for testing would be more casual and might not know how to compile and what to do with patches and whatnot.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on October 09, 2014, 05:21:17 AM
Here (https://www.dropbox.com/s/sc7vgpb1l4n0ruv/Simutrans%20JIT2%20Test.zip?dl=0) (hosted on dropbox) is a Windows executable. The DLL it comes with is the Visual C version of pthreads and is required as the usual one all pre-build versions of simutrans come with is the GCC pthread library. Extract it into any nightly installation of simutrans (might want to rename the executable) and it should work. Do note that you require Visual C++ libraries to be installed (which there is a good chance they may be already). One day I really should update my visual studios but Microsoft's new licence policies are too strict in my opinion. There is a high chance that SDL2 is not used by this build so performance may be worse than expected as a result. As this is based on nightly it should work with half-heights.

The version provided can both load existing games and save new ones. Existing games will load in legacy mode and should act the same as they did before (well the results will not be deterministically the same but from your observations it should be the same). When starting a new game (any pak set) you now have the option to use JIT2. The Just_in_time field accepts 3 values, with 0 and 1 being legacy modes and 2 being the new JIT2 system.

Be aware that I am not responsible for any loss of save data caused by using the above. I strongly recommend you backup existing saves before loading them with the above executable to prevent loss of data. Saves made using the above version will not be backwards compatible, meaning only this executable (and all builds based on the patch) can open them. If the patch is ever incorporated into the trunk then future versions of simutrans might be able to load saves produced using the above executable but there is no guarantee that will happen. I strongly advise to not use this executable for any long-term map and recommend you focus on trying out industry as passenger/mail mechanics should be unchanged.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on October 12, 2014, 11:44:11 PM
Finally I found a bit of time to test the new version of JIT2. I managed to switch jit2 for running game as well, but I found a problem with middle factories that have overcrowded input. These factories do not consume or produce anything, but still their demand counter goes up as if they were producing. When the new input cargo is delivered, the input storage stays at max and demand counter goes down by the amount delivered.

EDIT: saving and loading the game seems to fix this issue.
EDIT2: now I have a steel mill that has lot of coal but no iron ore, negative demand counters for both, and not ordering any more. There is enough supply of iron ore and demand for steel. There is another stell mill supplied by the same mines, running just fine.
EDIT3: saving and loading helped again. now the factory is ordering both coal and iron ore, even if one of them is empty. However this leads to oversupply of one input. I think that if the facotry is not producing only the empty input shall be ordered.

Further observations: power demand and production is now a bit jumpy, even if the factory is well supplied so that it could run on full power boost all the time. Hmmm, and when it runs out of stock then the production info shows full boost, instead of the base value.

I have tested the old version of JIT2 with max. demand buffer set to 10 and overfull penalty at 100% (not ordering if overfull) and it worked fine. I Must say that I liked the original JIT2 - just keep it as simple as possible.

One possible improvement would be, if a supplier can choose between more consumers - send the cargo to the one that has higher demand counter.

Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on October 13, 2014, 12:30:26 PM
Quote
I managed to switch jit2 for running game as well
I did kind of warn that was not supported...

They operate so differently that switching mid-game will cause all kinds of errors. In order for a JIT1 to JIT2 conversion to occur each factory would need to have a special purpose conversion method called.

Part of the reason for this is that JIT2 factory logic uses sate counter caches to avoid excessive computation of output amount when it is logically not possible. Classic factory logic does not use this so corrupts those values. The other part is that unions are used that are shared between demand counter and maximum in transit. This was done to avoid "useless" members and so keep down the factory object size.

A simple save based conversion is also not possible as extra values are saved when running in JIT2 mode (to avoid bloating classic saves with useless members). In this case the game will probably fail to load at all.

Saving and loading a few times will probably fix most of the problems, however chances are not all of them (as I never intended for conversions to occur). This is the problem with backwards compatibility support with such a major change. It would be very easy to convert existing JIT into JIT2 (as with my previous implementation) however people specifically requested it remain backwards compatible.

Quote
However this leads to oversupply of one input. I think that if the facotry is not producing only the empty input shall be ordered.
It should not make any difference. One input is full and the other is empty however it still is ordering as much as it consumes. A new shipment should arrive ~ when there is enough space for it. If not then yes some overflow occurs however this wastage will eventually correct the system so that there is enough space by the time the next shipment arrives. The other input being almost empty should not mater as it should always have enough if sufficiently supplied to keep producing.

Quote
Further observations: power demand and production is now a bit jumpy, even if the factory is well supplied so that it could run on full power boost all the time.
This is a problem with the actual power network logic! It uses some form of filter (low pass?) to determine how much power to supply a factory with so has considerable problems keeping up with the dynamic power demands of factories 100% of the time. However it is good enough to keep up >=99.5% of the time that it will still report to you at the end of month average that it is getting maximum power boost. If the line logic had immediate response then it would not suffer such oscillations.

The reason power plant output oscillates is because their output amount is now based on actual work done so is subject to rounding and accumulation. This shows how much the actual production oscillates in factories during their operation which is usually transparent to the user.

I had to fix an underflow bug with power logic for this to work. Before it would refund power demand into negative amounts if network utilization was less than 100%. Now it does not refund anything (power demand is 0) since it was fully supplied. This logic appeared to be there to allow multiple transformers per factory however since that is not allowed I doubt this change will affect anything and in any case I doubt it was correct.

It should be noted that the amount of power a supplier/factory/consumer consumes is now a per-unit production amount. Before it was a flat amount. This means that passenger and mail boosts will no longer make a factory more power efficient and low production will no longer make a factory less power efficient. It also means that fully boosted factories working at full production will now consume more power than before however this is offset by the fact that most industries seldom work fully where the constant power efficiency will result in huge energy savings.

Quote
Hmmm, and when it runs out of stock then the production info shows full boost, instead of the base value.
That is intentional as I described earlier in this topic. A factory that is not producing anything but supplied with power (even if that is 0 power and just a transformer) still gets 100% of their power demand satisfied so maximum boost. This is partly to allow for correct order logic (since it has to order at the boosted rate and not the base rate) and also to allow for the correct power boost graph (it is showing how much boost the factory got, not how much production it did). In my first attempt I had to do a hacky and mathematical error inducing work around for this specific case because of the illogical power boost logic behaviour.

Ultimately one would want the factory to be able to query power networks themselves and read off utilization from those (not caring how much power demand was actually fulfilled). This would mean that power boost then is directly related to power network utilization and cases of networks with no power in them would be correctly handled. Currently the only interaction factories have with power networks is the amount of power they demand and the amount of power they received/sent.

Quote
I Must say that I liked the original JIT2 - just keep it as simple as possible.
There is a lot more "under the hood" of this change than with the previous. For example all factory inputs and outputs are normalized so less production error will occur. Also I tried to add some kind of dynamic logic selection capability which could ultimately end up exposed to pak set authors allowing them to choose how they want a factory to behave. I also have made ordering more fair by letting every supplier have a chance to fulfil orders (this applies to classic and JIT1 as well).

Quote
One possible improvement would be, if a supplier can choose between more consumers - send the cargo to the one that has higher demand counter.
Currently it uses the standard round robin distribution logic as it always has however ordering will occur for at minimum 1 delta t period. If a demand counter if constantly being depleted most shipments from a shared supplier will end up going to the factory which has a large demand counter.

The problem with sending to "largest demand counter" is that in the case of two industries who are having their demand fully quenched and produce ~ the same rate then this opens up the possibility for dispatching bias to occur based on internal ordering where 1 supplier (which ticks earliest) will only send to the one factory and the second supplier only to the other. If it were implemented it would need a cut-on value representing when demand starts to accumulate.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on October 14, 2014, 07:03:42 PM
I did kind of warn that was not supported...

I know you warned, but I wanted to test on a running game where I could easily see the difference. And I really think that the player should be allowed to switch during the game, like it was with the first patch.

Quote
The problem with sending to "largest demand counter" is that in the case of two industries who are having their demand fully quenched and produce ~ the same rate then this opens up the possibility for dispatching bias to occur based on internal ordering where 1 supplier (which ticks earliest) will only send to the one factory and the second supplier only to the other. If it were implemented it would need a cut-on value representing when demand starts to accumulate.

Well with the current behavior - each factory can send during one tick - I see similar problem. Real exapmle: I have 3 coal mines and 3 iron ore mines supplying two steel mills. There is not enough coal for both, but one of them is running at 100% and the other one gets the rest of production. Just because when the coal mines have something to send, all of them send it to the same steel mill.

Similar effect is seen with other factories supplying shops. Most shops run at steady 100% and one or two are undersupplied. I would expect that all the shops would run at around 90%.


One comment to power consumption - Wouldn't it be better to split the power consumption (and other fixes to factory code) into a separate patch? I think they are not dependent on JIT2 and might be beneficial even for old JIT.  I think that smaller patches clearly focused on one thing are more likely to get accepted.

Update: I have noticed that you also changed the logic of factories with multiple outputs. (e.g. refinery, printworks) Now they consume less inputs, if some of their outputs are not wanted.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on October 14, 2014, 09:46:14 PM
Quote
And I really think that the player should be allowed to switch during the game, like it was with the first patch.
The first patch did not allow them to switch, it forced them to switch. I require interfaces to write the required code to do a live conversion but as far as I know the game does not support this (it will only change the setting and then hope that from that time onwards everything works).

Quote
Well with the current behavior - each factory can send during one tick - I see similar problem. Real exapmle: I have 3 coal mines and 3 iron ore mines supplying two steel mills. There is not enough coal for both, but one of them is running at 100% and the other one gets the rest of production. Just because when the coal mines have something to send, all of them send it to the same steel mill.

Similar effect is seen with other factories supplying shops. Most shops run at steady 100% and one or two are undersupplied. I would expect that all the shops would run at around 90%.
This should not happen as shops that are undersupplied will have >0 demand counters all the time so any free product will be shipped to them as they are always ordering goods. The round robin distribution should assure that at least a fraction of the product from each supplier is sent to them. The case described should only occur for consumers that are undersupplied with fewer suppliers than others since the goods will be distributed equally when supply is limiting. If this is not the case it is an error.

This does not do production allocation to try and make all industries work from their connected suppliers. It uses the same fair distribution model as always to allocate goods to consumers. If two suppliers have demand and a consumer sends goods it will send them to each 50% of the time. This does mean that if a consumer with a lot of suppliers has demand that is being fulfilled and shares a small subset with another consumer which is undersupplied (max demand) then it will still siphon off some of the goods from those suppliers even though for maximum consumer usage you would want them going fully to the under supplied consumer.

If a consumer has too much contention with all its suppliers I would view this as a bug with the economic model of the game and not the factory. It should try and assure that a consumer has enough suppliers to cope with all the contention.

Quote
One comment to power consumption - Wouldn't it be better to split the power consumption (and other fixes to factory code) into a separate patch? I think they are not dependent on JIT2 and might be beneficial even for old JIT.  I think that smaller patches clearly focused on one thing are more likely to get accepted.
The old industry code just could not support it because it lacked the necessary processes to compute dynamic power consumption. It had no idea of "work" done by a factory. Additionally I was specifically told to retain classic JIT behaviour (un altered in any easily noticed form) at the request of some people earlier in the thread.

The new power model is required for JIT2 because the continuous load on industries would cause excessive power consumption once they pass the 25% utilization threshold as they would otherwise consume full power. In any case I am sure most people will agree that the old power model was kind of non-sense with power consumption either being maximum or nothing independent on the actual production or work done by a factory. Pak balancer will also have an easier time with JIT2 as now power consumption scales with production and work done.

The jumpy power boost is annoying and something an improved power network logic patch might be able to fix. However I still think having this dynamic power consumption logic is better than static power consumption logic since now balancing power networks is a lot easier. A large scale power network might end up varying ~4-6% only and you will be fully boosting more factories than before possible.

Quote
Update: I have noticed that you also changed the logic of factories with multiple outputs. (e.g. refinery, printworks) Now they consume less inputs, if some of their outputs are not wanted.
Yes, this was mentioned earlier in the thread and met with mixed opinions. However the new core industry code has paved the way for besche based industry logic (well it is easy to add now, I would even recommend it as a next step after this patch) where by a pak author could choose what logic a factory would use rather than letting the game guess. In this case an updated classic logic could be added for special cases like refineries if that is the pak author's desire.

However I personally think the scaling is useful as in pak sets like pak64 those structures were notorious for having absolutely stupid oil usage. A fully operational one with 8k production per month consumes 24,000 units of oil which is a crazy logistical challenge as well as a total gold mine. Now it is harder to reach that utilization level and the oil will flow more smoothly to it, something I am sure some people will like.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on October 15, 2014, 05:26:08 AM
The first patch did not allow them to switch, it forced them to switch. I require interfaces to write the required code to do a live conversion but as far as I know the game does not support this (it will only change the setting and then hope that from that time onwards everything works).

It's not that long ago that changing settings for a running game was "impossible". It was very unpopular with the players.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on October 18, 2014, 09:01:24 PM
Did somebody other tested this? I think it is very interesting, but in the last two months I lacked serious time for in-depth testing.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on October 18, 2014, 10:31:45 PM
I have played a while with this patch. I must admit it is not a fresh game, but one I played for log time and I switched to JIT2. After few save/load cycles it started to run fine. But frome time to time some industries stop working - it happened a few times with steel mill - it ran out of iron, but the demand counter was deep in negative numbers and did not grow towards zero to restart the production. save and reload helped. I have to try some fresh game to see if it happens again. But if you want to investigate I can share the save.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on October 19, 2014, 12:20:24 AM
I cannot confirm if it is completely bug free as the changes made by this are pretty big (this is far bigger than the maximum in-transit change and that had many errors that made it into 120RC). If you can produce an error in a re-creatable way (from a save) then please explain how and I will look into it as soon as possible.

What you describe sounds like a status cache integrity failure. To prevent excessive polling of outputs I cache their status (how many are inactive) so that the corresponding new logic can be ignored (no need to order anything if all orders are full, or no need to produce anything if all inputs are empty etc) so a tick should execute a tiny bit faster, especially for disconnected industries (which often a large number are on servers). However if there is a logical error with the increment/decrement of the cache it will break quite badly and what you mention could occur.

I had very limited test cases. Using Pak64 I had power supplied to a combination of steel industry (single output producer) and oil industry (multiple output producer). I also checked the operation of consumers such as the coal power station (power plant), updraft generator (I thought it was "solar tower" according to the documentation 8 years ago odd, this is a constant power producer) and the construction wholesaler (multiple consumer). During the tests I also picked up an underflow error with power nets refunding a negative amount of demand if over supplying (illogical and error prone as power demand is signed so an underflow occurs).There were many small logical errors that I solved with my code as well so it is possible one such error has escaped un-detected however I do need a test case to reproduce it for a solution.

I ran a simple power station test on pak128 to check operation since there actual power factors are defined and everything appeared to be functioning as expected.

To check legacy support I loaded 3 different server games from Fifty over the last year. These games were pretty massive with probably well over 100 industries (pak64) and companies making hundreds of millions and the results appeared to be as expected with no noticeable deviation for someone used to old mechanics. I do know for a fact that results will not be 1:1 however but this all comes down to rounding with the new implementation probably producing more accurate results due to factories operating in a normalized mode.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on October 19, 2014, 09:30:23 AM
Did somebody other tested this? I think it is very interesting, but in the last two months I lacked serious time for in-depth testing.

In order to test it, I need to be able to switch it on (maybe even off as well) for existing games. Being unable to do so will also make it less likely that players will use it, depending on how often players start new games. (I keep playing the same map for years.)
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on October 19, 2014, 09:44:28 AM
You may try what I suggest in the long distance patch thread: add "dialog_tool[27]=,~" in menuconf.tab.
Then you will be able to change to JIT2, but you may run into the same bugs as I did.

I have managed to find a reproducible bug. Look at https://uran.webstep.net/~vladki/simutrans/, download the autosave07.sve and addons.zip - it is my savegame converted to jit2 - run it with pak128.Britain 1.14 (last single-height stable) and some recent nightly.
It will be loaded close to two stell mills, one wil be quickly recovering from buggy undersupply (demand -200 or so). Then you can watch the steel demanding industries slowly restart production and waiting steel trains to leave the station. When you load the game there are a few trains waiting for cargo, but when the steel is produced at full, at most one train shall be waiting for cargo.

Then let the game running for a while and move to the other end of map - find car dealer in Christminster. Normally it should be selling some 260 cars/month. (max with power boost is 320). It will slowly recover production. But the trains can deliver 60 cars at once, while the input storage of this shop is only 53. So it happens from time to time that the 7 extra cars are thrown away, and deduced from demand. I am not sure if that is the trigger for the bug. If you run the game until august around 20:00 (in game time), the demand buffer of this car dealer gets quickly satisfied by the car factory and drops to -4, never to rise again. There are still some cars on the way, but they are consumed without increasing the demand counter. It drops even more as some of the supplies overcrowd the dealer. On september 3:00 the last delivery happens, and car dealer is not ordering any more.

With fast forward I got the car dealer broken at august between 12:00 and 15:00. Tried 4 times, and always got the bug, maybe with different timing.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on October 19, 2014, 03:09:15 PM
You may try what I suggest in the long distance patch thread: add "dialog_tool[27]=,~" in menuconf.tab.
Then you will be able to change to JIT2, but you may run into the same bugs as I did.

Those "bugs" were what I was talking about. They would cause my test to fail. (They are not really bugs, because they are a known effect of doing somewhing JIT2 doesn't support.)
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on October 19, 2014, 06:47:45 PM
Quote
I have managed to find a reproducible bug. Look at https://uran.webstep.net/~vladki/simutrans/, download the autosave07.sve and addons.zip - it is my savegame converted to jit2 - run it with pak128.Britain 1.14 (last single-height stable) and some recent nightly.
It will be loaded close to two stell mills, one wil be quickly recovering from buggy undersupply (demand -200 or so). Then you can watch the steel demanding industries slowly restart production and waiting steel trains to leave the station. When you load the game there are a few trains waiting for cargo, but when the steel is produced at full, at most one train shall be waiting for cargo.

Then let the game running for a while and move to the other end of map - find car dealer in Christminster. Normally it should be selling some 260 cars/month. (max with power boost is 320). It will slowly recover production. But the trains can deliver 60 cars at once, while the input storage of this shop is only 53. So it happens from time to time that the 7 extra cars are thrown away, and deduced from demand. I am not sure if that is the trigger for the bug. If you run the game until august around 20:00 (in game time), the demand buffer of this car dealer gets quickly satisfied by the car factory and drops to -4, never to rise again. There are still some cars on the way, but they are consumed without increasing the demand counter. It drops even more as some of the supplies overcrowd the dealer. On september 3:00 the last delivery happens, and car dealer is not ordering any more.
Found the bug. In the case of a shipment arriving while the demand counter was maxed out (inactive) it would not re-activate it. The logic for ordering would not re-activate it because it considered it already active. Basically your typical "invalid state" problem sigh. Saving/loading fixed this as the activity cache is rebuilt on load (and will correctly detect the demand as being active).

I have a fix for this that I will publish on this thread later tonight (patch file and executable). I will also update the patch to the latest SVN nightly version making sure all conflicts are resolved. I will not post a new message (unless someone else posts) so please check back later.

Quote
Those "bugs" were what I was talking about. They would cause my test to fail. (They are not really bugs, because they are a known effect of doing somewhing JIT2 doesn't support.)
I am open to suggestions how to do a conversion more neatly.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on October 19, 2014, 09:21:53 PM
I think I will try to relase a stable (or rc) this week and then commit it afterwards. Just if there is some effect with server games, there is a somewhat stable (which is lacking at the moment).
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on October 20, 2014, 01:39:50 AM
Ok I have made some revisions to it. The latest patch file (should be compatible with SVN) is attached below. Here (https://www.dropbox.com/s/l3lca09wuf51e9x/Simutrans%20JIT2%20R2.zip?dl=0) is a pre-built (using nightly) executable for Windows targeting Visual C++ (version 10) which should work with any nightly installation of simutrans.

The following notable changes were made.

Quote
I think I will try to relase a stable (or rc) this week and then commit it afterwards. Just if there is some effect with server games, there is a somewhat stable (which is lacking at the moment).
Sounds good to me. People could then always try it out when it filters down to the nightly builds.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on November 06, 2014, 08:21:08 PM
HI, I have been playing for some time with jit2r2, and so far so good - no errors. And it is a converted game. If anybody wants to try on linux, here is 32-bit binary: https://uran.webstep.net/~vladki/simutrans/sim-gcc4-jit2r2 based on nigthly r7348.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on November 16, 2014, 12:08:16 AM
This weekend I will relase 120.0.1 and then the new JIT can be incorporated. Just waiting for the nightly mac builds.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on November 16, 2014, 12:17:40 AM
Ah excellent. It will be great when people can play it in the nightly builds.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Yona-TYT on November 16, 2014, 08:24:15 AM

This weekend I will relase 120.0.1 and then the new JIT can be incorporated. Just waiting for the nightly mac builds.


You refer to the r7373 ??
Title: Re: Suggestion for JIT mechanics revision.
Post by: An_dz on November 16, 2014, 11:05:35 AM
No, he refers to whatever revision when mac builds get back working, then prissi will release and JIT will be incorporated.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on November 16, 2014, 05:34:10 PM
Quote
No, he refers to whatever revision when mac builds get back working, then prissi will release and JIT will be incorporated.
They are not working? The nightly builds appear to be building fine for Windows, Mac and Linux. In fact the last nightly for all 3 was built earlier today. Or are you saying that there is an error in the Mac builds currently preventing them from running (not building as that appears to be working fine)?
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on November 16, 2014, 06:26:40 PM
In fact the last nightly for all 3 was built earlier today.

Prissi's comment was almost one and a half hour earlier still.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on November 16, 2014, 09:14:24 PM
Sorry, I only have computer time after 20 o'clock (GMT). So the stable has been released. I have to do some other stuff (like the Linux makeobj) and also try to get the haiku build in order and then JIT2 will be in the nightlies.
Title: Re: Suggestion for JIT mechanics revision.
Post by: An_dz on November 17, 2014, 05:07:50 AM
are you saying that there is an error in the Mac builds currently preventing them from running (not building as that appears to be working fine)?
I have chosen the wrong words, to me prissi was talking about waiting them to be built this last time before releasing as Mac is probably the last to be built.

Anyway this discussion is useless as the possible problem (that never existed) was fixed as prissi has released a new version and the point was discussing the revision number. All I have read was: "waiting for the mac nightly to build".
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on November 24, 2014, 11:36:42 PM
Back on topic. There were lots of single onliner if and so on. Cleaned up the code a little.

The ability to save for older version was broken. Hopefully this works again now.

Why does the new version only display half of the max and storage values compared to a similar game for the old version. It seems to me rather a display bug. Please check info_prod and so on.

Some questions remains:
Should it be enough to determine the type of a factory once instead on every steps?
For single and multiple consumer/factory procuder: COuld this logic be not joined. The overhead seems rather small compared to the huge repetitive case construct.

Despite the display(?) bug submitted in 7378
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on November 24, 2014, 11:49:45 PM
Quote
Should it be enough to determine the type of a factory once instead on every steps?
Yes, it should actually go as part of the factory besche (since it is not state as it does not vary during factory life) and could even be something pakset creators can configure. This would allow some special logic for certain factory types with unusual behaviour.

Quote
For single and multiple consumer/factory procuder: COuld this logic be not joined. The overhead seems rather small compared to the huge repetitive case construct.
Yes they could be removed and use the multiple logic instead. I was not sure about the trade offs having or not having them would bring.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on November 25, 2014, 12:04:32 AM
Sorry, editing took so long. See about the error with amount at factory. Maybe scaling of some factors had some side effect in displaying.

I think in order to get compatibility the type needs to be either calculated onload of the besch of taken directly from it. SO I would suggest to move this logic to factory_reader. (Or I can do it myself, if you like).

About joining logic. THat is much better, because if there are typos/errors, then you do not have to maintain almost the same code twice. Reusing is really helpful.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on November 25, 2014, 01:30:52 AM
Quote
Sorry, editing took so long. See about the error with amount at factory. Maybe scaling of some factors had some side effect in displaying.
I thought I corrected all the display errors.

Quote
I think in order to get compatibility the type needs to be either calculated onload of the besch of taken directly from it. SO I would suggest to move this logic to factory_reader. (Or I can do it myself, if you like).
I avoided touching those parts because it would couple the change to yet more parts of the game and I was not sure how they work and what could be changed with them. Making an already big change even bigger can lead to other problems so I wanted to make sure what I had was working well (which it appears to be from tests).

In any case this is not particularly critical for release and could be a future update. Yes it will affect performance but even on maps with hundreds of industries other factors probably are affecting performance more at the moment.

Quote
About joining logic. THat is much better, because if there are typos/errors, then you do not have to maintain almost the same code twice. Reusing is really helpful.
In theory some of the operations could be turned into function calls for even more re-use. I just was not sure where to break stuff apart.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on November 25, 2014, 10:26:57 PM
The display seems off; upon loading an old game (i.e. the pak 64 steel mill) then the number in ore are half and coal are twice the value. YOu see that when a truck with 42 tons of coal arrives the dispolay increases to 84 tons.

The production itself is synchron, and the output seeems similar. ONly the numbers of the screen are wrong. Maybe it has something to do with the consumption percentage.

I removed the code which determines the type of factory to the creation (baue) of factories (which is also used for loading).
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on November 26, 2014, 12:33:13 AM
Quote
The display seems off; upon loading an old game (i.e. the pak 64 steel mill) then the number in ore are half and coal are twice the value. YOu see that when a truck with 42 tons of coal arrives the dispolay increases to 84 tons.
Ops... Forgot to update the old display logic for the new way that production units are handled. When running in JIT2 mode it will correctly display the amounts but in 0/JIT1 it will not scale them appropriately.

Example, to get current amount this is used.
Code: [Select]
(uint32)(0.5 + (double)ausgang[index].menge * (double)(besch->get_produkt(index)->get_faktor()) / (double)(1<<(fabrik_t::precision_bits + DEFAULT_PRODUCTION_FACTOR_BITS))),

The old logic is still using this.
Code: [Select]
(sint32)(0.5+ausgang[index].menge / (double)(1<<fabrik_t::precision_bits)),
So it is not factoring in the scale factor correction.

I will try writing a patch for it in a little while. I am terribly sorry for the oversight.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on November 26, 2014, 09:48:53 PM
Do I understand it right that the JIT2 is in latest nightlies?

If so, then there are some bugs - most important - the value which  JIT is used is not saved properly - I save with 2, and load 1.

Save from previous games (with patched JIT2) loaded fine with jit=2, but there was another problem with power - factories did not use the power even if available. SO I tried to fix by save/reload - power was consumed but could not switch to jit2.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on November 26, 2014, 09:57:35 PM
Do I understand it right that the JIT2 is in latest nightlies?

Since r7378 by the looks of it.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on November 26, 2014, 10:18:15 PM
That cannot be saved right now without bumping a savegame. You can start your game with "-server" on the commandline, to save this setting.

Usually I do not bump the savegame until thing are a little tested at least. Otherwise people will lose their games.

But next nightly will remember when loading an old game from your simuconf.tab setting whether you want of old new enforcements.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on November 30, 2014, 02:36:07 AM
Here is a patch to fix the legacy factory information display to show the adjusted units. The units shown for factories with factors not 100% was being incorrectly displayed for the reasons mentioned earlier. This patch corrects that by adjusting with the factor in the case of running in legacy JIT0/1 modes.

As an additional change I updated the logic for generating the output text to use fixed point instead of double precision floating points. Not only does this remove possible platform differences to display results and mathematical error but also should perform slightly better. The performance gain is probably trivial (certainly not something you will ever see) but the precision and platform independence I would say is worth the change.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on November 30, 2014, 11:24:13 PM
Thank you, incorporated.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 01, 2014, 09:19:08 PM
Hi I have noticed a bug:

I have a newly spawned paper factory needing woodchips and chemicals. I started to deliver woodchips first, and filled the input storage, and the factory is still ordering more woodchips - paying for them and discarding them. The factory is not producing because it has no chemicals. Save (pak128.Britain): https://uran.webstep.net/~vladki/simutrans/britanie.sve. Look for paper mill at (90,3).
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 01, 2014, 11:22:46 PM
I cannot load the save using the nightly (r7389) and nightly version of pak128.Britain. It crashes during loading (the process hangs).
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 02, 2014, 12:02:10 AM
Tried with latest r7389 and updated the save - loads well. Just I play with old pakBritian (1.14) - without double/half slopes.
The game shlould load near car factory which currently runs out of plastic, so you can see the bug.

Another thing is that power usage is somewhot flaky - many factories do not use power even if they could. E.g. refinery and one of the steel mills supplying the car factory - they have undergroud transformers, so you can't see them, but other factories with visible transformers are broken too.

One more issue: IIRC the new factory rules for multi-output facories say that if some products are not produced, then less inputs are consumed. I have a printworks - input 100% paper, output 80% news + 20 % books. I produce only books, so I would expect that paper consumption would drop to 20%, but it is at cca 50%. (I know that this factory logic change was subject of discussion, with oil refinery being a bad example. But printworks are perfect fit for the new logic - 1ton of paper can be used to make either 1 ton of books or 1 ton of new, or 1/2 ton of both...)
(https://uran.webstep.net/~vladki/simutrans/simscr05.png)
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 02, 2014, 12:26:15 AM
The issue was with the way you linked the save. It became difficult to download completely as it was being managed as a webpage rather than as a file. I have now managed to download the entire save file and load it and am investigating.

EDIT:
Quote
Look for paper mill at (90,3).
The map has no such position. It is not in bounds. I tried (3,90) and that was a tunnel.

The only paper mill I can find on the map that uses chemicals is at (27,89) and it is being supplied both with chemicals and with woodpulp further more the map is not using JIT2 so demand buffers are not enabled (it is using maximum in-transit instead).

The car factory is out of plastic but appears to be ordering correctly in response to the classic JIT1 ordering behaviour. It also appears they are consuming power as expected with the classing JIT1 behaviour.

I will try and force it over to JIT2 (some how) and see what happens.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 02, 2014, 12:32:59 AM
I had problems saving the game with jit2 - it always returned back to JIT1. Setting jit2 in simuconf.tab ad using latest nightly helped.

Coordinates depend on rotation. But yes you found the right paper mill anyway - the only one that needs chemicals. I have supplied it for a while but stopped the supplies so it will eventually run out of chems.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 02, 2014, 01:12:43 AM
The papermill is working correctly. Once the order buffer for chemicals is full (matches the storage amount) it stops ordering pulpwood as expected. The issue why it over-ordered so much is due to the demand counter for chemicals starting at 0 and taking a considerable amount of time to fill up. It also had a fair amount in storage already which gets converted to over-order due to the mechanics of JIT2. The industry has several months of chemicals stored in it so it does over-order quite a bit. However without supplying it with more chemicals it does stop ordering pulpwood eventually. The main issue is that the effective months of storage between pulpwood and chemicals is vastly different so the chemical demand building up causes the pulpwood storage to overflow. If the storages were even (say both were 2 months of production) then you may find that they do not overflow. This is just a qwirk with how JIT2 works and something that pakset authors might want to consider in future revisions.

I am guessing there could be better logic to handle this specific case (such as setting the demand buffer limit for all goods to the minimum input storage for factories) but that would complicate things a bit more. However it still functions correctly if chemicals and pulpwood continuously flow. In the end you will still ship more to the factory if you keep both steadily supplied instead of trying to overflow it with pulpwood.

The issue with power is very strange as the original patch version that prissi merged from does not suffer the power issue (the refinery consumes power as expected oscillating from 99% to 100% due to the filter used for power distribution and varying load). This means that the error was introduced during the merger. As a rough guess it could be that when the boost logic is determined it fails to select BL_POWER so it thinks the industries do not need power (so will not run the boost logic to do so).

EDIT: I have found the cause of the power problem.
Power requirement is not computed when the factory is built. One would imagine it was and is if you build a factory in-game however during loading it is only computed at a later step after all factories are built from the save file (before linking occurs). As such when setting up the logic to use it assumes the factory power demand to be whatever is in the memory at that time (why some factories work while others do not).

EDIT2: Here is the patch to fix the power logic. I tested your map and the refinery was correctly consuming power. It works by reading the power consumption figures from the besch instead of from the factory. This should be backward compatible with old paksets (64 etc) as a large positive constant is used for the default setting when nothing is defined.

It should be noted that factories may consume more or less power in JIT2 which might not be reflected by the power creation rate. This is because JIT2 scales power consumption with the amount of work done giving pakset authors the ability to define a "per-unit produced" power consumption amount instead of it being a fixed amount when the factory is working more than at 75%. Factories working at a rate less than their un-boosted form will use less power (eg a pak64 coal mine supplying a steal mill) while factories working above their un-boosted production amount will consume more. Over a large network power should stabilize at approximately some percentage (if all connected factories are constantly producing at equilibrium) but can also still show high variance (most industries under supplied as no power is used when not working).
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 02, 2014, 09:39:14 PM
Thanks for the patch (not tried yet however).

To the papermill issue - In my (theoretical) version of JIT2 this was dealt so that - in case of non-producing factory - inputs are ordered only if they are not in stock at all. In other words, if there are some goods in input, and the factory is not producing, there is no need to order more. The reason for not producing might be either lack of other inputs or full output.

If you dont want to change that, you can set max demand to some low value. Some time ago I have fiddled with one of the early versions of JIT2 patch, and modified it so that the input buffer max was always 10 (I used some constant - min cargo amount delivered or something). It worked just fine as with max buffer = max input. It might help in cases like this paper mill.

Did you have time to look at the printworks issue as well?
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on December 02, 2014, 10:51:50 PM
INcorporated, thank you. With the code stabilising, maybe one should consider consolidate the single and multiple into the same routines?
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 02, 2014, 11:18:15 PM
Quote
To the papermill issue - In my (theoretical) version of JIT2 this was dealt so that - in case of non-producing factory - inputs are ordered only if they are not in stock at all. In other words, if there are some goods in input, and the factory is not producing, there is no need to order more. The reason for not producing might be either lack of other inputs or full output.
No that is not the way they work because that would stall the freight pipe-line. It orders exactly at the rate of consumption independent of what is actually consumed. It might take a long time for goods to reach the factory in which case a large in-transit amount will build up however when they do arrive it should mostly be a continuous production chain with goods arriving as they are consumed. If the output is fill it does not order anymore because it is producing nothing (in which case the in-transit amount will eventually cause an overflow to occur as it collapses). If you stop one of the goods then it will keep trying to order that good (and all other goods) at the rate of consumption until the demand buffer caps out (demand buffer equals the maximum storage). Other goods may overflow during this time but that is the result of a pipe-line stall rather than incorrect ordering logic (its ordering the right amounts, just suddenly nothing is being delivered for one of the goods). It cannot be abused because always you will ship more fulfilling the demands properly so that production is constant.

Quote
If you dont want to change that, you can set max demand to some low value. Some time ago I have fiddled with one of the early versions of JIT2 patch, and modified it so that the input buffer max was always 10 (I used some constant - min cargo amount delivered or something). It worked just fine as with max buffer = max input. It might help in cases like this paper mill.
Yes in this case it would help. In fact the best would simply be set the demand buffer max to the minimum factory storage amount of all inputs (internal units, not readable units) since that way if production was to stall then it would fill up at an appropriate amount and avoid excessive overflow due to the storage imbalance. However if you actually tried to keep the paper factory working as opposed to trying to find faults with the order logic it makes no difference still it will still order at exactly the same rate. It also is no worse than JIT1 where you could get a factory flooding itself with 15,000 units of coal and overflowing the hard maximum just because you connected it to a lot of coal mines far away.

One could also ask why a paper mill stores 3 months of chemicals yet not even 1 month of pulp-wood in a JIT driven market... As I stated it could also be something for pak authors to consider in future since currently most paks were designed with over-sized input storage to accommodate the bursty floods of JIT1 (pak64 seems an exception with large factories/consumers often having too smaller input storage for their production rate).

Quote
With the code stabilising, maybe one should consider consolidate the single and multiple into the same routines?
I thought this was already done until yesterday when I noticed that the original code was still being used while tracking down the source of the problem. I could possibly prepare it over the next few days.

It would be very useful if people could test the multiplayer stability of the feature. Instead I still see game severs running legacy builds.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 06, 2014, 06:30:00 PM
I have revised the code a bit as requested. All single input/output logic has been removed, being replaced with usage of the multiple logic (which was tested to be working fine). I also pulled out some constants and documented them for possible later adjustment. The logic for scaling output production in JIT2 was pulled out and turned into a separate method to reduce code duplication. The result was the removal of a considerable number of lines and hopefully will produce smaller code as well.

I also got fed up with some of the compile warnings that were being thrown at me during debugging. As a result I took an attempt at fixing them (simply adding explicit casts in most cases). These do not have to be included or could be a separate commit. The JIT2 revision was only with the factory cpp/h files, all other files were these warning fixes.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 08, 2014, 09:54:27 PM
I have a few more bugs for JIT2 (r7403). In the game I play I have one hardware factory (planks+steel = hardware). It was producing at 100%, but there were more shops and oversupply of planks and steel so I decided to connect it to power. I expected that the production will go up from 572 to about 1000 (consumers total 1080). But it went up only to some 700. When you watch the info box, you will notice that the factory often runs out of planks, even though the sawmill can supply much more. If you watch carefully, you will notice that it orders less planks than it consumes. Sometimes it "forgets" to increase the demand counter.

Another is mentioned earlier with screenshot - I have 4 printworks. 3 of them (bunched together) produce books and newspapers at almost 100% and consume also almost 100% paper. But the fourth one (not electrified), produces only books, but consumes too much paper: 381 boxes/month to make just 152 boxes/month of books. As the production says input 100% paper, output 80% news, 20% books, which totals 100%, I would expect 1:1 ratio for consumed paper and produced books, if no newspapers are printed.

Save game is here: https://uran.webstep.net/~vladki/simutrans/autosave01.sve
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 09, 2014, 12:44:46 AM
Quote
But it went up only to some 700. When you watch the info box, you will notice that the factory often runs out of planks, even though the sawmill can supply much more. If you watch carefully, you will notice that it orders less planks than it consumes. Sometimes it "forgets" to increase the demand counter.
That is not what I am experiencing. The Hardware Factory has more than enough planks and always has planks in storage but is lacking steel. The two steel mills cannot produce enough steel to supply it (one of them has insufficient coal supply). JIT2 mechanics will only place orders for factories if none of the inputs is under supplied (demand counter exceeds input storage). Since there is too little steel, the plank ordering will be bottlenecked at the rate of steel ordering.

Depending on the transport infrastructure used, there is a chance that there may be steel and no planks. However this time is meaningless since it fits into the production lost due to insufficient supply (the factory cannot always be working as there are not enough resources available). If sufficient steel was to be provided so that demand was satisfied then eventually both a plank and steel buffer would build up at the factory and it will operate at 100% (or less if the product is over supplied).

Quote
Another is mentioned earlier with screenshot - I have 4 printworks. 3 of them (bunched together) produce books and newspapers at almost 100% and consume also almost 100% paper. But the fourth one (not electrified), produces only books, but consumes too much paper: 381 boxes/month to make just 152 boxes/month of books. As the production says input 100% paper, output 80% news, 20% books, which totals 100%, I would expect 1:1 ratio for consumed paper and produced books, if no newspapers are printed.
That is due to the definition used for the cost of an output. Each output is assigned an equal share of the factory work done (in this case 50% of the consumption will go to each product). This means that 764 units (the amount of paper consumed if producing at 100%) will get split into 382 paper each output (764 / 2). Since books are produced at 20% of the production rate only 152.8 books will be produced (764 / 5). These match the values both you and I experience in game so the system is operating as intended.

This even makes logical sense. If you look at the product weights of newspaper (50 kg) and books (900 kg) each newspaper unit must obviously use less paper units than books. Sure the amount is not balanced perfectly (no way to adjust output weights, not that any pakset would use that feature immediately anyway) but it makes more sense than a newspaper using as much paper as a book. Eventually weight adjustment support could be added however for most cases equal weighting will work.

EDIT : In case it was missed, I did attach a JIT2 revision above which simplified some of the code a bit.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Václav on December 09, 2014, 05:26:52 PM
I enabled JIT2 from game - through (hidden) game setting dialogue - to test it.

It is good that JIT2 decreases energy consumption in some cases (under JIT1 net calls 1025MW, with JIT2 it calls only 593MW) but currently (with nightly 7408) it leads to break in production of all factories (it means that factories stop production even if they have no reason to do that).


I beg your pardon if I missed topic - and placed this question here, instead bugs board.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 09, 2014, 06:01:27 PM
Quote
It is good that JIT2 decreases energy consumption in some cases (under JIT1 net calls 1025MW, with JIT2 it calls only 593MW) but currently (with nightly 7408) it leads to break in production of all factories (it means that factories stop production even if they have no reason to do that).
I already explained why this is happening ages ago. Live conversion is not supported due to a lack of ability to re-compute critical values. You can convert a map following these simple steps.

When resuming the save so that JIT2 works you may need to load it once, change it back to JIT2 and then reload it so that it loads JIT2. I am not sure why this is necessary but it is along the lines of the constant not saving due to it being a nightly.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 09, 2014, 09:58:01 PM
@DrSuperGood: thanks for explanation. Printworks are now clear to me.

However the problem with steel seems so be very strange.
The map has just enough coal supply (+32t) as demanded by steel mills and power plant. It is the iron ore that is in short supply (-216t), so I would expect the stell mill to run out of ore, not coal. Maybe the deliveries are not smooth enough, I'll try to optimize.

I still see the hardware factory to run out of planks, not steel. Just watch the game longer. I think the problem is here: there is indeed shortage of steel (and abundace of planks). The demand counter for steel will sooner or later hit the maximum, while planks demand will be satisfied most of the time. But when stell counter is at max, planks counter will not grow either (and stay negative). Then when some batch of steel is available it is sent, but without the accompanying batch of planks. Thus the factory will always have some steel in input storage, but run out of planks. Perhaps the same is happening in the stell mill.

I think the logic when to stop orders needs some more thought. If I understand corectly now it is so that the orders stop if any input demand counter is at max. I would recommend to stop orders when the relevant input is full - it won't be for long - only until production is resumed. This will also quickly solve the previous issue with paper mill (only one input supplied). Oversupplies are discarded so the input cannot be overrun, and thus the time to recover will be short, and hopefully some stuff wil be still in transit.

The issue with supplying only one input on multiple factories was not an attempt to "cheat". But usually if having such a factory, you start by bulding supply links one by one. If the factory orders too much extra, you get money for nothing (and can use it to fund the remaining links.). Normally such link should stop deliveries quickly until you deliver the rest of required inputs.
Title: Re: Suggestion for JIT mechanics revision.
Post by: prissi on December 09, 2014, 11:05:47 PM
I did not submit the full warning patch, since some of the conversions are rather compiler dependent. I mainly look at MSVC, Dwachs uses CLang (I think) and then there is gcc is different flavours. Those three all differ broadly in fuzzyness, epcecially when conversions and signedness is concerned.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 10, 2014, 01:09:28 AM
Quote
It is the iron ore that is in short supply (-216t),
That means it is in oversupply. Negative demand means it has fully satisfied demand (it is demanding a negative amount or nothing at all). Such a large satisfaction number could be the result of a conversion artefact or an oversized shipment or even just a transient result as the system stabilizes.

Positive numbers mean there is a demand. If a steel mill has +32t coal that means it is trying to order an extra 32t of coal but is struggling to as the connected suppliers have none available for shipping.

The reason + represents demand and - over supply is that it makes sense next to the in-transit amount. Demand wants to try and add to the in-transit amount while oversupply appears to want to take away. I thought it would be more logical but I am hardly a UI expert. Then again it took me weeks to understand JIT0/1 simutrans industry metrics so I guess this is no surprise.

Quote
But when stell counter is at max, planks counter will not grow either (and stay negative). Then when some batch of steel is available it is sent, but without the accompanying batch of planks. Thus the factory will always have some steel in input storage, but run out of planks. Perhaps the same is happening in the stell mill.
It will always be negative because it is fully satisfied. The only time demand counters should be positive and larger than 10-20 is if there is insufficient supply. Both steel and planks appeared to be ordered at the correct rate matching each other.

Quote
I think the logic when to stop orders needs some more thought. If I understand corectly now it is so that the orders stop if any input demand counter is at max. I would recommend to stop orders when the relevant input is full - it won't be for long - only until production is resumed. This will also quickly solve the previous issue with paper mill (only one input supplied). Oversupplies are discarded so the input cannot be overrun, and thus the time to recover will be short, and hopefully some stuff wil be still in transit.
There are two order logics. Factories order all inputs in appropriate quantities to assure continuous production (+/- some rounding error). Eventually oversupply should self-correct due to the overflow mechanics with only the odd fractional unit of production being lost at most. It is completely pointless ordering more product if you cannot order more of another of the inputs since you need all inputs to produce something as a factory. If your proposal was implemented it would start the issue of a pipe-line collapse continiosuly and basically revert mechanics back to JIT1.

Consumers order each input separately and will do so all the time except when demand counter maxes out in which case consumption will be lost (as expected for insufficient supply).

Quote
The issue with supplying only one input on multiple factories was not an attempt to "cheat". But usually if having such a factory, you start by bulding supply links one by one. If the factory orders too much extra, you get money for nothing (and can use it to fund the remaining links.). Normally such link should stop deliveries quickly until you deliver the rest of required inputs.
Which is what they do if you start a map in JIT2 mode/found new industries in JIT2 mode where all demand counters default to max (so a factory cannot order more until all inputs are starting to be supplied). The issue with the paper mill is purely if a supply is suddenly cut or if you are using a converted map. In the case of cut supply it is obvious it will happen and hardly abuse able (you still make more money supplying everything). In the case of conversions (the paper mill) then it will eventually correct the demand counters so that the correct amount of product is arriving (none at all in your example as there were no chemicals being supplied).

Quote
I did not submit the full warning patch, since some of the conversions are rather compiler dependent. I mainly look at MSVC, Dwachs uses CLang (I think) and then there is gcc is different flavours. Those three all differ broadly in fuzzyness, epcecially when conversions and signedness is concerned.
The corrections were made for MSVC (an oldish version now, need to update...) and should not suffer fuzziness anymore than before since all conversions are explicitly defined. Currently they are only implicitly defined relying on the C++ standard to do them correctly (which normally it will since they are hardly ever in borderline conditions).

That said I did spot a number of alarming numeric range discrepancies with some functions. For example maintenance being returned as 64 bit but added to players in 32 bit which I would view as a specification error (no explicitly defined range for the numbers resulting in a single usable type for the situation). Another was that the storage type list only uses 8 bit indices yet the number of storage types is returned as 16 bits which implies from an interface point of view that there can be storage types not accessible (rage mismatch). However these are non-critical and should probably be fixed when/if the code is revised.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 10, 2014, 07:30:23 AM
The numbers I posted (-216 for iron ore and +32 for coal) are not demand counters, but sum of mine production - sum of stell mill (and power plant) production. So there really is shortage of iron ore and slight oversupply of coal, but the steel mill behaves as if it were the other way round.

For stopping orders, I wanted to say - stop orders only if the relevant demand counter is maxed out or if the relevant input is full. This will allow the factory to build some storage of all goods and be prepared for the moment where the last missing input is delivered. Demand counters should not affect other inputs. Now you order even if input is full, and stop ordering if other inputs demand is at maximum, that's IMHO wrong and causes the effects I experience. BTW the paper mill with chemicals was spawned long after converting the game to JIT2.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 10, 2014, 03:24:54 PM
Quote
The numbers I posted (-216 for iron ore and +32 for coal) are not demand counters, but sum of mine production - sum of stell mill (and power plant) production. So there really is shortage of iron ore and slight oversupply of coal, but the steel mill behaves as if it were the other way round.
It is not like that in the save you sent me. You are using 2 steel mills linked to the same 3 coal mines which are exclusive suppliers. Each coal mine produces exactly 1,124 tons/month of coal. This means your total coal supply is 3,372 tons/month.

One of your steel mills is un-powered and consumes 1,496 tons/month while the powered one consumes about 2,744 tons/month. Since both inputs are at 100% this means both require 4,240 tons/month of both coal and iron ore to operate fully. This means you are 868 tons/month (4240-3372) short on coal for 100% production.

However with iron ore you also have 3 mines but each produces 1,344 tons/month. This sums to 4,032 tons/month. When compared with the steel mill production this means that you are short 208 tons/month iron ore.

However since you are short 868 tons/month on coal the iron ore shortage does not mater since the coal will bottleneck steel production and the iron ore will remain satisfied (since it is not under supplied for the supply of coal provided).

Since the steel mill production is not equal (powered vs no power) but distribution is the powered steel mill will be bottlenecked more than the unpowered one. Running the game several months while I wrote this proved these results with the unpowered steel mill working at 100% while the powered steel mill is bottlenecked by coal as predicted above. Due to a lack of steel the hardware shop slowly builds up a steel demand which will limit production.

Anyway while running this test I noticed a pretty major error. At the beginning of the year the JIT2 flag is reset to JIT1! Yes new year starts and boom suddenly from JIT2 it is running in JIT1. This is not meant to happen.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 10, 2014, 03:57:06 PM
The powered steel mill is connected to one more coal mine on the other side of map

Sent using recycled electrons.

Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 10, 2014, 05:57:36 PM
Quote
The powered steel mill is connected to one more coal mine on the other side of map
That is the problem, it appears to not be connected to it. That coal mine is not sending anything to the steel mill so the steel mill is running out of coal. The map is so confusing it is hard to track down the links the coal should be taking.

The cause appears to be due to corrupted pakset information. The boats used to move the "bulk" from that mine do not exist properly so cannot move bulk. As such the coal mine is not connected to the steel mill. I will try and reload the save with the latest pak128 Britain but am getting assertion errors trying to do so with the latest nightly.

EDIT:

Sorry but I cannot load your map without errors. You are not using a standard build of pak128 Britain. As such every time I try to load it I get missing convoy errors which break the lines. Specifically 1.14 lacks the barge needed for the coal mine in the other side of the map and 1.15 lacks the trains used to ship the bulk between stations.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 10, 2014, 06:41:46 PM
I will post the paks i use when i get home. I have fixed the coal boat. There was a bug in dat file.

UPDATE: Necesary addons are here: https://uran.webstep.net/~vladki/simutrans/addons.zip
Other wise use the pak128.britain 1.14, the game does not work well with half-height versions.

Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 12, 2014, 04:30:54 AM
Quote
Other wise use the pak128.britain 1.14, the game does not work well with half-height versions.
I was using 1.14 but I lacked your addons. Now it works fine.

Quote
The numbers I posted (-216 for iron ore and +32 for coal) are not demand counters, but sum of mine production - sum of stell mill (and power plant) production. So there really is shortage of iron ore and slight oversupply of coal, but the steel mill behaves as if it were the other way round.
That is because you are delivering coal too inconsistently. Shipments larger than the input storage arrive causing an overflow (which becomes lost demand). The result is the steel mill working sub optimally due to an under supply of coal. In other words the steel mill is applying a penalty for bad delivery service (the huge goods train delivering to a tiny shop problem).

If you were to increase service frequency it should solve the problem. Alternatively it could be considered a pakset error (the input storages are too small for the type/production quantity).
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on December 12, 2014, 06:38:35 AM
Shipments larger than the input storage arrive causing an overflow (which becomes lost demand). The result is the steel mill working sub optimally due to an under supply of coal. In other words the steel mill is applying a penalty for bad delivery service (the huge goods train delivering to a tiny shop problem).

I still blame this on the consumer ordering too much, although I do see how complicated it is "making the consumer smart enough" to avoid this completely (in a transportation graph that can be almost infinitely complex).
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 12, 2014, 02:45:27 PM
Quote
I still blame this on the consumer ordering too much, although I do see how complicated it is "making the consumer smart enough" to avoid this completely (in a transportation graph that can be almost infinitely complex).
The most correct solution involving average travel time and the ability to regulate how much is unloaded from a stop would be to apply an artificial bottleneck on the last step loading. If you have a massive 1,000 unit transport loading into a 250 unit storage then it would allow only at most 250 units to be loaded in that transport. Based on average travel time it could then approximate how much can be loaded at the last stop so that when it arrives at the destination it will unlikely overflow. Transfer restrictions could do the rest. This is converging on what Experimental aims to do (although that is slightly buggy at times) where it uses the maximum in-transit value summed with current storage as a demand buffer without the ability to overflow storages.

However as you can see, that would be excessively complicated to implement.

JIT2 was not meant to be perfect, it was only intended to be a dramatic improvement over JIT1. In JIT2 in a properly functioning network it is impossible to get your coal power station buried in coal and then run dry because shipments fail to arrive in time before the storage depletes. JIT2 also implements dynamics that you need an appropriately sized convoy for deliveries, a concept that was not particularly emphasised before. The main issue is that some input storages might be too small since they were never really that important and balanced for JIT2 (the pak64+food marketplace is a good example where its tinned food storage is less than the minimum 2050 transporter available).
Title: Re: Suggestion for JIT mechanics revision.
Post by: sdog on December 13, 2014, 07:56:14 AM
JIT2 was not meant to be perfect, it was only intended to be a dramatic improvement over JIT1. In JIT2 in a properly functioning network it is impossible to get your coal power station buried in coal and then run dry because shipments fail to arrive in time before the storage depletes. JIT2 also implements dynamics that you need an appropriately sized convoy for deliveries, a concept that was not particularly emphasised before. The main issue is that some input storages might be too small since they were never really that important and balanced for JIT2 (the pak64+food marketplace is a good example where its tinned food storage is less than the minimum 2050 transporter available).
Could free station storage at the station the payload is delivered to counted towards input storage? Or would this remove the need to have appropriately sized convoys as a game element?
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on December 13, 2014, 08:48:00 AM
Could free station storage at the station the payload is delivered to counted towards input storage? Or would this remove the need to have appropriately sized convoys as a game element?

The biggest issue with this is that it is a new concept in the simulation. The way I understand it, stations are passive, they don't do anything, so industries would have to start polling stations for goods, rather than having it dumped into storage by the vehicles. Then there is the issue if stations are able to hold cargo not destined for another station, and adapting the GUI to show such things without confusing players.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 13, 2014, 04:45:39 PM
Quote
Could free station storage at the station the payload is delivered to counted towards input storage? Or would this remove the need to have appropriately sized convoys as a game element?
That is the concept of a warehouse and has been discussed before. Although I agree it would be useful, especially for unstable lines in the end it should make no difference. If a pakset is properly balanced so that input storage is suitable for available transport options then everything should be arriving just in time and work.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on December 13, 2014, 05:24:19 PM
If a pakset is properly balanced so that input storage is suitable for available transport options then everything should be arriving just in time and work.

It is rather silly that input storage is less than a single vehicle. I guess the thing here is that what can be put in a single vehicle in 2000, might be too much to have in storage in 1900. Especially for goods that are consumed at a slow rate, but go bad faster without refridgeration or similar. So it's another issue cause by lack of in-place industrial evolution in Simutrans.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 14, 2014, 12:04:32 AM
Quote
It is rather silly that input storage is less than a single vehicle. I guess the thing here is that what can be put in a single vehicle in 2000, might be too much to have in storage in 1900. Especially for goods that are consumed at a slow rate, but go bad faster without refridgeration or similar. So it's another issue cause by lack of in-place industrial evolution in Simutrans.
It is perfectly fine that in the old years you can expect factories to have considerably larger storages compared with the shipment unit. In the old days some large industries had enough material stored to operate for several months without shipments. It is only a very recent trend with the rise of computers that a bare minimum is kept in storage on site. The most notable example would be a car factory in the 1950s would have storage for many months yet a car factory in 2014 stores only enough for a few days of operation.

The main issue is that the storage sizes of some places are unrealistically small. The best example is the pak64 + food marketplace which stores only 25 canned food at times. The smallest you can ship boxed goods in 2050 is 40 units odd. Yes the building is old but it consumes a fair amount monthly. Raising it to 80 units storage is a lot more acceptable since it gives a reasonable buffer time for trucks. Yes the building is old but its rate of consumption is not and if anything such a high consuming old building should easily be able to accept the unloading of a single truck at least.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on December 14, 2014, 09:54:17 AM
The main issue is that the storage sizes of some places are unrealistically small. The best example is the pak64 + food marketplace which stores only 25 canned food at times. The smallest you can ship boxed goods in 2050 is 40 units odd. Yes the building is old but it consumes a fair amount monthly. Raising it to 80 units storage is a lot more acceptable since it gives a reasonable buffer time for trucks. Yes the building is old but its rate of consumption is not and if anything such a high consuming old building should easily be able to accept the unloading of a single truck at least.

Since it's canned food in particular, it's a bit odd, given how it won't rot overnight, or even for months. Had it been fresh meat, then having a storage capacity much less than monthly consumption would make sense before about 1950.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 14, 2014, 01:41:04 PM
Quote
Since it's canned food in particular, it's a bit odd, given how it won't rot overnight, or even for months. Had it been fresh meat, then having a storage capacity much less than monthly consumption would make sense before about 1950.
The reasoning was probably because it was consumed slower. The result was the following definition...
Code: [Select]
InputGood[0]=fish
InputCapacity[0]=50
InputSupplier[0]=0
InputFactor[0]=50
InputGood[1]=flour
InputCapacity[1]=50
InputSupplier[1]=0
InputFactor[1]=50
InputGood[2]=beer
InputCapacity[2]=200
InputSupplier[2]=0
InputFactor[2]=100
InputGood[3]=canned_food
InputCapacity[3]=25
InputSupplier[3]=0
InputFactor[3]=25

In the case of JIT2 these values would be more appropriate for the building.
Code: [Select]
InputGood[0]=fish
InputCapacity[0]=120
InputSupplier[0]=0
InputFactor[0]=50
InputGood[1]=flour
InputCapacity[1]=120
InputSupplier[1]=0
InputFactor[1]=50
InputGood[2]=beer
InputCapacity[2]=160
InputSupplier[2]=0
InputFactor[2]=100
InputGood[3]=canned_food
InputCapacity[3]=80
InputSupplier[3]=0
InputFactor[3]=25
(this is an example, not a change suggestion)

Note that this was done with eye only. Better results might be achieved by careful planning and possibly even a formula. The idea is for the input capacity to have at least enough to receive 1 "appropriately" sized shipment and use that as the starting point. For the market place this a truck of food. For something like a steel mill a 4-6+ tile long bulk train would be appropriate. You then multiply it by some amount to add robustness to the supply flow (to cope with variances of signals/traffic etc). Goods that are consumed slowly (canned food in marketplace) can be multiplied less since each unit represents more variance in time. Goods that are consumed very quickly (beer in marketplace) need to be multiplied by more as each unit of storage represents less resilience time.

The amount of resilience time desired depends on how "difficult" you want supplying an industry to be. Something like a coal or oil fired power station could be given a large resilience to be realistic (since they are designed to never stop working, possibly even running months without shipments). On the other hand the above marketplace could be given very little resilience to represent spoilage (especially of frozen goods!) so it requires a very efficient and precise network to deliver.

Currently many industries in pak64 have quite small input storages so may struggle to operate 100% of the time under JIT2 mechanics (the marketplace being the example which is impossible). On the other hand industries in pak128 have massive input storages so are very difficult to not supply them properly (a cement factory holds 8,000 tons of coal, enough for several standard months of operation so can accept multiple of any shipment size even in unpredictable transport times).
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on December 14, 2014, 02:29:03 PM
Let's say that a marketplace sells 1 barrel of fresh fish per day irrespective of the year (constant number of customers with constant taste for fish). That means that it must have storage for 1 barrel of fresh fish in 1900. At that time, one horse-drawn carrige, carrying on barrell of fish arrives from the fishery each day. But by 1950, refridgerated storage is available, so fresh fish can be kept in storage longer. So you get trucks carrying 10 barrels of fish, and the horse drawn carriges become obsolete, not just because of lesser capacity, but also because of their below average speed causing traffic jams.

The only ways to integrate the new trucks with the old marketplace, is either to give the marketplace an unrealistically high storage capacity in the past, or let the input storage overflow in the present (and technically also in the past, but the low capacity of the vehicles then makes this difficult to achive). The former is impossible if the newer vehicles come from an add-on.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 14, 2014, 02:42:54 PM
Pak64 should not really be played re 1930 odd and trucks should always be present.

The issue you describe is one of the industries not scaling over time. The Marketplace in 2050 (end game time) should be considerably different from that found in 1900 with different demand/storage amounts.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on December 14, 2014, 02:58:21 PM
The issue you describe is one of the industries not scaling over time. The Marketplace in 2050 (end game time) should be considerably different from that found in 1900 with different demand/storage amounts.

That's what I wrote several posts ago.

Pak64 should not really be played re 1930 odd and trucks should always be present.

I like starting out in the second half of the 19th century, or there is hardly any evolution in vehicles at all. It peaks at about 1950, and by late 1970 it's pretty much over. (Which seems true to life with regards to capacity and speed.) Road transport before 1930 is awful, but then I find road vehicles a necessary evil later on as well. (Too much to transport, too small vehicles and too small capacity at stops.)
Title: Re: Suggestion for JIT mechanics revision.
Post by: Václav on December 14, 2014, 05:52:15 PM
I found that with JIT2 powerplants do not produce power on the same level at all times - even if they have sufficient amount input good to work. Their production alterates between 100 and 99.

And trains' max. speed alterates too (again) - between moment max. speed and moment max. speed +1.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 14, 2014, 06:35:55 PM
Quote
Their production alterates between 100 and 99.
That is because of rounding. Their power output is scaled based on their work done so unfortunately rounding applies. Some ticks they consume slightly more than others.

Quote
And trains' max. speed alterates too (again) - between moment max. speed and moment max. speed +1.
Nothing to do with JIT2. Only factory code was changed with this.

Someone else altered those calculations recently so it is probably an error related to that.
Title: Re: Suggestion for JIT mechanics revision.
Post by: sdog on December 15, 2014, 03:05:13 AM
[...] The idea is for the input capacity to have at least enough to receive 1 "appropriately" sized shipment and use that as the starting point. For the market place this a truck of food.
It might be interesting to re-visit the old idea of having a very number of small end consumers. Art tested a few concepts, i think with pak britain. A few years ago. With large lorries and small consumers one would have to have a warehouse and send convoys to more than one consumer.

Quote
[...] For something like a steel mill a 4-6+ tile long bulk train would be appropriate. [...]
That seems to be quite low, while i don't know pak64 very well, in pak128 I usually start with 6 to 8 tile bulk trains in 1930, and run 12 tile trains as soon as Diesel-electics become available. Especially when it is for a long distance transport over 2000 tiles.

Quote
[...]
You then multiply it by some amount to add robustness to the supply flow (to cope with variances of signals/traffic etc). Goods that are consumed slowly (canned food in marketplace) can be multiplied less since each unit represents more variance in time. Goods that are consumed very quickly (beer in marketplace) need to be multiplied by more as each unit of storage represents less resilience time.

The amount of resilience time desired depends on how "difficult" you want supplying an industry to be. Something like a coal or oil fired power station could be given a large resilience to be realistic (since they are designed to never stop working, possibly even running months without shipments). On the other hand the above marketplace could be given very little resilience to represent spoilage (especially of frozen goods!) so it requires a very efficient and precise network to deliver.
This nicely summarizes the requirements on a pakset to be used with JIT2.

Quote
Currently many industries in pak64 have quite small input storages so may struggle to operate 100% of the time under JIT2 mechanics (the marketplace being the example which is impossible). On the other hand industries in pak128 have massive input storages so are very difficult to not supply them properly (a cement factory holds 8,000 tons of coal, enough for several standard months of operation so can accept multiple of any shipment size even in unpredictable transport times).
The cement mill i knew had very large coal storage it did not use more than 10% of. I supppose in the past they stored much more, buying when prices were low, or supply was easy. It is hard to guess how much could have been piled there, but i expect it would have been at least the equivalent of a dozen trains. Storage was very cheap, only a large level surface and some walls were needed. There was hardly any storage for limestone though. It was just delivered the 2 km from the quarry.

I think the last aspect raises an old question, that became acute with JIT 2. Some industries are always in the vicinity of the required resources. This limits the access to other resources though. (eg since cement mills are near to limestone, they are always far away from coal). With JiT2 more realistic industry placement algorithm becomes desirable. (yet not necessarily feasible).
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 15, 2014, 04:06:56 AM
Quote
That seems to be quite low, while i don't know pak64 very well, in pak128 I usually start with 6 to 8 tile bulk trains in 1930, and run 12 tile trains as soon as Diesel-electics become available. Especially when it is for a long distance transport over 2000 tiles.
Pak128 is balanced slightly differently. In 1930-1960 my calculations were that it was most economical to ship in multiples of 3. In 2050 you need 15+ tile long trains due to the low density of bulk cars (the post 2000 one is physically larger) and the massive power of engines (anything less is a waste). You can also ship at pretty high speeds using the more powerful engines. That said the break even point is usually 1-2 cars/trucks of bulk so the rest is revenue to offset the high maintenance of pak128.

Pak64 is balanced slightly differently. Unlike pak128 where convoy running cost is very low and the maintenance high, pak64 has low maintenance and high running costs. To put it in perspective you will spend >50% of the income shipping some goods with some goods not being viable for shipment via some means at all (you cannot ship paper at a reasonable speed by road without huge losses post 2000). As such throwing power to move goods fast is often not that profitable (or was, its changed recently since you no longer have that final massive wave of speed bonus penalty). To do this there is a neat little 60km/h engine that is by far the most economical way to move goods like bulk. That engine generally works out convoys in lengths of multiple of 3 size. As such for supplying some industries it might be appropriate to have a convoy length in the form of a multiple of 3 with 12 only being for very high demand industries. It is also worth noting that most trains max out at 12 tiles (2 cars/trucks/engines per tile) with the only exception being car transporters and the one diesel engine which is bigger than 1/2 tiles. In pak128 many goods could produce convoys longer than 12 station tiles at the end with the biggest being ~20 tiles so it is not a good metric for balance.

Where as something like steel mill coal could arrive in train length 6 (since steel mills use less coal), iron ore could be expected to arrive in 12 length convoys (since a steel mill can consumes several thousand units of iron ore a standard month).

Quote
I think the last aspect raises an old question, that became acute with JIT 2. Some industries are always in the vicinity of the required resources. This limits the access to other resources though. (eg since cement mills are near to limestone, they are always far away from coal). With JiT2 more realistic industry placement algorithm becomes desirable. (yet not necessarily feasible).
That would have to be part of a general economic revision. Something like a cement mill might be placed nearby a town (people need to work but not get the pollution) near one of its inputs (distance from the other is irrelevant and not considered). Certain cities could be given types such as industrial or commercial where industrial cities get processing industries (food processor, concrete works, builders yards etc) and commercial cities more consumers (department store, supermarket, etc).

However before such a thing is considered I would recommend revising the entire city growth mechanic based on what I mentioned elsewhere. Founding more cities should not make more total growth and more total industries. Industries should come at a "levelling up" rate based on global population. Total population growth should be based on total transport metrics and starting city number and spread across all towns based on their transport metric weight compared with the total. Industries need a cost weight for generation so that a tiny green energy facility uses "less population up" than a concrete chain.

I have had games turn into industry deserts as all that ever spawned were solar farms due to a lack of initial power placement. Something that really needs addressing...
Title: Re: Suggestion for JIT mechanics revision.
Post by: Ters on December 15, 2014, 06:40:11 AM
That seems to be quite low, while i don't know pak64 very well, in pak128 I usually start with 6 to 8 tile bulk trains in 1930, and run 12 tile trains as soon as Diesel-electics become available. Especially when it is for a long distance transport over 2000 tiles.

In pak64, I find trains longer than 6-8 tiles overly long compared to the world. (Although very short compared to the locomotive, at least for freight trains.) Power, weight and capacity seems balanced for this as well, but I don't know if that's because the past and present creators also see it that way, or if the thing with 6 tile long trains is just copied from Transport Tycoon. (The maker of Transport Tycoon might himself have found that 6 tiles fits the scale of the world best.)
Title: Re: Suggestion for JIT mechanics revision.
Post by: Vladki on December 15, 2014, 07:40:06 AM
Supplying more small markets with one truck is a challenge that is much easier with jit2 than before. But you may find out that the truck runs mostly half empty thus never making profit.
Without jit2 it will mostly load goods for one of the markets, run empty along the others. When one market is buried for several months, then another market is chosen. Some markets may never get anything. But with jit2 every market will get its goods on board.

Sent using recycled electrons.

Title: Re: Suggestion for JIT mechanics revision.
Post by: Václav on December 16, 2014, 07:50:38 AM
I found following:

Some factories don't show accepted goods.

I have lines that bring coal to powerplants - there it is right. Coal power plants show how much coal lefts to storage would be empty.

Also I have line that brings electronics to supermarket. Factory stops production because supermarket has full storage (because train brings more electronics than is needed - for long time), but supermarket does not show how much electronics lefts. So I cannot see how much time will pass until next ride of train.


I found that for short time that supermarket had a full storage - but there was only such amount of goods that storage can contain - but where is the rest? (it means if train brings 759 - and storage of supermarket may contain 15 - where is 745 units?) And train earnt as money as it should (currently about 840 thousand). It is very strange.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 16, 2014, 04:31:35 PM
Quote
So I cannot see how much time will pass until next ride of train.
I need a screenshot as it should be there. The form used is the following...

current amount/max amount : in-transit(+/-demand buffer)

current amount is always less than or equal to max amount and never less than 0.
in-transit has no reasonable limit and is always natural.
when demand buffer is greater than or equal to 0 goods will be ordered and in under supplied situations it will max out at ~max amount.

Quote
I found that for short time that supermarket had a full storage - but there was only such amount of goods that storage can contain - but where is the rest? (it means if train brings 759 - and storage of supermarket may contain 15 - where is 745 units?) And train earnt as money as it should (currently about 840 thousand). It is very strange.
Overflowing storages is not allowed in JIT2. A tiny shop just cannot hold more than 15 units at a time as their property is not big enough. One has to imagine that the rest of the 759 units is thrown away or sold elsewhere and to represent that the shop gets a demand penalty equal to the overflow. This mechanic was necessary so that a demand penalty could be created based on the amount overflowed. If you want something to work at 100% you have to respect the fact that you cannot use a 12 tile long train to deliver to a tiny corner shop.

That said many industries were balanced around JIT1 so do not have appropriate max amounts. This is something for pakset authors to consider once JIT2 gains popularity (if it does). Every input should at least be able to receive 1 full load from a end-game convoy. There should also be some buffer amount based on rate of consumption to allow for variances in delivery time.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Václav on December 16, 2014, 04:56:13 PM
Overflowing storages is not allowed in JIT2.
Ha ... Thanks. But on other side, when you need to transport something from one end of map to other end, you often load long train of common length (currently I use ten tiles long). Because train with only one waggon does not have a sense.

So, here is space for two changes.
1. Train will wait until it is unloaded (it means that goods are unloaded per parts - as they can go into shop)
2. Overflowing goods are unloaded to station - and there they wait for next transportation or usage by shop (similar to what I requested some time ago (http://forum.simutrans.com/index.php?topic=12790.0)). And storage of shop or factory should be empty before next supply.
Title: Re: Suggestion for JIT mechanics revision.
Post by: DrSuperGood on December 16, 2014, 05:33:00 PM
Quote
Ha ... Thanks. But on other side, when you need to transport something from one end of map to other end, you often load long train of common length (currently I use ten tiles long). Because train with only one waggon does not have a sense.
Yup, that is why you use a truck instead of a train or boat.
Title: Re: Suggestion for JIT mechanics revision.
Post by: Václav on December 16, 2014, 06:14:07 PM
Truck is dead. Long live the truck.  :)


But sometimes, and mostly on very long routes - where it is possible to release many trucks, it is a very difficult to regulate trucks in such way that they would leave loading bay alone.

And then, again, it would be needed if overflowing goods would be unloaded to station - from which goods could be given to shop or factory with delay.