News:

SimuTranslator
Make Simutrans speak your language.

Suggestion for JIT mechanics revision.

Started by DrSuperGood, July 06, 2014, 01:41:01 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

DrSuperGood

I cannot confirm if it is completely bug free as the changes made by this are pretty big (this is far bigger than the maximum in-transit change and that had many errors that made it into 120RC). If you can produce an error in a re-creatable way (from a save) then please explain how and I will look into it as soon as possible.

What you describe sounds like a status cache integrity failure. To prevent excessive polling of outputs I cache their status (how many are inactive) so that the corresponding new logic can be ignored (no need to order anything if all orders are full, or no need to produce anything if all inputs are empty etc) so a tick should execute a tiny bit faster, especially for disconnected industries (which often a large number are on servers). However if there is a logical error with the increment/decrement of the cache it will break quite badly and what you mention could occur.

I had very limited test cases. Using Pak64 I had power supplied to a combination of steel industry (single output producer) and oil industry (multiple output producer). I also checked the operation of consumers such as the coal power station (power plant), updraft generator (I thought it was "solar tower" according to the documentation 8 years ago odd, this is a constant power producer) and the construction wholesaler (multiple consumer). During the tests I also picked up an underflow error with power nets refunding a negative amount of demand if over supplying (illogical and error prone as power demand is signed so an underflow occurs).There were many small logical errors that I solved with my code as well so it is possible one such error has escaped un-detected however I do need a test case to reproduce it for a solution.

I ran a simple power station test on pak128 to check operation since there actual power factors are defined and everything appeared to be functioning as expected.

To check legacy support I loaded 3 different server games from Fifty over the last year. These games were pretty massive with probably well over 100 industries (pak64) and companies making hundreds of millions and the results appeared to be as expected with no noticeable deviation for someone used to old mechanics. I do know for a fact that results will not be 1:1 however but this all comes down to rounding with the new implementation probably producing more accurate results due to factories operating in a normalized mode.

Ters

Quote from: prissi on October 18, 2014, 09:01:24 PM
Did somebody other tested this? I think it is very interesting, but in the last two months I lacked serious time for in-depth testing.

In order to test it, I need to be able to switch it on (maybe even off as well) for existing games. Being unable to do so will also make it less likely that players will use it, depending on how often players start new games. (I keep playing the same map for years.)

Vladki

#107
You may try what I suggest in the long distance patch thread: add "dialog_tool[27]=,~" in menuconf.tab.
Then you will be able to change to JIT2, but you may run into the same bugs as I did.

I have managed to find a reproducible bug. Look at https://uran.webstep.net/~vladki/simutrans/, download the autosave07.sve and addons.zip - it is my savegame converted to jit2 - run it with pak128.Britain 1.14 (last single-height stable) and some recent nightly.
It will be loaded close to two stell mills, one wil be quickly recovering from buggy undersupply (demand -200 or so). Then you can watch the steel demanding industries slowly restart production and waiting steel trains to leave the station. When you load the game there are a few trains waiting for cargo, but when the steel is produced at full, at most one train shall be waiting for cargo.

Then let the game running for a while and move to the other end of map - find car dealer in Christminster. Normally it should be selling some 260 cars/month. (max with power boost is 320). It will slowly recover production. But the trains can deliver 60 cars at once, while the input storage of this shop is only 53. So it happens from time to time that the 7 extra cars are thrown away, and deduced from demand. I am not sure if that is the trigger for the bug. If you run the game until august around 20:00 (in game time), the demand buffer of this car dealer gets quickly satisfied by the car factory and drops to -4, never to rise again. There are still some cars on the way, but they are consumed without increasing the demand counter. It drops even more as some of the supplies overcrowd the dealer. On september 3:00 the last delivery happens, and car dealer is not ordering any more.

With fast forward I got the car dealer broken at august between 12:00 and 15:00. Tried 4 times, and always got the bug, maybe with different timing.

Ters

Quote from: Vladki on October 19, 2014, 09:44:28 AM
You may try what I suggest in the long distance patch thread: add "dialog_tool[27]=,~" in menuconf.tab.
Then you will be able to change to JIT2, but you may run into the same bugs as I did.

Those "bugs" were what I was talking about. They would cause my test to fail. (They are not really bugs, because they are a known effect of doing somewhing JIT2 doesn't support.)

DrSuperGood

#109
QuoteI have managed to find a reproducible bug. Look at https://uran.webstep.net/~vladki/simutrans/, download the autosave07.sve and addons.zip - it is my savegame converted to jit2 - run it with pak128.Britain 1.14 (last single-height stable) and some recent nightly.
It will be loaded close to two stell mills, one wil be quickly recovering from buggy undersupply (demand -200 or so). Then you can watch the steel demanding industries slowly restart production and waiting steel trains to leave the station. When you load the game there are a few trains waiting for cargo, but when the steel is produced at full, at most one train shall be waiting for cargo.

Then let the game running for a while and move to the other end of map - find car dealer in Christminster. Normally it should be selling some 260 cars/month. (max with power boost is 320). It will slowly recover production. But the trains can deliver 60 cars at once, while the input storage of this shop is only 53. So it happens from time to time that the 7 extra cars are thrown away, and deduced from demand. I am not sure if that is the trigger for the bug. If you run the game until august around 20:00 (in game time), the demand buffer of this car dealer gets quickly satisfied by the car factory and drops to -4, never to rise again. There are still some cars on the way, but they are consumed without increasing the demand counter. It drops even more as some of the supplies overcrowd the dealer. On september 3:00 the last delivery happens, and car dealer is not ordering any more.
Found the bug. In the case of a shipment arriving while the demand counter was maxed out (inactive) it would not re-activate it. The logic for ordering would not re-activate it because it considered it already active. Basically your typical "invalid state" problem sigh. Saving/loading fixed this as the activity cache is rebuilt on load (and will correctly detect the demand as being active).

I have a fix for this that I will publish on this thread later tonight (patch file and executable). I will also update the patch to the latest SVN nightly version making sure all conflicts are resolved. I will not post a new message (unless someone else posts) so please check back later.

QuoteThose "bugs" were what I was talking about. They would cause my test to fail. (They are not really bugs, because they are a known effect of doing somewhing JIT2 doesn't support.)
I am open to suggestions how to do a conversion more neatly.

prissi

I think I will try to relase a stable (or rc) this week and then commit it afterwards. Just if there is some effect with server games, there is a somewhat stable (which is lacking at the moment).

DrSuperGood

Ok I have made some revisions to it. The latest patch file (should be compatible with SVN) is attached below. Here is a pre-built (using nightly) executable for Windows targeting Visual C++ (version 10) which should work with any nightly installation of simutrans.

The following notable changes were made.

  • Fixed a logic error with activity caching causing demand counters to freeze. The error occurred when an input had a maxed demand counter and received a shipment which resulted in storage overflow and the demand counter to fall below max.
  • Added a meaning for maximum in-transit in JIT2. It will now clamp the minimum and maximum demand buffer sizes on load to some percentage of the maximum storage. This is for use during conversion from JIT1 to 2 which is a buggy process. By setting maximum in-transit percentage to some small value and then re-loading it will make sure all demand buffers are reasonable and everything should work. It is recommended to set the percentage to 0 for new games and playing outside of the conversion process.
  • Fixed an error with factory list logic showing nonsense values. This was caused by decoupled logic between the list and the actual factory operation.

QuoteI think I will try to relase a stable (or rc) this week and then commit it afterwards. Just if there is some effect with server games, there is a somewhat stable (which is lacking at the moment).
Sounds good to me. People could then always try it out when it filters down to the nightly builds.

Vladki

HI, I have been playing for some time with jit2r2, and so far so good - no errors. And it is a converted game. If anybody wants to try on linux, here is 32-bit binary: https://uran.webstep.net/~vladki/simutrans/sim-gcc4-jit2r2 based on nigthly r7348.

prissi

This weekend I will relase 120.0.1 and then the new JIT can be incorporated. Just waiting for the nightly mac builds.

DrSuperGood

Ah excellent. It will be great when people can play it in the nightly builds.

Yona-TYT


Quote from: prissi on November 16, 2014, 12:08:16 AM
This weekend I will relase 120.0.1 and then the new JIT can be incorporated. Just waiting for the nightly mac builds.


You refer to the r7373 ??

An_dz

No, he refers to whatever revision when mac builds get back working, then prissi will release and JIT will be incorporated.

DrSuperGood

QuoteNo, he refers to whatever revision when mac builds get back working, then prissi will release and JIT will be incorporated.
They are not working? The nightly builds appear to be building fine for Windows, Mac and Linux. In fact the last nightly for all 3 was built earlier today. Or are you saying that there is an error in the Mac builds currently preventing them from running (not building as that appears to be working fine)?

Ters

Quote from: DrSuperGood on November 16, 2014, 05:34:10 PM
In fact the last nightly for all 3 was built earlier today.

Prissi's comment was almost one and a half hour earlier still.

prissi

Sorry, I only have computer time after 20 o'clock (GMT). So the stable has been released. I have to do some other stuff (like the Linux makeobj) and also try to get the haiku build in order and then JIT2 will be in the nightlies.

An_dz

Quote from: DrSuperGood on November 16, 2014, 05:34:10 PM
are you saying that there is an error in the Mac builds currently preventing them from running (not building as that appears to be working fine)?
I have chosen the wrong words, to me prissi was talking about waiting them to be built this last time before releasing as Mac is probably the last to be built.

Anyway this discussion is useless as the possible problem (that never existed) was fixed as prissi has released a new version and the point was discussing the revision number. All I have read was: "waiting for the mac nightly to build".

prissi

#121
Back on topic. There were lots of single onliner if and so on. Cleaned up the code a little.

The ability to save for older version was broken. Hopefully this works again now.

Why does the new version only display half of the max and storage values compared to a similar game for the old version. It seems to me rather a display bug. Please check info_prod and so on.

Some questions remains:
Should it be enough to determine the type of a factory once instead on every steps?
For single and multiple consumer/factory procuder: COuld this logic be not joined. The overhead seems rather small compared to the huge repetitive case construct.

Despite the display(?) bug submitted in 7378

DrSuperGood

QuoteShould it be enough to determine the type of a factory once instead on every steps?
Yes, it should actually go as part of the factory besche (since it is not state as it does not vary during factory life) and could even be something pakset creators can configure. This would allow some special logic for certain factory types with unusual behaviour.

QuoteFor single and multiple consumer/factory procuder: COuld this logic be not joined. The overhead seems rather small compared to the huge repetitive case construct.
Yes they could be removed and use the multiple logic instead. I was not sure about the trade offs having or not having them would bring.

prissi

Sorry, editing took so long. See about the error with amount at factory. Maybe scaling of some factors had some side effect in displaying.

I think in order to get compatibility the type needs to be either calculated onload of the besch of taken directly from it. SO I would suggest to move this logic to factory_reader. (Or I can do it myself, if you like).

About joining logic. THat is much better, because if there are typos/errors, then you do not have to maintain almost the same code twice. Reusing is really helpful.

DrSuperGood

QuoteSorry, editing took so long. See about the error with amount at factory. Maybe scaling of some factors had some side effect in displaying.
I thought I corrected all the display errors.

QuoteI think in order to get compatibility the type needs to be either calculated onload of the besch of taken directly from it. SO I would suggest to move this logic to factory_reader. (Or I can do it myself, if you like).
I avoided touching those parts because it would couple the change to yet more parts of the game and I was not sure how they work and what could be changed with them. Making an already big change even bigger can lead to other problems so I wanted to make sure what I had was working well (which it appears to be from tests).

In any case this is not particularly critical for release and could be a future update. Yes it will affect performance but even on maps with hundreds of industries other factors probably are affecting performance more at the moment.

QuoteAbout joining logic. THat is much better, because if there are typos/errors, then you do not have to maintain almost the same code twice. Reusing is really helpful.
In theory some of the operations could be turned into function calls for even more re-use. I just was not sure where to break stuff apart.

prissi

The display seems off; upon loading an old game (i.e. the pak 64 steel mill) then the number in ore are half and coal are twice the value. YOu see that when a truck with 42 tons of coal arrives the dispolay increases to 84 tons.

The production itself is synchron, and the output seeems similar. ONly the numbers of the screen are wrong. Maybe it has something to do with the consumption percentage.

I removed the code which determines the type of factory to the creation (baue) of factories (which is also used for loading).

DrSuperGood

QuoteThe display seems off; upon loading an old game (i.e. the pak 64 steel mill) then the number in ore are half and coal are twice the value. YOu see that when a truck with 42 tons of coal arrives the dispolay increases to 84 tons.
Ops... Forgot to update the old display logic for the new way that production units are handled. When running in JIT2 mode it will correctly display the amounts but in 0/JIT1 it will not scale them appropriately.

Example, to get current amount this is used.

(uint32)(0.5 + (double)ausgang[index].menge * (double)(besch->get_produkt(index)->get_faktor()) / (double)(1<<(fabrik_t::precision_bits + DEFAULT_PRODUCTION_FACTOR_BITS))),


The old logic is still using this.

(sint32)(0.5+ausgang[index].menge / (double)(1<<fabrik_t::precision_bits)),

So it is not factoring in the scale factor correction.

I will try writing a patch for it in a little while. I am terribly sorry for the oversight.

Vladki

Do I understand it right that the JIT2 is in latest nightlies?

If so, then there are some bugs - most important - the value which  JIT is used is not saved properly - I save with 2, and load 1.

Save from previous games (with patched JIT2) loaded fine with jit=2, but there was another problem with power - factories did not use the power even if available. SO I tried to fix by save/reload - power was consumed but could not switch to jit2.

Ters

Quote from: Vladki on November 26, 2014, 09:48:53 PM
Do I understand it right that the JIT2 is in latest nightlies?

Since r7378 by the looks of it.

prissi

That cannot be saved right now without bumping a savegame. You can start your game with "-server" on the commandline, to save this setting.

Usually I do not bump the savegame until thing are a little tested at least. Otherwise people will lose their games.

But next nightly will remember when loading an old game from your simuconf.tab setting whether you want of old new enforcements.

DrSuperGood

Here is a patch to fix the legacy factory information display to show the adjusted units. The units shown for factories with factors not 100% was being incorrectly displayed for the reasons mentioned earlier. This patch corrects that by adjusting with the factor in the case of running in legacy JIT0/1 modes.

As an additional change I updated the logic for generating the output text to use fixed point instead of double precision floating points. Not only does this remove possible platform differences to display results and mathematical error but also should perform slightly better. The performance gain is probably trivial (certainly not something you will ever see) but the precision and platform independence I would say is worth the change.

prissi


Vladki

Hi I have noticed a bug:

I have a newly spawned paper factory needing woodchips and chemicals. I started to deliver woodchips first, and filled the input storage, and the factory is still ordering more woodchips - paying for them and discarding them. The factory is not producing because it has no chemicals. Save (pak128.Britain): https://uran.webstep.net/~vladki/simutrans/britanie.sve. Look for paper mill at (90,3).

DrSuperGood

I cannot load the save using the nightly (r7389) and nightly version of pak128.Britain. It crashes during loading (the process hangs).

Vladki

#134
Tried with latest r7389 and updated the save - loads well. Just I play with old pakBritian (1.14) - without double/half slopes.
The game shlould load near car factory which currently runs out of plastic, so you can see the bug.

Another thing is that power usage is somewhot flaky - many factories do not use power even if they could. E.g. refinery and one of the steel mills supplying the car factory - they have undergroud transformers, so you can't see them, but other factories with visible transformers are broken too.

One more issue: IIRC the new factory rules for multi-output facories say that if some products are not produced, then less inputs are consumed. I have a printworks - input 100% paper, output 80% news + 20 % books. I produce only books, so I would expect that paper consumption would drop to 20%, but it is at cca 50%. (I know that this factory logic change was subject of discussion, with oil refinery being a bad example. But printworks are perfect fit for the new logic - 1ton of paper can be used to make either 1 ton of books or 1 ton of new, or 1/2 ton of both...)

DrSuperGood

#135
The issue was with the way you linked the save. It became difficult to download completely as it was being managed as a webpage rather than as a file. I have now managed to download the entire save file and load it and am investigating.

EDIT:
QuoteLook for paper mill at (90,3).
The map has no such position. It is not in bounds. I tried (3,90) and that was a tunnel.

The only paper mill I can find on the map that uses chemicals is at (27,89) and it is being supplied both with chemicals and with woodpulp further more the map is not using JIT2 so demand buffers are not enabled (it is using maximum in-transit instead).

The car factory is out of plastic but appears to be ordering correctly in response to the classic JIT1 ordering behaviour. It also appears they are consuming power as expected with the classing JIT1 behaviour.

I will try and force it over to JIT2 (some how) and see what happens.

Vladki

#136
I had problems saving the game with jit2 - it always returned back to JIT1. Setting jit2 in simuconf.tab ad using latest nightly helped.

Coordinates depend on rotation. But yes you found the right paper mill anyway - the only one that needs chemicals. I have supplied it for a while but stopped the supplies so it will eventually run out of chems.

DrSuperGood

#137
The papermill is working correctly. Once the order buffer for chemicals is full (matches the storage amount) it stops ordering pulpwood as expected. The issue why it over-ordered so much is due to the demand counter for chemicals starting at 0 and taking a considerable amount of time to fill up. It also had a fair amount in storage already which gets converted to over-order due to the mechanics of JIT2. The industry has several months of chemicals stored in it so it does over-order quite a bit. However without supplying it with more chemicals it does stop ordering pulpwood eventually. The main issue is that the effective months of storage between pulpwood and chemicals is vastly different so the chemical demand building up causes the pulpwood storage to overflow. If the storages were even (say both were 2 months of production) then you may find that they do not overflow. This is just a qwirk with how JIT2 works and something that pakset authors might want to consider in future revisions.

I am guessing there could be better logic to handle this specific case (such as setting the demand buffer limit for all goods to the minimum input storage for factories) but that would complicate things a bit more. However it still functions correctly if chemicals and pulpwood continuously flow. In the end you will still ship more to the factory if you keep both steadily supplied instead of trying to overflow it with pulpwood.

The issue with power is very strange as the original patch version that prissi merged from does not suffer the power issue (the refinery consumes power as expected oscillating from 99% to 100% due to the filter used for power distribution and varying load). This means that the error was introduced during the merger. As a rough guess it could be that when the boost logic is determined it fails to select BL_POWER so it thinks the industries do not need power (so will not run the boost logic to do so).

EDIT: I have found the cause of the power problem.
Power requirement is not computed when the factory is built. One would imagine it was and is if you build a factory in-game however during loading it is only computed at a later step after all factories are built from the save file (before linking occurs). As such when setting up the logic to use it assumes the factory power demand to be whatever is in the memory at that time (why some factories work while others do not).

EDIT2: Here is the patch to fix the power logic. I tested your map and the refinery was correctly consuming power. It works by reading the power consumption figures from the besch instead of from the factory. This should be backward compatible with old paksets (64 etc) as a large positive constant is used for the default setting when nothing is defined.

It should be noted that factories may consume more or less power in JIT2 which might not be reflected by the power creation rate. This is because JIT2 scales power consumption with the amount of work done giving pakset authors the ability to define a "per-unit produced" power consumption amount instead of it being a fixed amount when the factory is working more than at 75%. Factories working at a rate less than their un-boosted form will use less power (eg a pak64 coal mine supplying a steal mill) while factories working above their un-boosted production amount will consume more. Over a large network power should stabilize at approximately some percentage (if all connected factories are constantly producing at equilibrium) but can also still show high variance (most industries under supplied as no power is used when not working).

Vladki

Thanks for the patch (not tried yet however).

To the papermill issue - In my (theoretical) version of JIT2 this was dealt so that - in case of non-producing factory - inputs are ordered only if they are not in stock at all. In other words, if there are some goods in input, and the factory is not producing, there is no need to order more. The reason for not producing might be either lack of other inputs or full output.

If you dont want to change that, you can set max demand to some low value. Some time ago I have fiddled with one of the early versions of JIT2 patch, and modified it so that the input buffer max was always 10 (I used some constant - min cargo amount delivered or something). It worked just fine as with max buffer = max input. It might help in cases like this paper mill.

Did you have time to look at the printworks issue as well?

prissi

INcorporated, thank you. With the code stabilising, maybe one should consider consolidate the single and multiple into the same routines?