Author Topic: RNG issues  (Read 450 times)

0 Members and 1 Guest are viewing this topic.

Offline Jando

RNG issues
« on: June 24, 2017, 09:03:28 AM »
Stupid me, how did I miss this thread back in April?

Interesting ideas for sure, James. One thing that I would wish from any new mechanic (thus also from passenger and mail classes) is to significantly reduce the amount of randomness in passenger and mail creation. I currently have a map with a small regional bus network (25 Leyland Tiger coaches) connecting about 10 smaller towns and villages (largest town 6.800 inhabitants) and - frankly - it's more or less the random number generator that determines how much, if any, profit that network will make in a month. I hope that in implementing passenger and mail classes you can avoid adding more randomness to the mechanic.

Moderator note: I have split this topic from this discussion of balance, as this appears to be a bug report rather than the discussion of revenue balance features.
« Last Edit: June 25, 2017, 10:08:53 AM by jamespetts »

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 15425
  • Total likes: 376
  • Helpful: 171
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: RNG issues
« Reply #1 on: June 24, 2017, 09:39:15 AM »
Passenger generation inherently relies on randomness to work, as we cannot simulate each citizen's internal life. What you are describing suggests some other problem than the mere fact of randomness, however. Could you possibly post a new thread about it in more detail, with a saved game?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Jando

Re: RNG issues
« Reply #2 on: June 24, 2017, 10:03:07 AM »
Passenger generation inherently relies on randomness to work, as we cannot simulate each citizen's internal life. What you are describing suggests some other problem than the mere fact of randomness, however. Could you possibly post a new thread about it in more detail, with a saved game?

Hi James, I'll gather the data and make a thread in development over the weekend.

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 15425
  • Total likes: 376
  • Helpful: 171
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: RNG issues
« Reply #3 on: June 24, 2017, 10:07:30 AM »
Thank you.

Edit: I have just pushed a fix to the random number generation algorithm: it was previously not properly seeded when running in the multi-threaded parts of the code (in particular, the passenger generation routine), meaning that there was actually not enough randomness, as lots of passengers would end up trying to go to the same place, rather than being spread out over different destinations. I do not know whether this will fix your issue, but I should be grateful if you could test and let me know whether it does. Thank you for the report.
« Last Edit: June 24, 2017, 11:31:55 AM by jamespetts »
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Jando

Re: RNG issues
« Reply #4 on: June 24, 2017, 11:59:36 AM »
I'll re-run my tests with the newest nightly tomorrow then. Thanks.

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 15425
  • Total likes: 376
  • Helpful: 171
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: RNG issues
« Reply #5 on: June 24, 2017, 12:43:57 PM »
Splendid, thank you.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Jando

Re: RNG issues
« Reply #6 on: June 25, 2017, 10:05:18 AM »
Tests re-run with the new nightly version. If anyone wants to run tests too, here is the saved game that I used for the tests: http://files.simutrans.com/index.php/s/JcyLC9ppj0Xuzqo

What's this about? It's a regional passenger network in 1941 connecting 12 villages and towns with about 25 Leyland Tiger buses. No large cities, town size varies from around 1.700 to 6.800 inhabitants. All buses run on time-scheduled lines, no changes are made to the map or network between different runs. Game is saved at the start of the month May and then is run in fast-forward for the month, then results are documented and game is loaded again for another monthly run. I made 15 runs in total, here are the results:

Code: [Select]
Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Transported 5.283 4.508 4.607 4.608 5.113 4.815 5.133 4.619 4.352 4.531 4.940 4.693 4.434 4.471 5.039
Running Costs 9.988 9.991 9.988 9.988 9.979 9.977 9.985 9.994 9.996 9.985 9.991 9.992 9.989 9.988 9.988
Revenue 17.031 14.005 14.916 14.533 16.639 15.450 15.973 14.619 13.857 14.473 16.137 15.255 14.052 14.312 16.386
Profit 6.193 3.165 4.079 3.698 5.809 4.626 5.140 3.776 3.011 3.640 5.297 4.415 3.212 3.476 5.549

Why all this? Because I wanted to know how large the influence of the random number generator is for balance in Simutrans-Extended. I noticed before these tests that different months with the basically same network can have quite varying results in terms of revenue or profit. Thus I thought it's best to have many runs with the exact same network to find out. :)

@James, variance between different runs was actually larger with the old client (before you made the recent fix to the RNG algorithm), before that fix I observed operating profits ranging from -600 to +6.800. But that may well be down to the fact that passengers were created more unevenly before the change, overall passenger creation seems better distributed between different stops now with the new version.

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 15425
  • Total likes: 376
  • Helpful: 171
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: RNG issues
« Reply #7 on: June 25, 2017, 10:13:50 AM »
That is an interesting chart. It is satisfactory to note that the range appears no longer to show orders of magnitude fluctuations, which would have been excessive. I have not conducted detailed statistical analysis on precisely how much random fluctuation that one might expect to see. Such a task would be a very large amout of work, and take time away from other development projects. I cannot, therefore, comment on the schedule in any detail. All that I can say is that, on the basis of a very approximate impression, there does not appear to be anything in the table to suggest that there is anything seriously wrong with the code. (That does not mean that I can be quite sure that there is, in fact, nothing wrong with the code, but, without conducting a very extensive analysis, the figures alone do not suggest such a problem).

The figures that you briefly give for the range of profit before the RNG fix were more concerning, but it is reassuring to note that this now appears to have been fixed.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Jando

Re: RNG issues
« Reply #8 on: June 25, 2017, 10:35:32 AM »
... All that I can say is that, on the basis of a very approximate impression, there does not appear to be anything in the table to suggest that there is anything seriously wrong with the code. ...

I totally agree with you on this.

A factor of roughly 2 in terms of operating profit on these test runs seems fine to me (lowest profit 3.011, highest 6.193). I do not know any real life figures as well but my feeling says a factor of 2 is pretty okay for Simutrans Extended.

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 15425
  • Total likes: 376
  • Helpful: 171
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: RNG issues
« Reply #9 on: June 25, 2017, 10:38:34 AM »
Excellent. Thank you for having reported the original issue - that was helpful.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Jando

Re: RNG issues
« Reply #10 on: June 25, 2017, 12:02:44 PM »
And - just in case you don't know:

In the previous version all passengers showing up at stops for pickup would come in multiples of 4, i.e. 8, 12, 16, etc up to 28 passengers while all passengers not showing up (registered at stops under "no route" or "too slow") would come in multiples of 5. With the new version this is no longer the case. :)