Online Game with SE

[steffen]

Hello,
After a long network game in standard I'd really like to try one with experimental. So, who's interested? I'm happy to host the game.
Before we start we have to decide on a few things:

A. Pak: I'd like to use Britain because it's most adapted to SE features and the sources are available so we can make balancing and other changes ourselves. Whilst support for more SE features is nice, I can live without them. However, the source availability is quite important to me.
B. Map: Don't really care, I'd suggest http://maps.simutrans.com/downloads/british%20isles%20768.zip
I found it on http://maps.simutrans.com/europe.html and it seems to be a good size and has plenty of flat space, hilly space, and coastal areas so that everyone gets to play the kind of transports that they want to play. I'm happy to use the map generator, but if someone really wants to place stuff manually then that's fine by me.
C. Cooperative or competitive (and if the latter, unregulated or do we set some rules?): Since this is one of the first big SE network games I'd like to start on cooperative.
D. Player numbers: I think 3-4 is ideal, and also works well on the britain map. I don't want 6 or more players tho.

And some settings:
3. Timeline: Definitely on.
4. Start date: I'd suggest 1870 but I really don't care.
5. Speed: If we start in the 19th century I think we should put up speed by one step compared to the default
6. Station Coverage Area: I think 2 is a bit small, as it makes it really hard to cover cities with pure or mostly rail/tram without demolishing half the city. So I'd like to make it 3.
7. Starting Cash: I'd say we should start on at least 1 million.
8. Revenue: I think we should raise the money earned by, say, 20% to make up for the effect of network playing. Since we won't be able to fully control the whole network there'll probably be more jams, gridlocks and disruptions.
9. Passengers at factories: I think we should reduce this setting by, say, half. Whilst the concentration of traffic to/from workplaces is realistic, the amount of factories in simutrans is not (and shouldn't be). With the default every workplace basically needs to a passenger hub and that's quite irritating.

[steffen]

10. Private cars: I'd prefer to have them off since they tend to cause deadlocks, and in a network game the affected player might not be online to fix it.

[jamespetts]

Steffen,

I should be very interested indeed to see an online game with Experimental, especially with Pak128.Britain-Ex. I shall be releasing a new version (0. very soon, so perhaps we should wait for that? As to the other points, my comments using your numbering:

B: Simutrans-Experimental is good at generating maps automatically (better than Standard for larger maps), so it'd be interesting to see how this pans out in an online game.
C: Simutrans-Experimental has been set up to allow competitive play more than works in Standard, so I'd be very interested in seeing something that allows for competitive play. A combination of the two (cooperative and competative) might work well, with more than one player playing at least some of the transport companies.
D: Do you mean company slots here or actual players (as the latter might be higher than the former if multiple players play the same company)? Perhaps leave it open for 5 company slots (not counting spectator and public service) and see where we go from there?

3. Yes, Pak128.Britain does not work well without the timeline.
4. This is probably a good date for an initial testing game.
5. Can we try the default for the time being? We shouldn't have things going too quickly if people are not actually playing for most of the time.
6. 3 is the default in Pak128.Britain-Ex in any event.
7. That seems a little high - is 250,000c not enough for 1870?
8. Pak128.Britain-Ex is not really balanced to work as a monopoly in any event, so this should not really need adjusting (to be fair, Pak128.Britain is not really fully balanced yet at all, but it seems sensible to start with default settings and adjust later so that it can be a better test).
9. This has already been reduced substantially in any event from Pak128.Britain-Ex 0.7.
10. I tend to prefer to have these on, as the congestion that they cause for road transport is an important factor affecting the effectiveness of 'buses without which the economics are distorted somewhat.

[steffen]

Steffen,

I should be very interested indeed to see an online game with Experimental, especially with Pak128.Britain-Ex. I shall be releasing a new version (0. very soon, so perhaps we should wait for that? As to the other points, my comments using your numbering:

Might as well start now and update when the new version is out
Or we could play with an unreleased version, are there nightly builds for Windows or is that just for Linux? Which branch is your main devel branch?

C: Simutrans-Experimental has been set up to allow competitive play more than works in Standard, so I'd be very interested in seeing something that allows for competitive play. A combination of the two (cooperative and competative) might work well, with more than one player playing at least some of the transport companies.

Well I was thinking "walk before you run", but I'd like to make the game after this one competitive.

D: Do you mean company slots here or actual players (as the latter might be higher than the former if multiple players play the same company)? Perhaps leave it open for 5 company slots (not counting spectator and public service) and see where we go from there?

I meant companies. If more than one person want to share a company that's fine by me, I don't see why one person would play multiple companies though (in the general case, but occasionally one might make some corrections to another person's company).

[speed] 5. Can we try the default for the time being? We shouldn't have things going too quickly if people are not actually playing for most of the time.

Ok, default it is

6. 3 is the default in Pak128.Britain-Ex in any event.

Great, 3 it is then

7. That seems a little high - is 250,000c not enough for 1870?

I'd like to have quite a buffer to be honest, how about 500k?

8. Pak128.Britain-Ex is not really balanced to work as a monopoly in any event, so this should not really need adjusting (to be fair, Pak128.Britain is not really fully balanced yet at all, but it seems sensible to start with default settings and adjust later so that it can be a better test).

Ok, default then

9. This has already been reduced substantially in any event from Pak128.Britain-Ex 0.7.

Ok let's try it with the default then.

10. I tend to prefer to have these on, as the congestion that they cause for road transport is an important factor affecting the effectiveness of 'buses without which the economics are distorted somewhat.

True, but they also mean that there's even more micromanagement required to keep busses&trucks at reasonable spacing. And most of the deadlocks we had in the standard game were caused by private cars.

James, would you like to join the game? I'm in IRC right now

[ӔO]

oh, could I join?

what is the server?

[jamespetts]

We're testing currently - it seems that Windows clients have trouble joining Linux servers for reasons as yet unascertained.

[ӔO]

ah, that could prove to be difficult indeed.

[jamespetts]

It's a thoroughly odd problem - I'm not really sure how such a thing might occur. It desyncs instantly if connected from a Windows machine, but runs fine for hours if connected from a Linux machine.

[prissi]

Different order of tables in GCC and MSVC? Different mxints (32 vs. 64 bit code?)

[jamespetts]

Prissi,

thank you for the thoughts. I know that it can't be the second, as I tried with a 32-bit Linux system connecting to a 64-bit Linux system. The other suggestion might be closer - but I must confess, I'm not sure what you mean by "order of tables" here, and a brief Google search didn't really elucidate. Can you elaborate? Thank you.

[VS]

I guess "table" should read "vtable" for maximal clarity...

[jamespetts]

Thank you, VS. How would I test to see whether the issue is a different vtable order in MSVC++ and GCC? I'm not sure that I fully understand that issue.

[Dwachs]

I think "order of tables" means something like:

If you index an array by random indexes, then ensure the array is always in the same order. Ie use sorted lists. One major point is that the lists that hold game objects like houses, trees etc have always the same order on all clients, in particular this order is independent of the order the pak files are read.

[jamespetts]

I think "order of tables" means something like:

If you index an array by random indexes, then ensure the array is always in the same order. Ie use sorted lists. One major point is that the lists that hold game objects like houses, trees etc have always the same order on all clients, in particular this order is independent of the order the pak files are read.

Ahh, thank you. I think that I understand partly now - but only partly. How is this sorting handled in Standard for things like the list of all the types of goods, lists of industries, lists of stops, etc.?

[Dwachs]

There are many such sorting operations in the register_besch-methods / alles_geladen methods (ie check hausbauer_t::alles_geladen()), to keep the besch's sorted in lists, ie houses, attractions, trees, etc. The problem with these lists is that the order if which pak's are read depends on your filesystem. To be able to access these lists randomly (ie index determined by simrand), one has to be sure that these lists are in the same order on all clients. (Most prominent operation: city builds a random house)

Lists that are created in-game should be save (ie list of goods at a station etc). These lists are (hopefully) generated/loaded the same way on each client. Any mismatch in these lists can be caused only by earlier divergence of the whole simulation.

[paco_m]

And why not abstracting this?
1) reading pak files for buildings
2) sorting the list (alphabetically or another determined order)
3) applying the random selection on 2)^

[jamespetts]

I have now tried with 9.3 and Pak128.Britain-Ex 0.8 using a 32-bit Linux server and a Windows client (32-bit executable on 64-bit system), and vice versa, with no difficulties. This appears to have been, in part, a pakset issue.

Edit: Spoke too soon - after playing for a short time, the Windows client desyncs instantly upon connecting to the Linux server.

Edit 2: Although, oddly, restarting the Windows client temporarily abates the problem: when the desync occurs, trying to reconnect results in an instant desync; but after restarting, it will remain connected for a while before desyncing (but it will desync after a few minutes).

Edit 3: When connecting a Linux client to a Windows server, the symptoms are different: it will desync fairly regularly (every minute or two), but will not desync instantly on joining.

Edit 4: Examining logs on the Linux machine shows that the reason given for the desync is a checklist mismatch, being a mismatch of the random seed.

Edit 5: Testing with Simutrans-Experimental 9.3 and Pak128.Britain (Standard), the problem does not occur with a Windows server and Linux client. Does anyone have any thoughts on why the problem might be unique to Simutrans-Experimental specific paksets?

Edit 6: The desyncs appear to continue even with the Standard pakset, although initially they were less frequent. It is not clear whether this is connected to the fact that the one and only train in my test setup did not have a locomotive in the standard pakset, so it was reduced to the default minimum speed of 4km/h, and rarely called into the stations as a result.

[steffen]

Someone (I think paco) suggested this could be due to a problem with the handling of SE-specific values. This would also explain why it happens less with a standard pak (if it indeed does happen less with a standard pak).

[jamespetts]

I'm not sure that it does really happen less with the standard Pak - from the above testing, it appears as if this might have been an incorrect appearance given by the fact that the standard pak did not have the GWR 517 class, so that the one and only train moved very slowly and consequently called into the stations far less often.

[steffen]

Maybe you could add some debug output that states what has become desynced?

[jamespetts]

It is the random number generator that becomes desynced - I know that already.

[steffen]

Ah I didn't realise that. Do you regularly sync up with std? Assuming the RNG is used there, too, and given that online play works with std, maybe they have the fix for this problem already?

[jamespetts]

It's not that simple, I'm afraid. There is no problem with the random number generator. The random seed desync is caused by any deviation in the code path between the client and the server such that the client calls the random number generator at a different point in the execution of the code to the server.  This can occur when the deviations in the code path is many, many methods before the first diverging random call, making this sort of problem stupendously difficult to isolate and fix.

[steffen]

I see, yes, that complicates things just a teeeeny bit.
Is there any way to log the calls to the RNG? By turning on a runtime/compiletime function or maybe by running it in a debugger?

[jamespetts]

I have alredy implemented a system for logging calls to the RNG, which I have used successfully (albeit after many weeks of hard work) to fix issues that arose when testing with a client and server on the same machine. The issue is finding the enormous amount of time likely to be required to run this exercise again in ways specific to the Linux/Windows issue, added to the complexity of which, I am not sure how to set up a debugger in Linux.

I was hoping that there may be specific things to which I could narrow things down by behavioural analysis combined with asking others about why there might - in general - be desyncs on Windows - Linux connexions but not on Windows - Windows or Linux - Linux connexions; but even that will take a considerable amount of time to process, so any help would be much appreciated.

[jamespetts]

Recent testing reveals the following (with Experimental 9.5 and Pak128.Britain-Ex 0.8):

1. desyncs do not occur in maps without any player transport networks (but with private cars and towns);
2. desyncs do not occur in maps that, whilst they have transport networks, none of them connect to any sources or destinations of passengers or goods;
3. desyncs occur very quickly in large, well developed maps;
4. desyncs occur after a longer period in small maps with two passenger transport routes;
5. turning off city growth on all cities in the map does not affect desyncs;
6. turning off private cars does not affect desyncs;
7. desyncs do not (or, at least, not always) occur at the same moment that vehicles arrive at stops;
8. desyncs do not appear to occur in maps with only one passenger route and no other network;
9. desyncs often appear to occur very shortly after a manual save on the server, but this is not invariably the case;
10. starting a map with no industries does not affect desyncs;
11. desyncs do not occur at the point of industries expanding or attractions being built; and
12. desyncs do not occur at regular points (such as month or year ends).

I am currently working on testing whether desyncs occur in a map with no towns but a moderately developed (rail) freight network. In the meantime, any thoughts on what might be responsible for the above pattern would be very much appreciated.

Edit: Further testing has shown that desyncs do eventually occur in maps with only one passenger route and no other network, but do not occur in maps with a moderately developed freight network but no towns.

[steffen]

Well you managed to exclude quite a lot of things already. So it seems the cause is something to do with passenger transport. So with my minimal knowledge of the code I can think of these causes:
- passenger generation
- passenger "storage" at stations or in vehicles
- passenger routing insofar as it differs from freight (so comfort, dining car, mailcar)
Did I miss anything?

[jamespetts]

Steffen,

the first two are possibilities (although I don't think that the code for passenger storage is any different from the code for storage of any other types of transportable), but passenger routing (as between stops) is no different to the routing of freight, and, moreover, comfort and catering (including TPOs) affect revenue, but not routing. Catering and TPOs can further be excluded as none of the vehicles in my test setups had these features. Also, for something that only affects revenue to affect the random number generator seems unlikely.

[steffen]

I see, I thought comfort can affect routing, too. Assuming the RNG is not involved in pax storage that leaves just pax generation

[jamespetts]

Ahh, no, comfort is not involved in routing. The reason for that is a two variable routing system would be very much more computationally intensive than a one variable routing system (as the question "which route is better - A or B?" has to be asked vast numbers of times every step, and comparing two journey times is trivial, but using algorithms to combine journey time and comfort and then see which is the best overall would greatly increase the complexity of that basic comparison).

As to passenger generation, some (but not all) of the code for passenger generation is used even when player transport is not involved, as passenger generation is invoked even when there are no stations; passengers can reach their destination by private car.

[steffen]

Hm so assuming all our discoveries/assumptions are correct, the problem must be in the part of pax generation that isn't employed by private car. Or the problem doesnt have a singular cause but requires two conditions/events to occur at the same time..

[jamespetts]

I am currently testing this hypothesis on a developed map that had a passenger network in which I have withdrawn all the 'buses (which were the only passenger carrying vehicles). It did desync once after some time of running: I am checking again to see whether this can be reproduced.

[steffen]

Let me know if I can help. I'm also in #simutrans

[jamespetts]

It's definitely not passenger generation, at least not only passenger generation: the map with all the 'buses withdrawn still desyncs. I shall double check by trying it after having removed all the 'bus stops.

Edit: Another oddity: the message telling users that the vehicle cannot find a route because it is overweight does not appear on Linux, but works perfectly on Windows.

Edit 2: Running the test game with all 'bus stops removed still leads to an (eventual) desync (6th of February 1986 game time) on a game with a freight-only road network. It seems that the problem is not related to passenger routing at all, or, alternatively, there are two problems, at least one of which is not related to passenger routing.

Edit 3: Deleting electricity.tab and privatecar.tab in both machines' /[pakset]/config folders makes no difference to the desyncs.

Edit 4: An interesting result. Compare the two screen-shots below. This is the passenger destination display. The first is from Windows and the second from Linux. The pattern of coloured squares should be identical on both. Notice how it is identical apart from one difference: the Linux screenshot has a teal square in the top right area where the Windows screenshot has a yellow square.

The yellow indicates that passengers have reached their destination successfully by player transport, whereas the teal indicates that passengers have reached their destination successfully by private car. I tracked this divergence for a minute or two before the game desynced. This square remained in the wrong colours consistently, whilst all of the other squares were consistent with each other on the two machines.

This suggests a problem somewhere with the private car system - a problem which occurs only infrequently but causes passengers to differ in their choice between private car transport and player transport on Linux and Windows machines when it does occur. Further investigation by me will have to wait until I have more time, as it is now past my bed time, but any thoughts or code investigations by anyone here would be very welcome.

[Ashley]

If you manage to fix the problems I'm interested in hosting a server for experimental (maybe pakBrit experimental...) so give me a shout