News:

SimuTranslator
Make Simutrans speak your language.

Headless servers + memory use

Started by Ashley, September 03, 2012, 11:53:39 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Ashley

Question - when you compile the game with posix/bit depth 0, does it still load the pakset graphics into memory? (easier to ask than to look at the code, sorry!)
Use Firefox? Interested in IPv6? Try SixOrNot the IPv6 status indicator for Firefox.
Why not try playing Simutrans online? See the Game Servers board for details.

prissi

Yes, it is easier ... but it does nothing with the data.

Ashley

How hard would you say it'd be to change this? It isn't such an issue for pak64, but loading all of pak128 graphics is a real memory hog. I'd be happy to look into it if you could give me some pointers.
Use Firefox? Interested in IPv6? Try SixOrNot the IPv6 status indicator for Firefox.
Why not try playing Simutrans online? See the Game Servers board for details.

Ters

I would assume a headless version of Simutrans holds only the original graphics, not a second set of rescaled or recolored images.

prissi

The headless server only hold the data in the size they were on the disk, while the normal one creates several copies (at least two usually). Moreover, the headless server does not need to create landscapes tiles. Once in main memory and never accessed it could very well be swapped quickly.

But there is certainly room for improvement (it would only improve the memory consumption, CPU would stay the same as now).

Ashley

A saving of 100-200MB (pak128 is big) would let me run many more servers :) CPU isn't really a limiting factor from what I've seen so far.
Use Firefox? Interested in IPv6? Try SixOrNot the IPv6 status indicator for Firefox.
Why not try playing Simutrans online? See the Game Servers board for details.

Ters

Quote from: prissi on September 05, 2012, 12:34:41 PM
Once in main memory and never accessed it could very well be swapped quickly.

As long as the images don't share the page with stuff that is used for something.

Ashley

I'll have to investigate real-world swapping behaviour and see if this holds true.
Use Firefox? Interested in IPv6? Try SixOrNot the IPv6 status indicator for Firefox.
Why not try playing Simutrans online? See the Game Servers board for details.

prissi

I do not claim to understand it; but loading the server without images required 20 kB more than with image data!?! (The patch was not complete, and led to quick crashes, but still ... )

Ashley

Care to post the patch? I'm interested to see what you changed.
Use Firefox? Interested in IPv6? Try SixOrNot the IPv6 status indicator for Firefox.
Why not try playing Simutrans online? See the Game Servers board for details.

Ters

Quote from: prissi on September 05, 2012, 08:31:13 PM
I do not claim to understand it; but loading the server without images required 20 kB more than with image data!?! (The patch was not complete, and led to quick crashes, but still ... )

I would assume that there's a few more if's, which in turn might cause the compiler to produce more flexible, but less compact machine code. Still, 20 kB sound like a lot, and too much for this explaination alone.

prissi

Well the heap is allocated in chunks (those image structures are alloced with new library function). If the order was slightly different, or the random start address of the heap was different, the library routine may alloc one chunk more (which might be the 20 kB).

Ashley

I don't quite follow, are you saying that not loading the image data led to the same memory footprint? That seems rather surprising!
Use Firefox? Interested in IPv6? Try SixOrNot the IPv6 status indicator for Firefox.
Why not try playing Simutrans online? See the Game Servers board for details.

Ters

It is technically possible, but I would only expect it if the amount of image data is small compared to everything else.

Dwachs

Here is a patch. Please test :) Reduces memory consumption for pak128 quite a bit...
Parsley, sage, rosemary, and maggikraut.

prissi

Clever idea with the one pixel left. My attempt with zero pixel lead to various crashes.

Ashley

Such a simple patch, for such an amazing effect. Memory usage has dropped from ~300MB resident to under 60MB for the same map! I've applied this patch against r5843 (111.3.1) on my pak128 server. Appears to work so far. Could probably run 10 servers with this change now :)
Use Firefox? Interested in IPv6? Try SixOrNot the IPv6 status indicator for Firefox.
Why not try playing Simutrans online? See the Game Servers board for details.

paichtis

There are constant desynchs on both 128 servers.
I start wondering if maybe this patch is causing them ? Perhaps a side-effect on the 'desynch-check' has been overlooked ?

Ashley

What country are you playing from, and what's your latency to the server? (e.g. do a ping to pulsar.entropy.me.uk and see how long it takes). Nothing in that patch should touch the network code... Not entirely impossible I suppose.
Use Firefox? Interested in IPv6? Try SixOrNot the IPv6 status indicator for Firefox.
Why not try playing Simutrans online? See the Game Servers board for details.

TurfIt

I'm seeing simrand mismatch disconnects so definitely code related, no idea if this patch could be behind it or not -  trying to track it down now...

paichtis

Ok, played again : server B was very smooth but A is stil near unplayable so I guess it's not related to the patch (sorry for the false report !)

(playing from France)


TurfIt

#21
I can't duplicate this on a local server with or without the patch. Connect to the Entrophy A server and it always disconnects within 5mins due to a simrand mismatch.

Client logging shows the only simrand calls between checklist checks before the failed check were at:
simhalt.cc:1767
simcity.cc:786
simcity.cc:1658
simcity.cc:1931
simfab.cc:756
simfab.cc:792

of course I have no way of knowing if the server did other calls or called these a different number of times. I can go no further without local duplication.

However, one strange thing - with the patch, my client can't connect to the server due to a pakset mismatch - "cotton plantation". I need to remove this factory from both client and server pak128 dirs for it to work. But I can connect to Entrophy server without the error??

And, if the server is compiled with multithread, the server crashes as soon a client joins. If the client is compiled with multithread, the client crashes as soon as it joins. Sigh.


EDIT: just remembered - Entropy is running an old version, from before the non-rectangular factory fixes were made. I was testing with r5961. r5919 has the checksum change which is tripping up the no graphics patch. But I'd bet it's that PITA cotton plantation again causing the simrand desyncs in the simfab add field routine in r5843...

Jaridan

the evil cotton plantage  8)

playing from germany, average ping around 50-70 and same prob ofc.

Ashley

Server A's map is just way too developed I think. New maps definitely required...
Use Firefox? Interested in IPv6? Try SixOrNot the IPv6 status indicator for Firefox.
Why not try playing Simutrans online? See the Game Servers board for details.

VS

Quote from: TurfIt on October 01, 2012, 01:52:51 AM
But I'd bet it's that PITA cotton plantation again causing the simrand desyncs in the simfab add field routine in r5843...
Yeah :-/ Is this something that does not need to be fixed anymore? 128 still has the old cotton plantation for compatibility; that one is definitely rotation-safe & disabled by DistributionWeight.

My projects... Tools for messing with Simutrans graphics. Graphic archive - templates and some other stuff for painters. Development logs for most recent information on what is going on. And of course pak128!

Dwachs

Quote from: VS on October 02, 2012, 09:47:05 AM
Yeah :-/ Is this something that does not need to be fixed anymore?
Yes, I hope ;)
Parsley, sage, rosemary, and maggikraut.

Fifty

I created the A server map in an older version of Simutrans, and perhaps even an older pak 128. This could have caused the problems. I also placed plantations manually, so I could have chosen the wrong version.

The original savegame can be found here:

http://simutrans-germany.com/files/upload/Portland_Map_v7m.sve

If it's as simple as replacing the cotton plantations, I'll be happy to do that.
Why do we park on the driveway and drive on the parkway?

TurfIt

Fixed patch attached. No more desyncs due the cotton plantation.

@dwachs- is the image reader version==0 change intentional? (size-10 vs size-12, node_info=new deleted)

Dwachs

Thanks for fixing this! It was a dumb error :/

About size-10:  Was lazy enough to not care about the size-12 thing. But could easily be added.

The call to new in the COLOUR_DEPTH==0 case should be changed to

besch = new(sizeof(uint16)) bild_besch_t();

I did not know what I am doing here. This call is translated to a call to

obj_besch_t::new( sizeof(bild_besch_t), sizeof(uint16))

anyway, as C++ already takes into account the size of the structure. Had to read up first, what the new operator is doing, and what the purpose of
new(size_t size, size_t extra) is..
Parsley, sage, rosemary, and maggikraut.

Dwachs

#29
Modified patch looks successfull! Thanks TurfIt. At last, valgrind does not crash (I had spurious crashes in the buffered loading code...)

Edit:

@Timothy: You may want to recompile your headless servers with the patched patch attached to this post :)

Edit2: incorporated in r5964.
Parsley, sage, rosemary, and maggikraut.