The International Simutrans Forum

 

Author Topic: Headless servers + memory use  (Read 8147 times)

0 Members and 1 Guest are viewing this topic.

Offline Ashley

  • Coder/Patcher
  • Devotees (Inactive)
  • *
  • Posts: 1288
    • entropy.me.uk
Headless servers + memory use
« on: September 03, 2012, 11:53:39 AM »
Question - when you compile the game with posix/bit depth 0, does it still load the pakset graphics into memory? (easier to ask than to look at the code, sorry!)

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10641
  • Languages: De,EN,JP
Re: Headless servers + memory use
« Reply #1 on: September 04, 2012, 10:50:00 PM »
Yes, it is easier ... but it does nothing with the data.

Offline Ashley

  • Coder/Patcher
  • Devotees (Inactive)
  • *
  • Posts: 1288
    • entropy.me.uk
Re: Headless servers + memory use
« Reply #2 on: September 04, 2012, 11:12:43 PM »
How hard would you say it'd be to change this? It isn't such an issue for pak64, but loading all of pak128 graphics is a real memory hog. I'd be happy to look into it if you could give me some pointers.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 5691
  • Languages: EN, NO
Re: Headless servers + memory use
« Reply #3 on: September 05, 2012, 05:03:28 AM »
I would assume a headless version of Simutrans holds only the original graphics, not a second set of rescaled or recolored images.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10641
  • Languages: De,EN,JP
Re: Headless servers + memory use
« Reply #4 on: September 05, 2012, 12:34:41 PM »
The headless server only hold the data in the size they were on the disk, while the normal one creates several copies (at least two usually). Moreover, the headless server does not need to create landscapes tiles. Once in main memory and never accessed it could very well be swapped quickly.

But there is certainly room for improvement (it would only improve the memory consumption, CPU would stay the same as now).

Offline Ashley

  • Coder/Patcher
  • Devotees (Inactive)
  • *
  • Posts: 1288
    • entropy.me.uk
Re: Headless servers + memory use
« Reply #5 on: September 05, 2012, 01:09:13 PM »
A saving of 100-200MB (pak128 is big) would let me run many more servers :) CPU isn't really a limiting factor from what I've seen so far.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 5691
  • Languages: EN, NO
Re: Headless servers + memory use
« Reply #6 on: September 05, 2012, 02:21:04 PM »
Once in main memory and never accessed it could very well be swapped quickly.

As long as the images don't share the page with stuff that is used for something.

Offline Ashley

  • Coder/Patcher
  • Devotees (Inactive)
  • *
  • Posts: 1288
    • entropy.me.uk
Re: Headless servers + memory use
« Reply #7 on: September 05, 2012, 02:51:34 PM »
I'll have to investigate real-world swapping behaviour and see if this holds true.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10641
  • Languages: De,EN,JP
Re: Headless servers + memory use
« Reply #8 on: September 05, 2012, 08:31:13 PM »
I do not claim to understand it; but loading the server without images required 20 kB more than with image data!?! (The patch was not complete, and led to quick crashes, but still ... )

Offline Ashley

  • Coder/Patcher
  • Devotees (Inactive)
  • *
  • Posts: 1288
    • entropy.me.uk
Re: Headless servers + memory use
« Reply #9 on: September 06, 2012, 12:03:27 AM »
Care to post the patch? I'm interested to see what you changed.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 5691
  • Languages: EN, NO
Re: Headless servers + memory use
« Reply #10 on: September 06, 2012, 04:40:02 AM »
I do not claim to understand it; but loading the server without images required 20 kB more than with image data!?! (The patch was not complete, and led to quick crashes, but still ... )

I would assume that there's a few more if's, which in turn might cause the compiler to produce more flexible, but less compact machine code. Still, 20 kB sound like a lot, and too much for this explaination alone.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10641
  • Languages: De,EN,JP
Re: Headless servers + memory use
« Reply #11 on: September 06, 2012, 02:22:57 PM »
Well the heap is allocated in chunks (those image structures are alloced with new library function). If the order was slightly different, or the random start address of the heap was different, the library routine may alloc one chunk more (which might be the 20 kB).

Offline Ashley

  • Coder/Patcher
  • Devotees (Inactive)
  • *
  • Posts: 1288
    • entropy.me.uk
Re: Headless servers + memory use
« Reply #12 on: September 12, 2012, 03:02:07 PM »
I don't quite follow, are you saying that not loading the image data led to the same memory footprint? That seems rather surprising!

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 5691
  • Languages: EN, NO
Re: Headless servers + memory use
« Reply #13 on: September 13, 2012, 08:06:26 PM »
It is technically possible, but I would only expect it if the amount of image data is small compared to everything else.

Offline Dwachs

  • DevTeam, Coder/patcher
  • Administrator
  • *
  • Posts: 4870
  • Languages: EN, DE, AT
Re: Headless servers + memory use
« Reply #14 on: September 29, 2012, 01:49:09 PM »
Here is a patch. Please test :) Reduces memory consumption for pak128 quite a bit...

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10641
  • Languages: De,EN,JP
Re: Headless servers + memory use
« Reply #15 on: September 29, 2012, 02:00:42 PM »
Clever idea with the one pixel left. My attempt with zero pixel lead to various crashes.

Offline Ashley

  • Coder/Patcher
  • Devotees (Inactive)
  • *
  • Posts: 1288
    • entropy.me.uk
Re: Headless servers + memory use
« Reply #16 on: September 29, 2012, 08:54:36 PM »
Such a simple patch, for such an amazing effect. Memory usage has dropped from ~300MB resident to under 60MB for the same map! I've applied this patch against r5843 (111.3.1) on my pak128 server. Appears to work so far. Could probably run 10 servers with this change now :)

Offline paichtis

  • *
  • Posts: 63
Re: Headless servers + memory use
« Reply #17 on: September 30, 2012, 08:06:14 PM »
There are constant desynchs on both 128 servers.
I start wondering if maybe this patch is causing them ? Perhaps a side-effect on the 'desynch-check' has been overlooked ?

Offline Ashley

  • Coder/Patcher
  • Devotees (Inactive)
  • *
  • Posts: 1288
    • entropy.me.uk
Re: Headless servers + memory use
« Reply #18 on: September 30, 2012, 09:43:02 PM »
What country are you playing from, and what's your latency to the server? (e.g. do a ping to pulsar.entropy.me.uk and see how long it takes). Nothing in that patch should touch the network code... Not entirely impossible I suppose.

Offline TurfIt

  • Dev Team, Coder/patcher
  • Devotee
  • *
  • Posts: 1434
Re: Headless servers + memory use
« Reply #19 on: September 30, 2012, 09:51:22 PM »
I'm seeing simrand mismatch disconnects so definitely code related, no idea if this patch could be behind it or not -  trying to track it down now...

Offline paichtis

  • *
  • Posts: 63
Re: Headless servers + memory use
« Reply #20 on: October 01, 2012, 12:30:48 AM »
Ok, played again : server B was very smooth but A is stil near unplayable so I guess it's not related to the patch (sorry for the false report !)

(playing from France)


Offline TurfIt

  • Dev Team, Coder/patcher
  • Devotee
  • *
  • Posts: 1434
Re: Headless servers + memory use
« Reply #21 on: October 01, 2012, 01:52:51 AM »
I can't duplicate this on a local server with or without the patch. Connect to the Entrophy A server and it always disconnects within 5mins due to a simrand mismatch.

Client logging shows the only simrand calls between checklist checks before the failed check were at:
Code: [Select]
simhalt.cc:1767
simcity.cc:786
simcity.cc:1658
simcity.cc:1931
simfab.cc:756
simfab.cc:792
of course I have no way of knowing if the server did other calls or called these a different number of times. I can go no further without local duplication.

However, one strange thing - with the patch, my client can't connect to the server due to a pakset mismatch - "cotton plantation". I need to remove this factory from both client and server pak128 dirs for it to work. But I can connect to Entrophy server without the error??

And, if the server is compiled with multithread, the server crashes as soon a client joins. If the client is compiled with multithread, the client crashes as soon as it joins. Sigh.


EDIT: just remembered - Entropy is running an old version, from before the non-rectangular factory fixes were made. I was testing with r5961. r5919 has the checksum change which is tripping up the no graphics patch. But I'd bet it's that PITA cotton plantation again causing the simrand desyncs in the simfab add field routine in r5843...
« Last Edit: October 01, 2012, 02:10:38 AM by TurfIt »

Offline Jaridan

  • *
  • Posts: 36
  • Languages: DE,EN, Scheme, c0~.~
Re: Headless servers + memory use
« Reply #22 on: October 02, 2012, 12:16:40 AM »
the evil cotton plantage  8)

playing from germany, average ping around 50-70 and same prob ofc.

Offline Ashley

  • Coder/Patcher
  • Devotees (Inactive)
  • *
  • Posts: 1288
    • entropy.me.uk
Re: Headless servers + memory use
« Reply #23 on: October 02, 2012, 07:48:26 AM »
Server A's map is just way too developed I think. New maps definitely required...

Offline VS

  • Senior Plumber (Devotee)
  • Devotees (Inactive)
  • *
  • Posts: 4856
  • Vladimír Slávik
    • VS's Simutrans site
  • Languages: CS,EN
Re: Headless servers + memory use
« Reply #24 on: October 02, 2012, 09:47:05 AM »
But I'd bet it's that PITA cotton plantation again causing the simrand desyncs in the simfab add field routine in r5843...
Yeah :-/ Is this something that does not need to be fixed anymore? 128 still has the old cotton plantation for compatibility; that one is definitely rotation-safe & disabled by DistributionWeight.

Offline Dwachs

  • DevTeam, Coder/patcher
  • Administrator
  • *
  • Posts: 4870
  • Languages: EN, DE, AT
Re: Headless servers + memory use
« Reply #25 on: October 02, 2012, 10:04:35 AM »
Yeah :-/ Is this something that does not need to be fixed anymore?
Yes, I hope ;)

Offline Fifty

  • *
  • Posts: 280
  • Languages: EN, ES
Re: Headless servers + memory use
« Reply #26 on: October 02, 2012, 07:53:28 PM »
I created the A server map in an older version of Simutrans, and perhaps even an older pak 128. This could have caused the problems. I also placed plantations manually, so I could have chosen the wrong version.

The original savegame can be found here:

http://simutrans-germany.com/files/upload/Portland_Map_v7m.sve

If it's as simple as replacing the cotton plantations, I'll be happy to do that.

Offline TurfIt

  • Dev Team, Coder/patcher
  • Devotee
  • *
  • Posts: 1434
Re: Headless servers + memory use
« Reply #27 on: October 04, 2012, 09:31:54 PM »
Fixed patch attached. No more desyncs due the cotton plantation.

@dwachs- is the image reader version==0 change intentional? (size-10 vs size-12, node_info=new deleted)

Offline Dwachs

  • DevTeam, Coder/patcher
  • Administrator
  • *
  • Posts: 4870
  • Languages: EN, DE, AT
Re: Headless servers + memory use
« Reply #28 on: October 05, 2012, 07:37:43 AM »
Thanks for fixing this! It was a dumb error :/

About size-10:  Was lazy enough to not care about the size-12 thing. But could easily be added.

The call to new in the COLOUR_DEPTH==0 case should be changed to
Code: [Select]
besch = new(sizeof(uint16)) bild_besch_t();
I did not know what I am doing here. This call is translated to a call to
Code: [Select]
obj_besch_t::new( sizeof(bild_besch_t), sizeof(uint16))
anyway, as C++ already takes into account the size of the structure. Had to read up first, what the new operator is doing, and what the purpose of
new(size_t size, size_t extra) is..

Offline Dwachs

  • DevTeam, Coder/patcher
  • Administrator
  • *
  • Posts: 4870
  • Languages: EN, DE, AT
Re: Headless servers + memory use
« Reply #29 on: October 05, 2012, 06:16:28 PM »
Modified patch looks successfull! Thanks TurfIt. At last, valgrind does not crash (I had spurious crashes in the buffered loading code...)

Edit:

@Timothy: You may want to recompile your headless servers with the patched patch attached to this post :)

Edit2: incorporated in r5964.
« Last Edit: October 05, 2012, 07:17:25 PM by Dwachs »