The International Simutrans Forum

 

Author Topic: Performance impact of pak file grouping  (Read 320 times)

0 Members and 1 Guest are viewing this topic.

Offline Ranran jp

  • *
  • Posts: 415
  • Languages: ja
Performance impact of pak file grouping
« on: April 04, 2019, 10:36:41 AM »
Regarding loading time issues:

Why doesn't 128.Britain-Ex merge pak files?
That makes the initial boot time much faster.
It seems that former 128.Britain-Ex was doing file merging.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18593
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Performance impact of pak file grouping
« Reply #1 on: April 04, 2019, 10:40:40 AM »
Regarding loading time issues:

Why doesn't 128.Britain-Ex merge pak files?
That makes the initial boot time much faster.
It seems that former 128.Britain-Ex was doing file merging.

The use of individual pak files is due to the way in which the Linux makefile for the pakset works, and the nightly pakset build comes from Linux. When in the old days I used to distribute a manually compiled version of the pakset, I used Windows, and the Python script in Windows is configured to build merged .pak files.

I did not realise that the pakset loading time was affected by which of the two were used. How much of a difference does this make? When I load Simutrans myself, I tend to use the merged pak files as it is easier for me to compile locally than download.

Offline Ranran jp

  • *
  • Posts: 415
  • Languages: ja
Re: Performance impact of pak file grouping
« Reply #2 on: April 06, 2019, 10:54:05 PM »
How much of a difference does this make?
There is this difference in HDD. (´・ω・`)

Pak128.Britain-EX (normal)


Pak128.Britain-EX (merged ver.)



However, it is very fast if the cache is left next time you boot.
This is the same with standard.
However, it is strange that extended takes a very long time to start loading compared to standard. It is the same even if it installs in SSD. (pak loading time will be shorter)



Reference information:
paksetfolder sizepaksloading time
pak128402MB569sec
pak128.german280MB175026sec
pak192.comic470MB241930sec

EDIT:
The merged experiment britain pakset consists of the following 32 paks.
https://i.imgur.com/ttVcUXc.png
This is based on pak128Britain.bat in the source folder.

Unmerged pak128.Britain-EX has 4500 over pak files. Thus I think it is the pakset with the longest startup time in Simutrans.
« Last Edit: April 07, 2019, 12:46:50 AM by Ranran »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18593
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Performance impact of pak file grouping
« Reply #3 on: April 07, 2019, 12:05:19 AM »
Thank you for the demonstration. I have split this from the path explorer development topic as that is quite separate.

This is interesting. I am not sure what is behind this. It will take some considerable effort to reconfigure the makefile to provide for merged .pak files, so it may be some time before I am able to look into this.

If anyone would like to look into this in the meantime (either as to how to modify the makefile or why this makes a difference to performance in the first place), that would be most useful.

Offline Vladki cz

  • Devotee
  • *
  • Posts: 2634
    • My addons, mostly roadsigns
  • Languages: EN, CS
Re: Performance impact of pak file grouping
« Reply #4 on: April 07, 2019, 12:02:23 PM »
The delay before loading starts may be due to another bug, you have to move mouse after choosing pakset to load.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18593
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Performance impact of pak file grouping
« Reply #5 on: April 07, 2019, 12:05:27 PM »
The delay before loading starts may be due to another bug, you have to move mouse after choosing pakset to load.

This is, I understand, an issue from Standard.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 9460
  • Languages: De,EN,JP
Re: Performance impact of pak file grouping
« Reply #6 on: April 07, 2019, 12:20:06 PM »
I hardly doubt this, since I mainly develop on a laptop. So after a mouseclick on the trackpad buttons (real button) the mouse will not move at all, but still load the pakset immediately.

The slow speed is simply from the overhead of the OS/libraries, i.e. getting a new file handle, requesting memory for it, loading etc. However, since after first time loading the file informations are still in the cache, the second laoding will be as fast as with fewer, larger files.

Offline Ranran jp

  • *
  • Posts: 415
  • Languages: ja
Re: Performance impact of pak file grouping
« Reply #7 on: April 07, 2019, 02:00:13 PM »
Quote from: Vladki on Today at 12:02:23 PM
The delay before loading starts may be due to another bug, you have to move mouse after choosing pakset to load.

This is, I understand, an issue from Standard.
I think this issue is unique to extended. Standard starts loading immediately without doing anything.
And, as Vladki points out, when I operate the mouse, loading may start in extended pakset loading.

The slow speed is simply from the overhead of the OS/libraries, i.e. getting a new file handle, requesting memory for it, loading etc.
Does the decrease in the number of files due to merging mean that the loading time at the first startup simutrans can be reduced with certainty?

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 9460
  • Languages: De,EN,JP
Re: Performance impact of pak file grouping
« Reply #8 on: April 08, 2019, 07:42:40 AM »
The loading time issue is much more severe on windows (although my last linux profiling was a long time ago). But it seems Linux file system driver loads the entire directory when opening it, and hence access to more files is almost as fast as if cached on windows (i.e. the second run).

However, fewer larger files have a higher chance of being continuously and thus read with less head movements, if not on an SSD. Also, depending on sector/cluster size, there is less extra junk read from the hard disk. So there will be always a slight advantage for packed paks.