News:

Simutrans Sites
Know our official sites. Find tools and resources for Simutrans.

Performance impact of pak file grouping

Started by Ranran(retired), April 04, 2019, 10:36:41 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Ranran(retired)

Regarding loading time issues:

Why doesn't 128.Britain-Ex merge pak files?
That makes the initial boot time much faster.
It seems that former 128.Britain-Ex was doing file merging.
ひめしという日本人が開発者達の助言を無視して自分好みの機能をextendedに"強引に"実装し、
コードをぐちゃぐちゃにしてメンテナンスを困難にし(とりわけ道路と建物関連)、
挙句にバグを大量に埋め込み、それを知らんぷりして放置し(隠居するなどと言って)別のところに逃げ隠れて自分のフォーク(OTRP)は開発を続けている
その事実と彼の無責任さに日本人プレイヤーは目を向けるべき。らんらんはそれでやる気をなくした(´・ω・`)
他人の振り見て我が振り直せ。ひめしのようにならないために、らんらんが生み出したバグや問題は自分で修正しなくちゃね(´・ω・`)

jamespetts

Quote from: Ranran on April 04, 2019, 10:36:41 AM
Regarding loading time issues:

Why doesn't 128.Britain-Ex merge pak files?
That makes the initial boot time much faster.
It seems that former 128.Britain-Ex was doing file merging.

The use of individual pak files is due to the way in which the Linux makefile for the pakset works, and the nightly pakset build comes from Linux. When in the old days I used to distribute a manually compiled version of the pakset, I used Windows, and the Python script in Windows is configured to build merged .pak files.

I did not realise that the pakset loading time was affected by which of the two were used. How much of a difference does this make? When I load Simutrans myself, I tend to use the merged pak files as it is easier for me to compile locally than download.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Ranran(retired)

#2
Quote from: jamespetts on April 04, 2019, 10:40:40 AMHow much of a difference does this make?
There is this difference in HDD. (´・ω・`)

Pak128.Britain-EX (normal)


Pak128.Britain-EX (merged ver.)



However, it is very fast if the cache is left next time you boot.
This is the same with standard.
However, it is strange that extended takes a very long time to start loading compared to standard. It is the same even if it installs in SSD. (pak loading time will be shorter)




Reference information:

paksetfolder sizepaksloading time
pak128402MB569sec
pak128.german280MB175026sec
pak192.comic470MB241930sec

EDIT:
The merged experiment britain pakset consists of the following 32 paks.
https://i.imgur.com/ttVcUXc.png
This is based on pak128Britain.bat in the source folder.

Unmerged pak128.Britain-EX has 4500 over pak files. Thus I think it is the pakset with the longest startup time in Simutrans.
ひめしという日本人が開発者達の助言を無視して自分好みの機能をextendedに"強引に"実装し、
コードをぐちゃぐちゃにしてメンテナンスを困難にし(とりわけ道路と建物関連)、
挙句にバグを大量に埋め込み、それを知らんぷりして放置し(隠居するなどと言って)別のところに逃げ隠れて自分のフォーク(OTRP)は開発を続けている
その事実と彼の無責任さに日本人プレイヤーは目を向けるべき。らんらんはそれでやる気をなくした(´・ω・`)
他人の振り見て我が振り直せ。ひめしのようにならないために、らんらんが生み出したバグや問題は自分で修正しなくちゃね(´・ω・`)

jamespetts

Thank you for the demonstration. I have split this from the path explorer development topic as that is quite separate.

This is interesting. I am not sure what is behind this. It will take some considerable effort to reconfigure the makefile to provide for merged .pak files, so it may be some time before I am able to look into this.

If anyone would like to look into this in the meantime (either as to how to modify the makefile or why this makes a difference to performance in the first place), that would be most useful.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Vladki

The delay before loading starts may be due to another bug, you have to move mouse after choosing pakset to load.

jamespetts

Quote from: Vladki on April 07, 2019, 12:02:23 PM
The delay before loading starts may be due to another bug, you have to move mouse after choosing pakset to load.

This is, I understand, an issue from Standard.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

prissi

I hardly doubt this, since I mainly develop on a laptop. So after a mouseclick on the trackpad buttons (real button) the mouse will not move at all, but still load the pakset immediately.

The slow speed is simply from the overhead of the OS/libraries, i.e. getting a new file handle, requesting memory for it, loading etc. However, since after first time loading the file informations are still in the cache, the second laoding will be as fast as with fewer, larger files.

Ranran(retired)

Quote from: jamespetts on April 07, 2019, 12:05:27 PMQuote from: Vladki on Today at 12:02:23 PM
The delay before loading starts may be due to another bug, you have to move mouse after choosing pakset to load.

This is, I understand, an issue from Standard.
I think this issue is unique to extended. Standard starts loading immediately without doing anything.
And, as Vladki points out, when I operate the mouse, loading may start in extended pakset loading.

Quote from: prissi on April 07, 2019, 12:20:06 PMThe slow speed is simply from the overhead of the OS/libraries, i.e. getting a new file handle, requesting memory for it, loading etc.
Does the decrease in the number of files due to merging mean that the loading time at the first startup simutrans can be reduced with certainty?
ひめしという日本人が開発者達の助言を無視して自分好みの機能をextendedに"強引に"実装し、
コードをぐちゃぐちゃにしてメンテナンスを困難にし(とりわけ道路と建物関連)、
挙句にバグを大量に埋め込み、それを知らんぷりして放置し(隠居するなどと言って)別のところに逃げ隠れて自分のフォーク(OTRP)は開発を続けている
その事実と彼の無責任さに日本人プレイヤーは目を向けるべき。らんらんはそれでやる気をなくした(´・ω・`)
他人の振り見て我が振り直せ。ひめしのようにならないために、らんらんが生み出したバグや問題は自分で修正しなくちゃね(´・ω・`)

prissi

The loading time issue is much more severe on windows (although my last linux profiling was a long time ago). But it seems Linux file system driver loads the entire directory when opening it, and hence access to more files is almost as fast as if cached on windows (i.e. the second run).

However, fewer larger files have a higher chance of being continuously and thus read with less head movements, if not on an SSD. Also, depending on sector/cluster size, there is less extra junk read from the hard disk. So there will be always a slight advantage for packed paks.

Vladki

I have done some tests. On my computer, with empty cache, loading pak128-britain-ex takes 1 minute 20 seconds.  (But only 4 seconds if cached).
I have modified the Makefile using the code from pak128.CS (which inherited Makefile from pak128) - to make one pak file per directory. So the result is 41 files instead of 4887, and the loading time dropped to 20 seconds. (Cached load is 4 seconds).  So this is indeed worth doing.

https://github.com/jamespetts/simutrans-pak128.britain/pull/109

jamespetts

Thank you for testing: that is very helpful. Now incorporated.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.