Started by jamespetts, December 23, 2019, 11:51:35 PM
0 Members and 1 Guest are viewing this topic.
Quote from: jamespetts on December 23, 2019, 11:51:35 PMIf anyone knows of any suitable multi-threaded compression libraries with a compatible licence (Simutrans uses the Artistic Licence, so, sadly, the GPL will not be compatible unless a pre-compiled library be used; but the LGPL should be fine), or (even better) just a multi-threaded implementation of the current compression algorithm, that would be very helpful.
QuoteThese are used for vpn and filesystem (zfs) compression, so speed is most important for them, and compression ratio still quite good.
QuoteRegarding pedestrians, as they are purely decorative, wouldn't it be good to switch them off completely for very large games? I also turn off water animation.
Quote from: freddyhayward on January 15, 2020, 03:00:39 AMWhat if a small optional 'stop tax' (say 10.00c per stop, per month, separate from maintainence) were introduced via settings to encourage players to build fewer stops, particularly within local transport networks. A player building a 10km bus line might then choose to build 10 stops, 1 per km, rather than 20-30. Even better, the money might eventually pay off the public service debt!
Quote from: jamespetts on January 15, 2020, 01:16:22 AMwhich ultimately points to "deflate", which consumes 90.73% of time. Loading, by contrast, takes a fraction of the time taken by saving.
Quote from: jamespetts on January 15, 2020, 01:16:22 AMIn the meantime, the rule of thumb would seem to be that 400-500 towns is probably the most that should be generated for an online game expected to continue to the modern era and result in a well developed map with plentiful local passenger transport.
Quote from: DrSuperGood on January 15, 2020, 05:44:41 AMPeople generally are not building them more than 1 per km due to the distance between stops being too small for decent average speed.
Quote from: jamespetts on January 16, 2020, 02:30:26 AMDr. Supergood - thank you for the suggestion regarding the buffer size. May I ask where in zlib's API this can be modified?
Quote from: jamespetts on January 16, 2020, 02:30:26 AMI have just pushed a modification to this system (which is optional and can be disabled in simuconf.tab) to use a relative system rather than an absolute system. In this system, the three categories of Standard are retained, but their boundaries are determined by the town's rank in population size. The top ranked town will always be a "capital" (this is hard-coded to avoid rounding errors meaning that no town counts as a "capital" in games with few towns). Simuconf.tab settings can then set the percentage rank above which a town will be a "city" and a "capital", with values of 20% and 2% respectively set in the default simuconf.tab (the actual defaults if no values be specified are 0, which reverts to the old behaviour using the absolute sizes). ...This should help our town size distribution to retain an approximately Zipfian form.
Quote from: Freahk on January 15, 2020, 11:02:52 AMI don't know about bridgewater-brunel but in the stephenson-siemens game there are quite dense bus networks in some cities with stop spacings around 625m which, in combination with express lines competed other players bus services within cities quite well and increased the transported/passenger stat ratio in the townhalls quite a bit.
Quote from: Vladki on January 16, 2020, 04:03:44 PMAlso note the real-world density of stops. I have measured the path I commute daily, and distances between stops are from 150 m to almost 1 km. Average is somewhere at 400-500 m. And as we aim for reality, we should expect that inner city transport may have stops at similar density.
Quote from: prissi on January 16, 2020, 05:45:47 AMTo elaborate on DrSuperGood, Simutrans compresses already in the background, i.e. one buffer is written to while the already filled buffer is compressed. Increasing the size means a longer delay before starting to compress, but then longer continutation at max writing speed. But many servers (usually virtual ones) do not profit from that at all, since the only offer a single one. Only on the next tier four core are standard and thus here it would help.
//compression$ time bzip2 binary.svereal 0m12.544suser 0m12.016ssys 0m0.438s//decompression$ time bzip2 -d binary.sve.bz2real 0m5.882suser 0m5.047ssys 0m0.766s
//compression$ time gzip binary.svereal 0m6.170suser 0m6.094ssys 0m0.078s//decompression$ time gunzip binary.sve.gzreal 0m0.739suser 0m0.656ssys 0m0.094s
//compression$ time zstd -3 binary.svebinary.sve : 26.66% (98183317 => 26175069 bytes, binary.sve.zst)real 0m1.042suser 0m0.891ssys 0m0.125s//decompression$ time zstd -d binary.sve.zstbinary.sve.zst : 98183317 bytesreal 0m0.445suser 0m0.281ssys 0m0.156s
$ time pzstd -3 -p 2 binary.sve//compressionbinary.sve : 26.70% (98183317 => 26213185 bytes, binary.sve.zst)real 0m0.474suser 0m0.844ssys 0m0.141s//decompression$ time pzstd -d -p 2 binary.sve.zstbinary.sve.zst : 98183317 bytesreal 0m0.170suser 0m0.297ssys 0m0.094s
//compression$ time pzstd -3 -p 4 binary.svebinary.sve : 26.70% (98183317 => 26213185 bytes, binary.sve.zst)real 0m0.398suser 0m1.344ssys 0m0.109s//decompression$ time pzstd -d -p 4 binary.sve.zstbinary.sve.zst : 98183317 bytesreal 0m0.145suser 0m0.297ssys 0m0.188s
//compression$ time lz4 binary.sveCompressed filename will be : binary.sve.lz4Compressed 98183317 bytes into 44430881 bytes ==> 45.25%real 0m0.345suser 0m0.250ssys 0m0.094s//decompression$ time lz4 -d binary.sve.lz4Decoding file binary.sveSuccessfully decoded 98183317 bytesreal 0m0.276suser 0m0.141ssys 0m0.125s
Quote from: Freahk on January 17, 2020, 05:07:25 PMIf I had the save I could throw a few compresion algorithms at it and see what happens.It's up to you deciding if that would be of assistance to you or not.
Quote from: Qayyum on January 17, 2020, 08:37:01 AMI pick ZSTD -3, Time too short leads to issues regarding memory.
Quote from: prissi on January 17, 2020, 03:25:48 AMSo while Zlib delivers 11 MB/s Snappy gives 413 MB/s, but size is 1.5-2 as big. However, even 19MB/s is more than 100 Mbit/s and faster than most clients can recieve over longer distances. (This is a 32bit built with -O3 and no assertions though!)
time gzip -1 bb-17-jan-2020.sve.uncompressed real 0m54,607suser 0m53,413ssys 0m1,168s
time gunzip bb-17-jan-2020.sve.uncompressed.gz real 0m29,111suser 0m28,022ssys 0m1,088s
time lz4 -1 ./bb-17-jan-2020.sve.uncompressedreal 0m14,402suser 0m10,699ssys 0m1,420s
time lz4 ./bb-17-jan-2020.sve.uncompressed.lz4real 0m35,982suser 0m4,392ssys 0m1,291s
time zstd -1 ./bb-17-jan-2020.sve.uncompressedreal 0m15,672suser 0m16,081ssys 0m0,899s
time zstd -d ./bb-17-jan-2020.sve.uncompressed.zst real 0m11,133suser 0m8,162ssys 0m1,088s
time pzstd -1 -p2 ./bb-17-jan-2020.sve.uncompressedreal 0m11,082suser 0m16,794ssys 0m1,139s
time pzstd -1 -d -p2 ./bb-17-jan-2020.sve.uncompressed.zst real 0m6,517suser 0m8,834ssys 0m1,899s
time pzstd -1 -p4 ./bb-17-jan-2020.sve.uncompressedreal 0m6,582suser 0m18,278ssys 0m1,399s
time pzstd -1 -d -p4 ./bb-17-jan-2020.sve.uncompressed.zst real 0m4,334suser 0m9,052ssys 0m2,096s
time zstd -1 -T2 ./bb-17-jan-2020.sve.uncompressedreal 0m11,130suser 0m17,928ssys 0m0,981s
time zstd -1 -T4 ./bb-17-jan-2020.sve.uncompressedreal 0m6,586suser 0m18,287ssys 0m1,139s
time zstd -d -T4 ./bb-17-jan-2020.sve.uncompressed.zst real 0m10,966suser 0m7,833ssys 0m1,144s
Quote from: jamespetts on January 18, 2020, 12:08:00 AMDr. Supergood - it would indeed be splendid if someone had the time and knowledge to write code for streaming the saved games. Would anyone like to volunteer?
Quote from: DrSuperGood on January 18, 2020, 01:16:37 AMThe changes needed are not trivial. The entire net code part is a mess with respect to such features. Ideally it should be made asynchronous however this is difficult due to standard's requirements to support single thread only builds. This means one cannot use dedicated receive and transmission threads, instead having to rely on polling.While at it one could also implement other features... Like more than 1 player joining from the same save file. Possibly even allowing the game to progress during the transfer depending on server administrator preferences.Ability to query server status asynchronously. Currently unresponsive servers cause the client to freeze until a timeout occurs.The solution is not as simple as piping/nesting the compression results via socket since the server also has to writing out the results to file. While saving it would also have to read this written out data and send it to the client(s) that are joining.
Quote from: jamespetts on January 18, 2020, 01:08:58 AMIs the one from August any use?
fd->gzfp = gzopen(filename, "wb2"); gzbuffer(fd->gzfp, 65536);
Quote from: Freahk on January 18, 2020, 02:43:18 AMThe default zipped is taking *only* 225 seconds. >> Bridgewater brunnel must be running on a potato to be timing out clients at 10 mins and still not sending the file...
Quote from: TurfIt on January 18, 2020, 04:39:44 AMWhyTF does Extended crash upon loading the pakset if you've moved the mouse while it's loading?? ?? What sorta bug you got in there...
Quote from: prissi on January 18, 2020, 01:43:16 PMIf I read some more on zstd, It seems that the zstd can read also zlib file, so it would have backward compatibility without any extra effort. That is nice.I wonder though how TurfIt gets more than four times of the datarate on writing binary files. Does experimental dump large tables at one point?
Quote from: jamespetts on January 18, 2020, 11:41:22 AMAlso, Freahk - how many would you need?
QuoteHow different from each other would they need to be; would a set of backups a few hours apart work?
Quote from: Freahk on January 18, 2020, 06:01:29 PMI don't know but I will give it a try.
Quote from: jamespetts on January 18, 2020, 11:41:22 AMTurfIt - thank you very much for that: that is most interesting. Are you able to share the code that you used to integrate zstd?
Quote from: DrSuperGood on January 18, 2020, 08:04:28 AMThis has been raised a lot. It is very low on memory so the server performs very poorly due to page faults. I would not be surprised if saving causes page thrashing.
Quote from: prissi on January 18, 2020, 01:43:16 PMIf I read some more on zstd, It seems that the zstd can read also zlib file, so it would have backward compatibility without any extra effort. That is nice.
Quote from: prissi on January 18, 2020, 01:43:16 PMI wonder though how TurfIt gets more than four times of the datarate on writing binary files. Does experimental dump large tables at one point?
Quote from: TurfIt on January 18, 2020, 10:08:44 PMZstandard.
Quote from: jamespetts on January 19, 2020, 05:39:56 PMDr. Supergood - can I check whether you are still interested in working on this? It would help me to know whether to spend my time on this in the near future or whether to spend the same time working on bug fixes and other enhancements.
Quote from: prissi on January 19, 2020, 10:54:50 AM(My laptop was the most expensive model from Panasonic in 2017, about $3000. CPU is special made model with 6 cores and 2.9 GHz nominal which does not exist in databases. Resource monitor has 6 threads up to 3.6 GHz speedup. So I was a little surprised about a factor of four. But this gets offtopic fast.)
Quote from: jamespetts on January 25, 2020, 06:35:54 PMIt is of course up to Dr. Supergood what he would like to spend his time doing; but I will proceed when I am able to integrate this, if I can overcome the library issue.
Quote from: DrSuperGood on January 26, 2020, 03:26:32 AMSo to get the task straight. The existing zip library (zlib?) is to be entirely replaced by Zstd which is to operate in multi threaded mode? And the changes have to be applied to both makefile (for Linux) and VisualStudio?
Quote from: The licenceBSD LicenseFor Zstandard softwareCopyright (c) 2016-present, Facebook, Inc. All rights reserved.Redistribution and use in source and binary forms, with or without modification,are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name Facebook nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" ANDANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AREDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FORANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ONANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THISSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.