Started by jamespetts, December 23, 2019, 11:51:35 PM
0 Members and 1 Guest are viewing this topic.
time gzip -1 bb-17-jan-2020.sve.uncompressed real 0m54,607suser 0m53,413ssys 0m1,168s
time gunzip bb-17-jan-2020.sve.uncompressed.gz real 0m29,111suser 0m28,022ssys 0m1,088s
time lz4 -1 ./bb-17-jan-2020.sve.uncompressedreal 0m14,402suser 0m10,699ssys 0m1,420s
time lz4 ./bb-17-jan-2020.sve.uncompressed.lz4real 0m35,982suser 0m4,392ssys 0m1,291s
time zstd -1 ./bb-17-jan-2020.sve.uncompressedreal 0m15,672suser 0m16,081ssys 0m0,899s
time zstd -d ./bb-17-jan-2020.sve.uncompressed.zst real 0m11,133suser 0m8,162ssys 0m1,088s
time pzstd -1 -p2 ./bb-17-jan-2020.sve.uncompressedreal 0m11,082suser 0m16,794ssys 0m1,139s
time pzstd -1 -d -p2 ./bb-17-jan-2020.sve.uncompressed.zst real 0m6,517suser 0m8,834ssys 0m1,899s
time pzstd -1 -p4 ./bb-17-jan-2020.sve.uncompressedreal 0m6,582suser 0m18,278ssys 0m1,399s
time pzstd -1 -d -p4 ./bb-17-jan-2020.sve.uncompressed.zst real 0m4,334suser 0m9,052ssys 0m2,096s
time zstd -1 -T2 ./bb-17-jan-2020.sve.uncompressedreal 0m11,130suser 0m17,928ssys 0m0,981s
time zstd -1 -T4 ./bb-17-jan-2020.sve.uncompressedreal 0m6,586suser 0m18,287ssys 0m1,139s
time zstd -d -T4 ./bb-17-jan-2020.sve.uncompressed.zst real 0m10,966suser 0m7,833ssys 0m1,144s
Quote from: jamespetts on January 18, 2020, 12:08:00 AMDr. Supergood - it would indeed be splendid if someone had the time and knowledge to write code for streaming the saved games. Would anyone like to volunteer?
Quote from: DrSuperGood on January 18, 2020, 01:16:37 AMThe changes needed are not trivial. The entire net code part is a mess with respect to such features. Ideally it should be made asynchronous however this is difficult due to standard's requirements to support single thread only builds. This means one cannot use dedicated receive and transmission threads, instead having to rely on polling.While at it one could also implement other features... Like more than 1 player joining from the same save file. Possibly even allowing the game to progress during the transfer depending on server administrator preferences.Ability to query server status asynchronously. Currently unresponsive servers cause the client to freeze until a timeout occurs.The solution is not as simple as piping/nesting the compression results via socket since the server also has to writing out the results to file. While saving it would also have to read this written out data and send it to the client(s) that are joining.
Quote from: jamespetts on January 18, 2020, 01:08:58 AMIs the one from August any use?
fd->gzfp = gzopen(filename, "wb2"); gzbuffer(fd->gzfp, 65536);
Quote from: Freahk on January 18, 2020, 02:43:18 AMThe default zipped is taking *only* 225 seconds. >> Bridgewater brunnel must be running on a potato to be timing out clients at 10 mins and still not sending the file...
Quote from: TurfIt on January 18, 2020, 04:39:44 AMWhyTF does Extended crash upon loading the pakset if you've moved the mouse while it's loading?? ?? What sorta bug you got in there...
Quote from: prissi on January 18, 2020, 01:43:16 PMIf I read some more on zstd, It seems that the zstd can read also zlib file, so it would have backward compatibility without any extra effort. That is nice.I wonder though how TurfIt gets more than four times of the datarate on writing binary files. Does experimental dump large tables at one point?
Quote from: jamespetts on January 18, 2020, 11:41:22 AMAlso, Freahk - how many would you need?
QuoteHow different from each other would they need to be; would a set of backups a few hours apart work?
Quote from: Freahk on January 18, 2020, 06:01:29 PMI don't know but I will give it a try.
Quote from: jamespetts on January 18, 2020, 11:41:22 AMTurfIt - thank you very much for that: that is most interesting. Are you able to share the code that you used to integrate zstd?
Quote from: DrSuperGood on January 18, 2020, 08:04:28 AMThis has been raised a lot. It is very low on memory so the server performs very poorly due to page faults. I would not be surprised if saving causes page thrashing.
Quote from: prissi on January 18, 2020, 01:43:16 PMIf I read some more on zstd, It seems that the zstd can read also zlib file, so it would have backward compatibility without any extra effort. That is nice.
Quote from: prissi on January 18, 2020, 01:43:16 PMI wonder though how TurfIt gets more than four times of the datarate on writing binary files. Does experimental dump large tables at one point?
Quote from: TurfIt on January 18, 2020, 10:08:44 PMZstandard.
Quote from: jamespetts on January 19, 2020, 05:39:56 PMDr. Supergood - can I check whether you are still interested in working on this? It would help me to know whether to spend my time on this in the near future or whether to spend the same time working on bug fixes and other enhancements.
Quote from: prissi on January 19, 2020, 10:54:50 AM(My laptop was the most expensive model from Panasonic in 2017, about $3000. CPU is special made model with 6 cores and 2.9 GHz nominal which does not exist in databases. Resource monitor has 6 threads up to 3.6 GHz speedup. So I was a little surprised about a factor of four. But this gets offtopic fast.)
Quote from: jamespetts on January 25, 2020, 06:35:54 PMIt is of course up to Dr. Supergood what he would like to spend his time doing; but I will proceed when I am able to integrate this, if I can overcome the library issue.
Quote from: DrSuperGood on January 26, 2020, 03:26:32 AMSo to get the task straight. The existing zip library (zlib?) is to be entirely replaced by Zstd which is to operate in multi threaded mode? And the changes have to be applied to both makefile (for Linux) and VisualStudio?