when simutrans can load images over 65534?

[Manche]

hi, I am a collector nightly build information (for some years).
(a organized nightly build information posted on my website in Japanese.)

I tried to use build r7411, that could load images over 65534.
but build r7252, that couldn't load images over 65534.

I hypothesized two things,"simutrans don't load surplus images excess amount of 65534"
or "simutrans can load 65534 from r7411".

please give me when simutras can load images over 65534.

[DrSuperGood]

65534 what? Is this sprite pixel area? Or is it number of separate sprite images? It sounds like a limit caused by using a uint16_t variable/parameter.

The nightly builds follow the SVN closely with exception of a few months ago where a make error resulted in Windows nightly failing to build. Each commit to the SVN contains a brief description what was changed (although I agree it often lacks detail).

[Manche]

65534 what? Is this sprite pixel area? Or is it number of separate sprite images? It sounds like a limit caused by using a uint16_t variable/parameter.

The nightly builds follow the SVN closely with exception of a few months ago where a make error resulted in Windows nightly failing to build. Each commit to the SVN contains a brief description what was changed (although I agree it often lacks detail).

I don't know  sprite pixel area and number of separate sprite images.
I only know 'can use png file(s) less than 65534'.

sure, I know '65534' is max length of uint16_t.
Japanese developer rewrite uint16_t to uint32_t because of to enable use add-on(s) more than 65534.

[Ters]

65534 what?

Images.

[Markohs]

This was discussed on a private developers forum post that maybe can be a good idea to move to public discussion. It was agreed to expand the number of images but no final action was taken.

switching that 16 bit id to 32 bit is indeed a solution, but raises memory consumption in-game.

http://forum.simutrans.com/index.php?topic=10105.0

[Manche]

This was discussed on a private developers forum post that maybe can be a good idea to move to public discussion. It was agreed to expand the number of images but no final action was taken.

switching that 16 bit id to 32 bit is indeed a solution, but raises memory consumption in-game.

http://forum.simutrans.com/index.php?topic=10105.0

I can't see that thread with message 'Specified thread or bulletin board, seems not found or is not allowed to view.'.

[Markohs]

Yep, it's private.

To make it short, we discussed two aproaches to solve this issue, but I think switching the uint16 to uint32, was accepted as a solution. We'll see what prissi has to say about this, but I guess he will just change it to uint32 soon.

[Manche]

I agree switch uint16 to uint32, too. 
sometimes, I was plagued for simutrans can't load image over 65534, I couldn't keep play the same save data (also, played whet is over two years).

switch uint16 to uint32, how many issues are happen?

[DrSuperGood]

Do any paksets have over 65534 sprite images in them? Or is this some limit on the rasterizer algorithm?

I have played some pretty well developed games in both experimental (which might have already changed that for all I know) and pak64 and never encountered such a limit.

[Combuijs]

No, paksets themselves stay within that limit for the moment. But if you add a lot of addons (basically all you can get in the same format) you run out of images.

Every image is only stored once, so duplicates are filtered out.

[Ters]

switch uint16 to uint32, how many issues are happen?

Two things first: The game will use more memory, and somewhat related, it will run more slowly. (It will also run slower simply because you have many images.)
At some point, Simutrans might start crashing because it runs out of memory. Switching to uint32 means they can have more images than regular Simutrans has room for in memory (even virtual, actual amount of RAM on computer doesn't directly matter).

I have played some pretty well developed games in both experimental (which might have already changed that for all I know) and pak64 and never encountered such a limit.

You seem like a logistics-minded type of player, given your focus on industry demand and supply. There are players who just want thing to look pretty, to the point of obsession, and don't care much about how vehicles run, as long as they move and bring some life to the scenery they've created. And some are probably a mix of both. No two players seems to look at and treat Simutrans the same.

[Manche]

Two things first: The game will use more memory, and somewhat related, it will run more slowly. (It will also run slower simply because you have many images.)
At some point, Simutrans might start crashing because it runs out of memory. Switching to uint32 means they can have more images than regular Simutrans has room for in memory (even virtual, actual amount of RAM on computer doesn't directly matter).

there is no problem another two issues for here?

so I think solution of memory problem is 'prepare a 64bit edition a separate for traditional edition'.
64bit application can use memory more than 32bit one.
but preparing 64bit edition may be burden for developer and contributor.

Do any paksets have over 65534 sprite images in them? Or is this some limit on the rasterizer algorithm?

I have played some pretty well developed games in both experimental (which might have already changed that for all I know) and pak64 and never encountered such a limit.

you can have a face 65534 problem, if you add all of Japanese author(s) addon.
Japanese user made many to many addons.

incidentally, my always use pak64 with many Japanese vehicle(include some of other country vehicle,too).

[DrSuperGood]

so I think solution of memory problem is 'prepare a 64bit edition a separate for traditional edition'.
64bit application can use memory more than 32bit one.
but preparing 64bit edition may be burden for developer and contributor.

This has been discussed many times. The conclusion was that migrating to full 64bit would slow the game down more than the extra memory and faster fixed point calculations it would bring are worth. Specifically due to the larger instruction and pointer sizes. That said there should be 64 bit builds available as the game is not particularly coupled to 32bit (just 64bit is not supported). Experimental for example is distributed as 64bit.

Also some inline assembly code would need to be rewritten in order for the game to perform well. Experimental 64bit suffers from abysmal performance due to some graphic calls not being optimized with inline assembly.

There is a compromise which James pulled out for experimental due to the server game running into the standard 32bit memory limit. It is possible to compile a 32bit build with extended addressing mode which allows for >>4GB virtual memory size. In fact 32bit extended address processes can allocate more memory than most household PCs can allow. Since it is still 32bit the performance is very similar to normal 32bit builds.

you can have a face 65534 problem, if you add all of Japanese author(s) addon.
Japanese user made many to many addons.

incidentally, my always use pak64 with many Japanese vehicle(include some of other country vehicle,too).

How many of those sprites are actually in use at any given time? For example an old steam locomotive in 2050 is unlikely to be used unless for specific decorative reasons. The solution in this case would be to dynamically load sprites at run time rather than all sprites on load. This could add resources stalls but it would reduce the number of sprites massively as any sprite not actively in use would not be in memory.

This could be further extended to only visible sprites. The chances of someone having 65534 different sprites in view at any given time is probably impossible. Obviously the code would be more complicated for this and probably have some large overhead.

[Ters]

When I brought up that at some point Simutrans would run out of memory, I was thinking far ahead. 64-bit memory space and swapping images in and out memory are solutions to problems that is some distance away yet.

[prissi]

It is a little more effort needed than just make this a 32 bit pointer, as almost every object in the map stores this id and hence increase their size by 4 bytes. So it would have some effect on memory (like another 16 MB ram or so for a game with 4 million object (not much really). Also some routines need further changes. (I forgot if this was finished or not).

The only way to get over the maximum 65535 for now is putting two paksets in the same folder, like pak128.britain and pak128. However, all pak128 are not made to be mixed since their sizes do not match up.

pak64 has about 6372 images. That leaves place for 8000 different train cars. I am pretty sure the depot will be impossible to use before. (Not to mention that there are not 8000 cars on the japanese webseite, there are about 18 pages with 20 entires with on average less than 8 different cars in them, i.e. ~3000 entries) Not to mention that simutrans merges identical images (like left and right  or back views), so the actual limit is even higher.

Some other sets:
pak128.britain 31256
pak128.japan 4406
pak128 36932
pak96.comic 7166
pak192.comic 10575

Even with pak128 you need more than 3500 train cars (assuming all have 8 completely different view). For pak128 there are 14 pages, again with about 20 entries and assuming 16 different cars: Yes, if you put every thing into a game (ignoring that several are there twice or three times and are from many place and so on), you might be just able to reach this limit.

We also have 64 bit nightlies for Linux, since the compatibility on Linux with 32 bit is really really bad. (Why is the a Linux standard dristribution if nobody uses those components anymore.) But that has nothing to do with this pointer.

If someone shows us a sensible mix of addons to reach this number, we increase it.

[Ters]

Since some freight cars have more than 8 different images (4/8 empty, 4/8 freight A, 4/8 freight B, 4/8 freight C, ...), the number of cars required for pak128 to reach the limit is somewhat less. It's still a lot of vehicles. But there are perhaps other add-ons that use image slots like crazy. Buildings can with multiple seasons, layouts and animations require more images than vehicles.

[Markohs]

Just increment the id to 32bit, we are not in the year 95 anymore. The increase in memory usage is not huge, we will survive

[DrSuperGood]

The increase in memory usage is not huge, we will survive

It is not so much memory usage, but also performance decrease since extra bytes of memory have to be read etc. Apparently some people still run Simutrans on systems with quite restricted resources.

[Ters]

The number of bytes read is almost certainly more important than the number of bytes used, as images ids tend to hang around in besch objects, which there are fairly few of. Besch objects are however read very often.

[Junna]

I constantly run out of images. I'd say it is a pressing problem.

You can easily run out of images with a few building sets and vehicles (prissi mentions the Japanese website - you can easily get up to 65535 images assuming you have double-height ways, buildings and station facilities and so on, it's very easy. You won't even go half-way before you reach the point.)

[Markohs]

 I just looked on wiki and a intel i5 cache line it's 64 bytes wide, and core 2 duo, too. You are just discussing over 2 bytes. Older systems, can't even run Windows 7. Aren't you taking old systems compatibility a bit too far here? Any modern CPU won't really be much affected by this 2 bytes increase.

I doubt it will take a big difference in performance, really. But I didn't benchmarked it, so I might be wrong.

This is just my point of view, I respect that others don't see the things like I do, of course.

[DrSuperGood]

I just looked on wiki and a intel i5 cache line it's 64 bytes wide, and core 2 duo, too. You are just discussing over 2 bytes. Older systems, can't even run Windows 7. Aren't you taking old systems compatibility a bit too far here? Any modern CPU won't really be much affected by this 2 bytes increase.

The target specs for Simutrans I thought were somewhere between late Pentium 3 and Early Pentium 4. I have played on many servers in the last year odd that were powered off Pentium 4 processors. Although a few years old now, Intel i3,5 and 7 are still very modern processors.

Personally I agree with you and even believe migrating to 64bit would not be as bad as people make out (all the fixed point mathematics would be a lot faster and the graphic code should not be that much slower once re-optimized). However I think a lot of Simutrans users do not have 64bit OS.

[Markohs]

Well, I agree, but I was not really demanding 64 bit builds here, I was just expressing that turning image_id from 16-bit to 32, shoudn't be a big problem. A 64-bit build adds the problem that all pointers double size (from 4 byte, to , this might impact performance more than the image_id change.

[Spenk009]

Regarding "between late Pentium 3 & early Pentium 4: Wikipedia states that pentium 4s were introduced almost 15 years ago. I can't imagine many people regularly updating their games and running hardware that's at least 7 years old.

64bit aside, could we make a poll (maybe even site-wide) asking for peoples specs? I see few people on maps of 128² tiles and unaltered paks.

[Ters]

We have, or very recently had, a prominent player using AMD K6 (possibly K6-2) and Windows 2000 (or maybe he had just gotten Windows XP). We noticed this when the nightlies accidentally started requiring SSE. I think Simutrans is quite popular in countries were average wages are a quarter (or less) of what they are in the western world. (Although there are quite a few Windows XP machines around even in wealthy Norway. I even did Windows 98 support three years ago. Simutrans no longer supports that, if only because of Unicode.) Pentium IIIs and AMD K6-2s might be old even for them now, but dropping Pentium 4 will probably alienate many players. Players whos voice might not be heard on an English-speaking board.

[Markohs]

It's not that we drop support for that old processors. It's just that simutrans whould require *sightly* more memory and I/O bandwith if we turn the id from 16 to 32 bit. This won't break anything. I see also some machines with Windows XP here in Spain/Catalonia, ofc. But they are not really common, and remember even Microsoft has droped support for that OS. It's nothing spectacular, let's accept those machines are obsolete.

Comeon, it's just converting a number from 16-bit to 32-bit, this should not be so dramatic.

[DrSuperGood]

It probably is not as simple as changing a uint16 to unit32 in a declaration. Every piece of code that manipulates the variable would need to be revised to make sure that uint16 bottlenecks are not introduced elsewhere, that is unless it uses a special type definition of uint16 was used for everything that manipulates the images. For future maintainability it might be a good idea to bring in a custom type definition for that field so that future migrations are easier.

[Markohs]

It's not hard, I've changed it in the past, only some things need to be changed in simgraph.cc/h and not much more, iirc. maybe simimg.h too, but I remember that piece of code, was more or less well non-coupled with the rest of code, it was designed more or less okay.

[Ters]

It's not that we drop support for that old processors. It's just that simutrans whould require *sightly* more memory and I/O bandwith if we turn the id from 16 to 32 bit. This won't break anything. I see also some machines with Windows XP here in Spain/Catalonia, ofc. But they are not really common, and remember even Microsoft has droped support for that OS. It's nothing spectacular, let's accept those machines are obsolete.

Comeon, it's just converting a number from 16-bit to 32-bit, this should not be so dramatic.

If SImutrans stops running at playable speeds for some users, it is certainly dramatic for them. I can't believe that some people need threading to render Simutrans at acceptable speeds. When that is the case, the size if image ids might be just as important. Then there's the issue that Simutrans with SDL is very slow, which SDL2 apparently cures, without any good explaination that I'm aware of. Changing the size of image ids should only be done after someone with a slow system verifies that it works OK.

[Markohs]

I can't believe that some people need threading to render Simutrans at acceptable speeds.

That's maybe the root of the discussion. Believe me, it's quite a speedup for many people. And people with old computers like me (Core 2)

[Manche]

Supporting legacy machine is right.
but does not support newer machine is right?

and everybody are using legacy machine?

I guess a create item win9x(and 32bit machine) to nightly(also stable).

now, after all, uncertain when simutrans can load images over 65534?

[DrSuperGood]

Current support is obviously pretty critical. However generally current systems are backwards compatible so that is not a problem.

The problem comes down to system libraries and compilers. For example Simutrans uses a highly dated specification of C++ despite G++/Windows compilers/ARM compilers all being available for the latest specifications. The reason I have been told is so that old operating system version (which cannot run such compiler builds) need to be supported.

For the most part raising a 16bit field to 32bit on a 32bit architecture should be minimal even on oldish systems. Performance will be lowered but probably not in a critical way.

[Sarlock]

We can assume that players using older systems will be running a pak64 sized pakset and using smaller(ish) maps.  I wonder what the impact would be to a player using pak64 and a 256x256 map?  It may be a very small impact to them, with a huge upside to future Simutrans artists who like to combine images from multiple sources.

Then the next step will be to overcome the 2GB RAM barrier...

[Ters]

Supporting legacy machine is right.
but does not support newer machine is right?

With 16-bit image ids, both old and new machines will still work. Players just can't have as much eye candy. With 32-bit image ids, old machines might not be able to run Simutrans anymore, while those with new machines can stuff the game full of more eye candy than they can ever have time to download.

So it's not about supporting old machines versus supporting new machines, but about whether it is worth risking support for old machines just to allow someone else to install more optional add-ons, most of which only offer greater visual diversity.

[Sarlock]

Though it is important to note that the two largest paksets are already half of the limit.  Add some custom addons and they're easily over 40,000 images.

I think the best question to answer at this point is what the actual impact to an old machine running Simutrans would be.  I suspect it's not great.