64-bit version of Experimental?

[Carl]

Would it be possible for Experimental to support occupancy of more than 2GB of system RAM? I've been pushing near this limit on a couple of maps. While one can use certain third-party tools to unlock an executable for these purposes -- seemingly without error -- if it is a relatively simple change to make then it could be useful to have this supported natively.

This is, I suppose, a selfish request, but it might also be a good exercise in future-proofing the program. It's not at all unusual for systems to have upwards of 8GB of ram now, and this will only increase in the future. While an ordinary map is not, under current conditions, likely to get close to the 2GB limit, the fact that some uses of the program can get to this limit is hopefully some reason to consider making the change.

[jamespetts]

Hmm, yes, I noticed that I had memory issues running your large game in debug mode. I notice that Microsoft Visual Studio Express 2012 desktop is out and has a 64-bit mode built in. I am downloading that and will see how that works with Simutrans-Experimental.

[Carl]

Thanks for looking into this, James.

[jamespetts]

It is rather helpful that you prompted me to do so, actually, as the MS Visual Studio Express 2012 is, in my tests so far, easier to use and more responsive than the 2010 edition that I have been using so far.

[Sarlock]

Would it be possible for Experimental to support occupancy of more than 2GB of system RAM? I've been pushing near this limit on a couple of maps. While one can use certain third-party tools to unlock an executable for these purposes -- seemingly without error -- if it is a relatively simple change to make then it could be useful to have this supported natively.

This is, I suppose, a selfish request, but it might also be a good exercise in future-proofing the program. It's not at all unusual for systems to have upwards of 8GB of ram now, and this will only increase in the future. While an ordinary map is not, under current conditions, likely to get close to the 2GB limit, the fact that some uses of the program can get to this limit is hopefully some reason to consider making the change.

Glad for this request, I have had the same thought.  As Simutrans matures, especially Experimental, it is becoming a very powerful tool to simulate a large real-world network.  I am working on something myself, to the same idea as your GB map, though it has not progressed enough for me to release a first update.

In a few years, 2GB will seem small.  I just got a new computer last week that has 16 GB installed.  Simutrans-Exp runs very nicely on this system.

I learned to program BASIC on a computer with 16k RAM.  1 million times less memory...

[kierongreen]

Now that would have been an interesting exercise - trying to write a reasonable transport simulator for the 1980s era home computers. I don't think BASIC would have been a realistic option if you were to get at all decent performance, and I suspect map sizes more than 64x64 tiles would have been completely out of the question...

[waerth]

I actually patched simutrans with the 4gb limit as I like to play on max size maps (about 4000x4000 or so the program does not allow bigger :( ) and with 2000 towns on the map (I am nuts I know!!) I have been playing with that for some time now. I really hope we will be able to create bigger maps :p

Maybe an option is to point people to this patch from where they download the experimental .exe? Saying that if you have more than 4 gb of Ram it is recommended to patch simutrans with that as it will stop crashes from happening (when it reaches to about 1800 mb of memory it crashes without the patch).

W

[Carl]

That's one option, but for the purposes of e.g. bug reports and sharing saves, it would be preferable if there was an officially supported 64-bit version.

[jamespetts]

Attempting to compile in 64-bits with Visual Studio 2012 gives me this error for a multi-threaded build:

Code Select Expand


1>  loadsave.cc
1>dataobj\loadsave.cc(527): warning C4267: 'argument' : conversion from 'size_t' to 'unsigned int', possible loss of data
1>dataobj\loadsave.cc(530): warning C4267: 'argument' : conversion from 'size_t' to 'int', possible loss of data
1>dataobj\loadsave.cc(615): warning C4267: 'argument' : conversion from 'size_t' to 'int', possible loss of data
1>dataobj\loadsave.cc(620): warning C4267: 'argument' : conversion from 'size_t' to 'unsigned int', possible loss of data
1>dataobj\loadsave.cc(923): warning C4267: 'argument' : conversion from 'size_t' to 'const int', possible loss of data
1>dataobj\loadsave.cc(986): warning C4267: 'argument' : conversion from 'size_t' to 'const int', possible loss of data
1>weg.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>simplay.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>simview.obj : error LNK2019: unresolved external symbol __imp_pthread_mutex_lock referenced in function "public: void __cdecl std::_Wrap_alloc<class std::allocator<char> >::construct<char *,char * &>(char * *,char * &)" (??$construct@PEADAEAPEAD@?$_Wrap_alloc@V?$allocator@D@std@@@std@@QEAAXPEAPEADAEAPEAD@Z)
1>tunnel.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>wayobj.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>leitung2.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>loadsave.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>powernet.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>simgraph16.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>crossing.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>freelist.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>gebaeude.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>label.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_lock
1>weg.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>simplay.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>simview.obj : error LNK2019: unresolved external symbol __imp_pthread_mutex_unlock referenced in function "public: void __cdecl std::_Wrap_alloc<class std::allocator<char> >::construct<char *,char * &>(char * *,char * &)" (??$construct@PEADAEAPEAD@?$_Wrap_alloc@V?$allocator@D@std@@@std@@QEAAXPEAPEADAEAPEAD@Z)
1>tunnel.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>wayobj.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>leitung2.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>loadsave.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>powernet.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>simgraph16.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>crossing.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>freelist.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>gebaeude.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>label.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_unlock
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzReadOpen referenced in function "public: bool __cdecl loadsave_t::rd_open(char const *)" (?rd_open@loadsave_t@@QEAA_NPEBD@Z)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzReadClose referenced in function "public: char const * __cdecl loadsave_t::close(void)" (?close@loadsave_t@@QEAAPEBDXZ)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzRead referenced in function "private: unsigned __int64 __cdecl loadsave_t::read(void *,unsigned __int64)" (?read@loadsave_t@@AEAA_KPEAX_K@Z)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzWriteOpen referenced in function "public: bool __cdecl loadsave_t::wr_open(char const *,enum loadsave_t::mode_t,char const *,char const *,char const *)" (?wr_open@loadsave_t@@QEAA_NPEBDW4mode_t@1@000@Z)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzWrite referenced in function "private: unsigned __int64 __cdecl loadsave_t::write(void const *,unsigned __int64)" (?write@loadsave_t@@AEAA_KPEBX_K@Z)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzWriteClose referenced in function "public: char const * __cdecl loadsave_t::close(void)" (?close@loadsave_t@@QEAAPEBDXZ)
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_attr_init referenced in function "public: void __cdecl loadsave_t::set_buffered(bool)" (?set_buffered@loadsave_t@@QEAAX_N@Z)
1>simview.obj : error LNK2001: unresolved external symbol __imp_pthread_attr_init
1>simworld.obj : error LNK2001: unresolved external symbol __imp_pthread_attr_init
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_attr_destroy referenced in function "public: void __cdecl loadsave_t::set_buffered(bool)" (?set_buffered@loadsave_t@@QEAAX_N@Z)
1>simview.obj : error LNK2001: unresolved external symbol __imp_pthread_attr_destroy
1>simworld.obj : error LNK2001: unresolved external symbol __imp_pthread_attr_destroy
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_attr_setdetachstate referenced in function "public: void __cdecl loadsave_t::set_buffered(bool)" (?set_buffered@loadsave_t@@QEAAX_N@Z)
1>simview.obj : error LNK2001: unresolved external symbol __imp_pthread_attr_setdetachstate
1>simworld.obj : error LNK2001: unresolved external symbol __imp_pthread_attr_setdetachstate
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_create referenced in function "public: void __cdecl loadsave_t::set_buffered(bool)" (?set_buffered@loadsave_t@@QEAAX_N@Z)
1>simview.obj : error LNK2001: unresolved external symbol __imp_pthread_create
1>simworld.obj : error LNK2001: unresolved external symbol __imp_pthread_create
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_join referenced in function "public: void __cdecl loadsave_t::set_buffered(bool)" (?set_buffered@loadsave_t@@QEAAX_N@Z)
1>simworld.obj : error LNK2001: unresolved external symbol __imp_pthread_join
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_mutex_init referenced in function "public: void __cdecl loadsave_t::set_buffered(bool)" (?set_buffered@loadsave_t@@QEAAX_N@Z)
1>simgraph16.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_init
1>simview.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_init
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_mutex_destroy referenced in function "public: void __cdecl loadsave_t::set_buffered(bool)" (?set_buffered@loadsave_t@@QEAAX_N@Z)
1>simgraph16.obj : error LNK2001: unresolved external symbol __imp_pthread_mutex_destroy
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_barrier_init referenced in function "public: void __cdecl loadsave_t::set_buffered(bool)" (?set_buffered@loadsave_t@@QEAAX_N@Z)
1>simview.obj : error LNK2001: unresolved external symbol __imp_pthread_barrier_init
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_barrier_destroy referenced in function "public: void __cdecl loadsave_t::set_buffered(bool)" (?set_buffered@loadsave_t@@QEAAX_N@Z)
1>loadsave.obj : error LNK2019: unresolved external symbol __imp_pthread_barrier_wait referenced in function "private: unsigned __int64 __cdecl loadsave_t::write(void const *,unsigned __int64)" (?write@loadsave_t@@AEAA_KPEBX_K@Z)
1>simview.obj : error LNK2001: unresolved external symbol __imp_pthread_barrier_wait
1>simworld.obj : error LNK2019: unresolved external symbol __imp_sem_init referenced in function "private: void __cdecl karte_t::world_xy_loop(void (__cdecl karte_t::*)(short,short,short,short),bool)" (?world_xy_loop@karte_t@@AEAAXP81@EAAXFFFF@Z_N@Z)
1>simworld.obj : error LNK2019: unresolved external symbol __imp_sem_destroy referenced in function "private: void __cdecl karte_t::world_xy_loop(void (__cdecl karte_t::*)(short,short,short,short),bool)" (?world_xy_loop@karte_t@@AEAAXP81@EAAXFFFF@Z_N@Z)
1>simworld.obj : error LNK2019: unresolved external symbol __imp_sem_wait referenced in function "private: static void * __cdecl karte_t::world_xy_loop_thread(void *)" (?world_xy_loop_thread@karte_t@@CAPEAXPEAX@Z)
1>simworld.obj : error LNK2019: unresolved external symbol __imp_sem_post referenced in function "private: static void * __cdecl karte_t::world_xy_loop_thread(void *)" (?world_xy_loop_thread@karte_t@@CAPEAXPEAX@Z)
1>C:\Users\James\Documents\Development\Simutrans\simutrans-experimental-sources\x64\Debug\Simutrans-Experimental-debug.exe : fatal error LNK1120: 22 unresolved externals
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

and this error for a single threaded build:

Code Select Expand


1>------ Build started: Project: Simutrans-Experimental, Configuration: Debug (single threaded) x64 ------
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzReadOpen referenced in function "public: bool __cdecl loadsave_t::rd_open(char const *)" (?rd_open@loadsave_t@@QEAA_NPEBD@Z)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzReadClose referenced in function "public: char const * __cdecl loadsave_t::close(void)" (?close@loadsave_t@@QEAAPEBDXZ)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzRead referenced in function "private: unsigned __int64 __cdecl loadsave_t::read(void *,unsigned __int64)" (?read@loadsave_t@@AEAA_KPEAX_K@Z)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzWriteOpen referenced in function "public: bool __cdecl loadsave_t::wr_open(char const *,enum loadsave_t::mode_t,char const *,char const *,char const *)" (?wr_open@loadsave_t@@QEAA_NPEBDW4mode_t@1@000@Z)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzWrite referenced in function "private: unsigned __int64 __cdecl loadsave_t::write(void const *,unsigned __int64)" (?write@loadsave_t@@AEAA_KPEBX_K@Z)
1>loadsave.obj : error LNK2019: unresolved external symbol BZ2_bzWriteClose referenced in function "public: char const * __cdecl loadsave_t::close(void)" (?close@loadsave_t@@QEAAPEBDXZ)
1>C:\Users\James\Documents\Development\Simutrans\simutrans-experimental-sources\x64\Debug (single threaded)\Simutrans-Experimental-debug.exe : fatal error LNK1120: 6 unresolved externals
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========


I am not sure quite what the problem is. Are there 64-bit versions of the bzip2 and pthreads libraries?

[prissi]

Bzip2 you can (must) anyway compile yourself. Aparently phtreads must be compiled too. But I was told the MSVC Express 64bit compiler does not work properly (problems with setjump), so I am surprised that you get any compilation at all. (The MSVC 64 bit works, but not express). Did you install the 64 bit version of the SDK?

[jamespetts]

Ahh, I wasn't aware of these issues with Express. I didn't install the SDK - just VS Express 2012 (is that the version that has trouble with 64-bit compilation?).

As to compiling pthreads and bzip2 - are their licences such that allow us to do that with a project under the Artistic Licence? The LGPL should probably suffice for our purposes, but not the full GPL if that is what they are licensed under.

[prissi]

The bzip2 you use is very likely anyway compiled by Dwachs. So you can do this too.

I think the openttd-usefull library file also comes with a 64bit version. It may be your best guess.

[jamespetts]

Thank you; that is helpful.

Hmm - I have already set it to use the Open TTD library's 64-bit version, which has not helped.

Would I have to compile bzip2 as a separate .lib file, or could it be compiled at the same time as and as part of the main executable?

Edit: I have now managed to compile the .lib file and get the single threaded version going - thank you for your help. Will a similar technique work for pthreads, do you think?

[prissi]

I think pthreads has a 64 bit version; if not, then you need to dig for one. Since it uses some assumption of the setjmp structures, which are no longer true in 64 bit mode some dedicated 64 bit code is needed. Hence the compatibility layer that allows for calling 32 bit dlls from 64 bit code will not work. Make sure, you have all 64 bit versions (and only those) in your path.

EDIT: Short googling revealed that pthreads should compile with x86_64 even on MSVC 2008.

[jamespetts]

Hmm - the trouble that I am having is that we seem to need pthreadsVCE2.lib when compiling with Visual Studio Express. pthreadsVCE2.lib comes in a 32-bit version but the 64-bit directory contains only pthreadsVC2.lib, which is not the same and does not work.

I tried to download the pthreads sources, and could compile a 64-bit version of pththreadsVC2.lib, but can still not work out how to set it to compile a 64-bit (or, indeed, any) version of pthreadsVCE2.lib.

[jamespetts]

I have now added a single threaded Windows 64-bit version to the latest .zip files with the binaries for 11.10. (The Linux platform has had a native 64-bit version for a while, of course).

[prissi]

VCE is with exceptions, which are not really used in simutranc The VC version should do fine.

[jamespetts]

Ahh, that does work, thank you! From what I have read, the VC version is probably better in any case. I had thought that Simutrans would only work with the VCE version.

[Junna]

I cannot seem to find a pre-compiled bzip2 that works for the x64 version, could that be shared here? I cannot figure out how to compile it from source, either...

[jamespetts]

Bzip2 is statically linked; do you need one for the purposes of compiling your own Simutrans-Experimental? If you are not compiling yourself, you do not need this.

[Junna]

Bzip2 is statically linked; do you need one for the purposes of compiling your own Simutrans-Experimental? If you are not compiling yourself, you do not need this.

Yes, that is what I need it for.

[jamespetts]

I have uploaded it here. I hope that that helps!

[Junna]

Thank you, compilation works now.