The International Simutrans Forum

 

Author Topic: Missing paks with latest compile  (Read 6221 times)

0 Members and 1 Guest are viewing this topic.

Offline wlindley us

  • Devotee
  • *
  • Posts: 958
    • Hacking for fun and profit since 1977
  • Languages: EN, DE
Missing paks with latest compile
« on: September 23, 2012, 01:09:40 PM »
I did a 'git pull' and then:

Code: [Select]
nice make -j3
And the resulting build directory contains vehicle.LMR-Patentee.pak but not vehicle.LMR-Patentee-Tender.pak ... which makes it very difficult to play a game in 1840.  The trains/ directory contains lmr-patentee-tender.dat but no pakfile gets created?  The same appears true with many of the ships and airplanes.

[/code]

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18375
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Missing paks with latest compile
« Reply #1 on: September 23, 2012, 01:20:22 PM »
Hmm - that's odd. I don't use the makefile myself, but the Python scripts, so it's difficult for me to track down the problem. Has anyone else had this?

Offline wlindley us

  • Devotee
  • *
  • Posts: 958
    • Hacking for fun and profit since 1977
  • Languages: EN, DE
Re: Missing paks with latest compile
« Reply #2 on: September 23, 2012, 01:22:24 PM »
Actually the pakfile becomes vehicle.LMR-Patentee.pak -- which I don't understand.  What prepends the "vehicle." and what makes some of the letters capitalized when the datfile name has no capitals?  And why does a single invocation of 'makeobj' in the makefile, result in dozens or hundreds of .pak files instead of just one combined pak?  Things are not working the way they are documented.

Alternately, ...what Python scripts? where?

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18375
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Missing paks with latest compile
« Reply #3 on: September 23, 2012, 01:41:06 PM »
Hmm, this is very odd, as the Python scripts (makeALL.mos and parameter.mos, in the Github repository) produce a single .pak file for each directory (one for trains, one for ways, etc.) rather than one per object,

Offline VS

  • Senior Plumber (Devotee)
  • Devotee
  • *
  • Posts: 4855
  • Vladimír Slávik
    • VS's Simutrans site
  • Languages: CS,EN
Re: Missing paks with latest compile
« Reply #4 on: September 23, 2012, 02:27:46 PM »
What prepends the "vehicle." and what makes some of the letters capitalized when the datfile name has no capitals?  And why does a single invocation of 'makeobj' in the makefile, result in dozens or hundreds of .pak files instead of just one combined pak?  Things are not working the way they are documented.
I take it you have never used makeobj on its own, then? When compiling "wholesale", compiled objects get file names in form of "type.name.pak". BTW, one dat file can hold many objects... so what determines the name is not file, but its contents. Finally, if you want to get one pak with many objects, you will have to either add some parameters when compiling, or merge afterwards.

Offline wlindley us

  • Devotee
  • *
  • Posts: 958
    • Hacking for fun and profit since 1977
  • Languages: EN, DE
Re: Missing paks with latest compile
« Reply #5 on: October 12, 2012, 06:03:45 PM »
Things have gotten worse.  The latest "git pull" and then "make" gives me, for example, JennyLind.pak (presumably from jennylind.dat) but not JennyLind-tender.pak ... there are 12 .dat files for Mk1 equipment:

mk1-bg.dat, mk1-bsk.dat, mk1-bso.dat, mk1-guv-b.dat, mk1-pos-rm.dat, mk1-rb.dat, mk1-rmb.dat, mk1-sk.dat, mk1-sub-brake.dat, mk1-sub.dat, mk1-tpo-rm.dat, mk1-tso.dat

but only 8 .pak files get created:

BR-Mk1-BG.pak, BR-Mk1-GUV.pak, BR-Mk1-POS.pak, BR-Mk1-RB.pak, BR-Mk1-RMB.pak, BR-Mk1-Sub-Brake-front.pak, BR-Mk1-Sub-Brake-rear.pak, BR-Mk1-Sub.pak

I do not understand how we get the filename with mixed case BR-Mk1-BG.pak from the source mk1-bg.dat -- simply calling 'makeobj' won't do that, at least according to the --help message, and there's nothing in the makefile that could do that... something very spooky is going on here, and it's not working at all.

As is, the pak is unplayable, in 1850 I get only mail carriages and no passenger equipment at all.  I get the Jenny Lind locomotive but no tender to match... and the Planet tender with no locomotive to match. 

How to debug this?

Offline VS

  • Senior Plumber (Devotee)
  • Devotee
  • *
  • Posts: 4855
  • Vladimír Slávik
    • VS's Simutrans site
  • Languages: CS,EN
Re: Missing paks with latest compile
« Reply #6 on: October 12, 2012, 06:55:48 PM »
How to debug this?
I'd check timestamp on makefile. Obviously James uses something else, so it might be out of date.
« Last Edit: October 12, 2012, 07:00:51 PM by VS »

Offline wlindley us

  • Devotee
  • *
  • Posts: 958
    • Hacking for fun and profit since 1977
  • Languages: EN, DE
Re: Missing paks with latest compile
« Reply #7 on: October 13, 2012, 02:38:29 AM »
The makefile is a bit older, but it is coded to compile "all" the paks in a directory... so why are existing dat-files that used to compile into paks, not doing so now?

Offline sdog

  • Devotee
  • *
  • Posts: 2039
Re: Missing paks with latest compile
« Reply #8 on: October 13, 2012, 06:34:33 AM »
please read on after the horizontal line, found out more while writing, making my first part almost obsolete

It's quite a while since i wrote the Makefile, and i can't remember everything.

The names ought to be created by a function in the Makefile

Code: [Select]
#generating filenames
#with this function the filenames are assembled, by removing the dir, adding prefix
#and suffix
make_name = $(NAMEPREFIX)$(notdir $1)$(NAMESUFFIX)

[...]

$(DIRS128):
        @echo "===> PAK128 $@"
        @mkdir -p $(PAKDIR)
        @$(MAKEOBJ) quiet PAK128 $(PAKDIR)/$(call make_name,$@) $@/ > /dev/null



I just pulled it, and it seems to work fine. The script above builds the pak filenames from the dir and adds your suffix and prefix from the config.default file to it. What i get is below:

(Check the version history on github, Nathanael fixed quite a lot in that Makefile, it's been quite broken when i made it.)

Code: [Select]
6586657 Oct 13 02:14 brit-attractions-ex.pak
 3372279 Oct 13 02:14 brit-boats192-ex.pak
  716487 Oct 13 02:14 brit-boats224-ex.pak
 1201981 Oct 13 02:14 brit-boats-ex.pak
 3569167 Oct 13 02:14 brit-bus-ex.pak
13464013 Oct 13 02:14 brit-citybuildings-ex.pak
  182726 Oct 13 02:14 brit-citycars-ex.pak
 3216217 Oct 13 02:14 brit-depots-ex.pak
    2313 Oct 13 02:14 brit-goods-ex.pak
  864596 Oct 13 02:14 brit-grounds-ex.pak
  275913 Oct 13 02:14 brit-gui128-ex.pak
  256295 Oct 13 02:14 brit-gui64-ex.pak
  114415 Oct 13 02:14 brit-hq-ex.pak
83431135 Oct 13 02:14 brit-industry-ex.pak
  152567 Oct 13 02:14 brit-london-underground-ex.pak
  715288 Oct 13 02:14 brit-maglev-ex.pak
   29951 Oct 13 02:14 brit-pedestrians-ex.pak
    8809 Oct 13 02:14 brit-smokes-ex.pak
 9010121 Oct 13 02:14 brit-stations-ex.pak
 2179062 Oct 13 02:14 brit-townhall-ex.pak
 7172589 Oct 13 02:14 brit-trains-ex.pak
  408714 Oct 13 02:14 brit-trams-ex.pak
  122165 Oct 13 02:14 brit-trees-ex.pak
19730958 Oct 13 02:14 brit-ways-ex.pak
    6904 Oct 13 01:57 compat.tab
    4096 Oct 13 01:57 config
  239494 Dec 11  2010 demo.sve
    8726 Oct 13 02:14 ground.Outside.pak
    5640 Dec 11  2010 licence.txt
    9676 Oct 13 01:57 readme.txt
    4096 Oct 13 01:57 text

Are you certain those strange file names with internal capitalisation are not remnants from an older build, eg made by James manually?

Second question, is this really experimental we are talking about? Back when we made the Makefile, it wasn't included for months into standard pak128.britain.


NEW


Forget everything i wrote, just checked on github website, not just my git pull and there's a different Makefile there.
Code: [Select]
@$(MAKEOBJ) quiet PAK256 $(PAKDIR)/ $@/ > /dev/nullthis line should take the name from the .dat file 'name' entry, not the filename, working through each file piece by piece.

Just saw that pak128.britain-experimental is now forked from nerodens standard repo, not James' standard branch in the experimental repo.

The outside.pak rules are really nice, those made quite a lot of problems.

I can't find out why i have the outdated Makefile in my repo, and it didn't get updated by the new Makefile. Also the old Makefile i have is not in the version history anymore. seems it got removed from git and the new one added instead of just overwriting it. Just looked at it and i have a horrible mess on my disk, those massive changes quite confused my git here.


How to fix it:
add a target dir entry to the makeobj call, it will take all the arguments passed afterwards as source files and put them in the former single file. (have a look at the old code snippet above)

Offline wlindley us

  • Devotee
  • *
  • Posts: 958
    • Hacking for fun and profit since 1977
  • Languages: EN, DE
Re: Missing paks with latest compile
« Reply #9 on: November 13, 2012, 02:07:09 PM »
Aha!  Here's the problem.  trains/lmr-4wheel-first.dat  is broken.


 It would have been much easier to find if makeobj's error output was actually sent to stderr instead of stdout, since stdout is > /dev/null by the standard makefile.

Code: [Select]
   reading file trains/lmr-4wheel-first.dat
   writing file simutrans/pak128.Britain-Ex/vehicle.LMR-4Wheel-First.pak
      packing vehicle.LMR-4Wheel-First
Reading freightimage[0][s]
Reading freightimage[0][s][0]
Reading emptyimage[s][0]
[.....]
Appending emptyimage[s][1]
Appending emptyimage[s][2]
*** FATAL ***:
Missing emptyimage[s][3]!

Offline wlindley us

  • Devotee
  • *
  • Posts: 958
    • Hacking for fun and profit since 1977
  • Languages: EN, DE
Re: Missing paks with latest compile
« Reply #10 on: November 13, 2012, 02:12:42 PM »
Even running makeobj manually the problem occurs.  Strangely, the dat file does not reference emptyimage[][3] freightimage[][3] but makeobj reports adding them?  And why would there be an emptyimage[][3] ... when only liveries 0..2 are defined?  The same errors occur when running makeobj manually on lmr-4wheel-open-coach.dat as well. What's going on?

And, shouldn't it be true that:
  • Errors in makeobj get reported, instead of dumped into /dev/null
  • A failed makeobj operation should abort the make process instead of blithely continuing and making you think it worked

Code: [Select]
   reading file trains/lmr-4wheel-first.dat
   writing file simutrans/pak128.Britain-Ex/vehicle.LMR-4Wheel-First.pak
      packing vehicle.LMR-4Wheel-First
Reading freightimage[0][s]
Reading freightimage[0][s][0]
Reading emptyimage[s][0]
Reading emptyimage[s][1]
Reading emptyimage[s][2]
Reading emptyimage[s][3]
Reading freightimage[s][0]
Reading freightimage[s][1]
Reading freightimage[s][2]
Reading freightimage[s][3]
Reading freightimage[s]
Appending emptyimage[s][0]
Appending emptyimage[s][1]
Appending emptyimage[s][2]
*** FATAL ***:
Missing emptyimage[s][3]!
« Last Edit: November 13, 2012, 02:21:27 PM by wlindley »

Offline wlindley us

  • Devotee
  • *
  • Posts: 958
    • Hacking for fun and profit since 1977
  • Languages: EN, DE
Re: Missing paks with latest compile
« Reply #11 on: November 13, 2012, 02:36:43 PM »
Follow-up:  Main problem is that, from  the simutrans-experimental directory, 'make' does not update makeobj.o or the makeobj executable.  Thus I was using a makeobj from 2011... you have to go into the makeobj subdirectory and manually run 'make' ... this pitfall ought to be corrected or at least documented.

Nevertheless, when running 'makeobj'  fails, it should not do so silently; it should send errors to stderr, and return a nonzero exit code to abort the main pak makefile.

End result: It works!  **happy dance**

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18375
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Missing paks with latest compile
« Reply #12 on: November 13, 2012, 11:35:00 PM »
Thank you for posting all of that - glad that you managed to get it working! Any suggestions on how to update the code so that Makeobj properly reports its errors? Perhaps a "verbose" command line swtich to make it optional...?

Offline sdog

  • Devotee
  • *
  • Posts: 2039
Re: Missing paks with latest compile
« Reply #13 on: November 13, 2012, 11:51:06 PM »
having error messages at all shouldn't be affected by the verbose setting, or the stderr pipeline. Having more warnings or verbose error messages on stdout would be something to be switched by a verbose setting.

The way was shown by wlindley already:
Quote
Nevertheless, when running 'makeobj'  fails, it should not do so silently; it should send errors to stderr, and return a nonzero exit code to abort the main pak makefile.

The Makefile pipes all stdout output to /dev/null it wouldn't appear anywhere.
@$(MAKEOBJ) quiet PAK128 $(PAKDIR)/ $@/ > /dev/null
It would show stderr on the terminal though. When it returns a non-zero exit code, make will also automatically exit at that point.

Offline wlindley us

  • Devotee
  • *
  • Posts: 958
    • Hacking for fun and profit since 1977
  • Languages: EN, DE
Re: Missing paks with latest compile
« Reply #14 on: November 13, 2012, 11:57:07 PM »
In besch/writer/*.cc -- the 18 occurrences printf("*** FATAL ***...."); exit(0) --  should instead be directed to stderr, and use exit(1); or some other error code.

Offline TurfIt

  • Dev Team, Coder/patcher
  • Devotee
  • *
  • Posts: 1321
Re: Missing paks with latest compile
« Reply #15 on: November 14, 2012, 12:49:43 AM »
Agreed. In Standard at r6050 (only 16 occurrences found)

Offline TygerFish

  • *
  • Posts: 107
Re: Missing paks with latest compile
« Reply #16 on: December 05, 2012, 04:09:56 AM »
Thank you all for sharing this info... I was getting only a single train while building from the latest pak source.

James -- when trying to use those python scripts, I get the following error:


Code: [Select]
  File "makeALL.mos", line 2
    !mose min_version 3



Is there a different command I should be using than "python makeALL.mos"?  What does the .mos extension signify, anyway?  I've never seen that before.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18375
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Missing paks with latest compile
« Reply #17 on: December 05, 2012, 11:52:18 PM »
Hmm, that's odd: I don't have difficulties with this. The problem is that I didn't create these scripts. All that I know is that you need to install Python to run them. Perhaps you don't have the latest version of Python...?

Offline TygerFish

  • *
  • Posts: 107
Re: Missing paks with latest compile
« Reply #18 on: December 06, 2012, 07:35:37 PM »
As far as I know -- v3.3 from here: http://www.python.org/download/releases/3.3.0/  Running on Windows 7 Ultimate 64-bit, installed x86_64 MSI, which put Python in my PATH.

I tried both running from a parent directory and from the same directory as the buildall.mos script, and both produced the same error. 

Hmm, that's odd: I don't have difficulties with this. The problem is that I didn't create these scripts. All that I know is that you need to install Python to run them. Perhaps you don't have the latest version of Python...?

Offline TygerFish

  • *
  • Posts: 107
Re: Missing paks with latest compile
« Reply #19 on: December 07, 2012, 03:46:43 AM »
I figured out why that python script wouldn't work AND why it has a .mos extension!

First, you need to install MOSE: http://forum.simutrans.com/index.php?topic=2403.0  If you're using Windows, it has a nice installer that associates the .mos extension with itself for handy double-clicking.

Second, you need to update the parameter.mos file to point to your MakeObj.exe (James -- we know what your username is on your machine.  :) )

After that, it worked like a charm!  Unfortunately, just like the .bat version, it's only outputting a tiny number of the trains off of current source...

EDIT: I've tried opening the makeobj projects I downloaded from jamespetss/simutrans-experimental to recompile, but Visual Studio just opens them as XML files.  Even tried in VS2008 (they say Version="9.00"), but same result.
EDIT: Managed to get one of the projects to convert to VS2010, but building fails because of two missing files: '..\tpl\debug_helper.cc' and '..\utils\cstring_t.cc'
« Last Edit: December 07, 2012, 04:04:13 AM by TygerFish »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18375
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Missing paks with latest compile
« Reply #20 on: December 07, 2012, 09:42:23 AM »
Glad that you've managed to get it working!

Can't you just download the binary Makeobj? I am not sure why it is not putting out a proper number of .pak files - what I have on my computer, and that which is on the Github repository, works fine. What text output do you get? What does your parameters.mos file say?

Offline TygerFish

  • *
  • Posts: 107
Re: Missing paks with latest compile
« Reply #21 on: December 07, 2012, 11:42:37 PM »
When I run makeALL.mos, The console window outputs a series of "Building aircraft/attractions/etc." notifications, each followed by one or more lines of "Makeobj version 52 for simutrans 111.0 Experimental 10.4 and higher".  The "Building trains" section has no unusual output, and the result does have a BritRail-Ex.pak file that's almost 9MB.  But when loading the pak in game, only a very few trains appear.

My parameter.mos file has two non-comment lines: !makeobj= followed by an absolute path to my makeobj.exe file, and !OUTPUT=./Pak128.Britain-Ex-0.9.0/

Glad that you've managed to get it working!

Can't you just download the binary Makeobj? I am not sure why it is not putting out a proper number of .pak files - what I have on my computer, and that which is on the Github repository, works fine. What text output do you get? What does your parameters.mos file say?
« Last Edit: December 08, 2012, 01:29:02 AM by TygerFish »

Offline TygerFish

  • *
  • Posts: 107
Re: Missing paks with latest compile
« Reply #22 on: December 08, 2012, 06:59:44 PM »
Turns out I did have an outdated version of makeobj!  I re-downloaded from https://github.com/downloads/jamespetts/simutrans-experimental/Makeobj-experimental.zip which outputs "Makeobj version 53 for simutrans 111.2.1 Experimental 10.12 and higher" as it's run.  Re-built with that and it looks like all the trains are back!  :)

When I run makeALL.mos, The console window outputs a series of "Building aircraft/attractions/etc." notifications, each followed by one or more lines of "Makeobj version 52 for simutrans 111.0 Experimental 10.4 and higher".  The "Building trains" section has no unusual output, and the result does have a BritRail-Ex.pak file that's almost 9MB.  But when loading the pak in game, only a very few trains appear.

Offline greenling

  • Lounger
  • *
  • Posts: 1728
  • Simutransarchology it my hobby!
  • Languages: DE,EN
Re: Missing paks with latest compile
« Reply #23 on: December 08, 2012, 07:31:51 PM »
Tygerfish
thank you for this info.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18375
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Missing paks with latest compile
« Reply #24 on: December 08, 2012, 10:41:30 PM »
Excellent! Glad that you have sorted out the problem.