The International Simutrans Forum

 

Author Topic: Big "fix it" patch for simutranslator, and please stop making DOS line endings..  (Read 13000 times)

0 Members and 1 Guest are viewing this topic.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Gaaaaah....

OK.  This is a patch.  It's zipped because it's large.  First unzip it. Then apply by going into the top directory for pak128.britain and running "patch -p1 air-simutranslator.diff".

This does several things:
(1) It adds a "simutranslator" target to the Makefile, which builds zip files suitable to upload to simutranslator.
(2) It renames 3 of the air/ files for consistency (these were driving me nuts).
(3) It adds correct terminators (---) at the bottom of files for the benefit of simutranslator, which needs them.
(4) It converts the Makefile and everything in air/ back to Windows/UNIX/Linux line endings.  They got "DOSified" by accident at some point.

You need to figure out how to stop committing files with DOS line endings.  What text editor are you using?  Rule 1: do not use Notepad.  If it's anything else (Wordpad etc.), I can probably explain how to get it working in Windows/UNIX/Linux mode.

It's also possible that TortoiseSVN is causing the problems.  If so, it's more tedious to fix, but there is a way and I can explain it.

(EDIT: if you ever switch entirely to git, I know how to make things work there too; it's even easier.)
« Last Edit: May 13, 2012, 08:51:00 PM by neroden »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18555
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
For reference, I use MSVC++ Express both for the code and for the pakset work.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
OK, this is a followup patch.

I believe it is not wise to have spaces in object names.  It causes makeobj to generate pak files with spaces in their names, which in turn causes trouble on UNIX systems sometimes.  Although it seems to work right now, it's not a good idea.

So this patch removes the spaces from the internal name of the Ford Trimotor.  It must be applied AFTER the large patch (due to the annoying business with DOS line endings).

EDIT: James Petts, I don't think you've introduced any DOS line endings lately, though I'll check; I believe James Hood has, so that was addressed to him.

EDIT 2: I have now updated Simutranslator with all the new aircraft and new English translations for all of them.  If I can get these patches into pak128.britain (standard), I will then move on to the remaining directories, one at a time, and when I'm done I'll extract a new full set of translations from Simutranslator to put into svn.
« Last Edit: May 13, 2012, 10:39:04 PM by neroden »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18555
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Ahh, thank you for the Ford Trimotor patch. I had already fixed this on Experimental, using the name: Ford-Trimotor-NSA. Perhaps this would be an easier and shorter version to use on Standard, too?

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Ahh, thank you for the Ford Trimotor patch. I had already fixed this on Experimental, using the name: Ford-Trimotor-NSA. Perhaps this would be an easier and shorter version to use on Standard, too?

Hmm.  That's up to James Hood.  I supposed that most people wouldn't know what NSA meant and figured that the full version of the name should be included because of that. 

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18555
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Ahh, but people woulndn't see "NSA" in game, as it'd be translated. In en.tab, there is the line:

Code: [Select]
Ford-Trimotor-NSA
Ford Trimotor - Northern & Scottish Airways

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Ah... but most of the English translations for airplane names don't give the airline name.  :-)  They actively remove it, in fact.

I figured this was a deliberate stylistic choice which I was trying to maintain.  If you want the airline name in the Trimotor, we should insert it in the translations for all the others too. 

I figured the airline name was just a reminder to us, the programmers, regarding what livery was used in the artwork.

EDIT: NEW PATCH.  attractions/stone-attractions.dat had one annoying typo, which I want to fix before it's too late (Cemetry->Cemetery).  Of course, this file's in DOS format so the patch changes EVERY SINGLE LINE again, and I have to zip the patch.   If the files in standard were all "de-DOSed" I could submit shorter patches.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18555
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Ahh, yes, you're right about the airline names, actually - I'd forgotten about that. I shall change it now.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
OK, question: why is there a "Fountain" in the stone-attractions, and another "Small Fountain" as a separate file, with very similar graphics and completely different .dat contents?  Though I do think multiple fountains should exist, I don't think they should have the same artwork and wildly different stats.

EDIT: OK, confusion: it's a "Small Fountain" in the stone-attractions as a "cur", and a "Fountain" in its own file as a "mon".  So I guess that's the difference...
« Last Edit: May 14, 2012, 01:22:22 AM by neroden »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18555
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Hmm, if they have the exact same graphics, this is probably an error. Was this duplication in Standard, Experimental, or both?

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Hmm, if they have the exact same graphics, this is probably an error. Was this duplication in Standard, Experimental, or both?

See my edit above; graphics aren't exactly the same (not bit-for-bit identical), just practically the same.  One's a "mon" and the other's a "cur", which I suppose is the reason here... I still don't get it.  This is in Standard.

EDIT: while I'm poking around, anyone have some naming ideas to differentiate the 4 Cathedrals, 2 Abbeys, and 2 Castles in the translations?  For now I'm sticking with the boring "Cathedral" for all the catehdrals, but....

EDIT 2: Just finished going through attractions for standard.  That was tedious.  James Petts, if you are interested in these fixes, they're on the standard-fixes branch of my github (though you'll have to cherry-pick them, to avoid a full merge from standard, which might break stuff).

Now I await TheHood's integration of my patches.  And I go to sleep.
« Last Edit: May 14, 2012, 01:57:47 AM by neroden »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18555
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
I tend to prefer to use the same name: there's no reason in principle to distinguish them - the different objects are there for graphical variety, not because they are fundamentally different types of buildings.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
I tend to prefer to use the same name: there's no reason in principle to distinguish them - the different objects are there for graphical variety, not because they are fundamentally different types of buildings.

I got four cathedrals in one city, which was "interesting", and I proceeded to start renaming them (to things like "Rose Cathedral" and "St Savior Cathedral").  But sure, we can leave them as "Cathedral".

Offline Milko

  • Devotee
  • *
  • Posts: 828
Hello neroden

You need to figure out how to stop committing files with DOS line endings.  What text editor are you using?  Rule 1: do not use Notepad.  If it's anything else (Wordpad etc.), I can probably explain how to get it working in Windows/UNIX/Linux mode.

I'm designing planes for pakBritain. I put the files "dat" using wordpad of windows. I am attaching files to messages in the forum instead. I have to use precautions to place the files (de-DOSed)? Or precautions must be used by him in github which loads the files?

Giuseppe

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Hello neroden

I'm designing planes for pakBritain. I put the files "dat" using wordpad of windows. I am attaching files to messages in the forum instead. I have to use precautions to place the files (de-DOSed)? Or precautions must be used by him in github which loads the files?

Giuseppe

It's really James Hood's responsibility to get this right when he checks the files into the repository, not your responsibility.  So I need to know what tools he's using.

Wordpad should be able to save files with UNIX line endings anyway.  I'm having trouble finding the documentation on this, and I don't use Windows any more so I can't check it myself.  So it might not.

If you get something like the free Notepad++ (or practically any other text editor apart from the ones which come with Windows), you can "Save As... UNIX Format" and guarantee that you're getting UNIX line endings.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
For reference, I use MSVC++ Express both for the code and for the pakset work.

Which version?

I think we solved your situation with git options.  Git has an option to basically force all check-ins to have UNIX line endings.
With SVN, you have to set the option on *every single text file individually*.  Which we should probably do, but that's tedious, isn't it?

Offline kierongreen

  • Dev Team, Coder/patcher
  • Moderator
  • *
  • Posts: 2269
Quote
With SVN, you have to set the option on *every single text file individually*.  Which we should probably do, but that's tedious, isn't it?
If that's only once I don't see that as being overly tedious, but yes if it's causing problems would be an idea to change.

Offline VS

  • Senior Plumber (Devotee)
  • Devotee
  • *
  • Posts: 4855
  • Vladimír Slávik
    • VS's Simutrans site
  • Languages: CS,EN
Well, with tortoise you can select a number of files and set the property on them all, so no problem there ;)

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
neroden,

Thanks for the patches - I appreciate the need for consistency and your efforts at attempting to resolve it. Unfortunately right now I don't have time to do anything about this (nor do I know when that might be). I'm happy for other people to make these changes in SVN, but that may cause problems down the line when I modify stuff and "undo" the changes. I know that's a frustrating answer - sorry.

For now I can explain what I do - I try to be systematic but bear in mind that nearly everything I've done in pak128.Britain has been self-taught and driven by necessity/what I or others wanted at the time so it's a bit of a mish-mash of processes cobbled together to suit immediate needs. I do this for fun and not as a professional coder and although I appreciate why your patch is important please understand if it's not top of my to-do list (especially if it involves changing 1000 settings one-by-one!).

1) draw stuff in blender/edit in gimp
2) write dat file in WordPad (although some of the older ones will be left over from a previous computer which ran linux and I used kwrite)
3) compile with makeobj and test
4) copy files over to SVN directory
5) TortoiseSVN update/commit to sourceforge SVN*
6) wait for nightly build to do it's business/occasionally make manual release on sourceforge with new version number

*I use sourceforge because that's what the other paksets use and that's what prissi invited me to use. I have no other reasons either way. Blend files exist on github because there is more space for large files on there (so I understand). I've forgotten how to upload stuff there, but have no objection to making that the main repository other than (the large one of) it not being where other simutrans paksets are based.

I'm open to suggestions about what the best way of doing things differently is, but as I say, please bear in mind I get little time for Simutrans and when I do I like to draw new objects rather than tidy up dat files.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18555
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
I recommend using Git instead of SVN. When using Git, one can merge in others' work with very little effort (actually, one can do that to some extent with .patch files for SVN, but it's not quite the same, and can't deal with non-text files, deleting files, or changing the structure, as far as I am aware).

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
I recommend using Git instead of SVN. When using Git, one can merge in others' work with very little effort (actually, one can do that to some extent with .patch files for SVN, but it's not quite the same, and can't deal with non-text files, deleting files, or changing the structure, as far as I am aware).

How would that work with Sourceforge and the other paksets/nightly builds? I'd rather not complicate that too much as that is what people play/download...

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18555
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Hmm - I'm not sure about Sourceforge, since I don't really use it. Nightly builds can be set up to work with Git, as far as I understand, but I suppose that that would require the co-operation of those who administer them.

Offline sdog

  • Devotee
  • *
  • Posts: 2039
Should it be difficult or inconvenient for Ansgar to build nightlies from git, i could volunteer to do so. (only two caveat and one precondition. The Makefile needs to be maintained, i won't start playing around with those python scripts. I can't host it on the machine i build, this would most likely not work well with university regulations. give me any place to scp or sftp it to, or if cloudspace is acceptable this works. I am not experienced with doing such things, i have an idea what i'm doing though.)


If you want i can start building nightlies from the github mirror within a week or so, that is as a proof of concept.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
OK, given your workflow I would recommend switching to github for your primary repository -- if you can tolerate using the command line.  It's a little bit of a pain to set up under Windows, but it only needs to be set up ONCE and it should make merging changes from other people a lot easier.  You've already got an account on github and you already have a repository there. 

If you haven't got git installed on your home computer, here's a guide for the preferred configuration with github:
http://help.github.com/win-set-up-git/

This configuration automatically fixes line endings on check-in.  The business with SSH keys is basically about secure communication with github; you must have done a bunch of this in order to upload the blend files, already.

Then, open up "Git Bash" whenever you need to type something at the command line in git.

Git workflow is a little bit different from Subversion workflow.  James Petts is using it, but it might take some practice to get used to it.

Basically, with git, you have your own copy of the repository locally. (There's a little setup involved in doing this the first time, which I can explain if you decide to do it; I don't remember it off the top of my head because I only had to do it once.)  So in order to check in data, you have to first tell git to check it in locally with
Code: [Select]
git commit <names of new or modified files> and then push it to github with
Code: [Select]
git push origin-- Not quite as convenient as TortoiseSVN, I admit.

But the nice part about it is how easy it is to merge other people's changes if they're also using git.
For instance, you set up a reference to my repository with
Code: [Select]
git remote add neroden git://github.com/neroden/simutrans-pak128.britain.git
And then in future you can merge an entire pack of changes from me really easily.  If I ask you to merge my "standard-fixes" changes, you type
Code: [Select]
git pull neroden standard-fixesto add the changes locally, and then
Code: [Select]
git push origin to send it back to github.

If I somehow mess things up and the "pull" causes merge conflicts, run
Code: [Select]
git reset --hard and tell me to fix my patch ;)

Now, one alternative is to give me SVN write access, in which case I just go in and fix this stuff myself.  But with git, you don't have to do that.  You can pull one particular set of changes from me (or anyone else using git) really easily, without having to give me generic privileges to write to the repository.

sdog: I'm already maintaining the Makefile.  Also, is anyone building nightlies for experimental?  Experimental is already based in git.  It would definitely be wise to be able to build nightlies from git for that reason alone.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 9454
  • Languages: De,EN,JP
(Sidenote: Actively pull means he has to keep track also of your stuff. Makes even more work for a maintainer ... )

But on topic: For sourceforge SVN access, you need a sourceforge account. If you are registered there, I can give it to your too.
« Last Edit: May 15, 2012, 09:58:34 PM by prissi »

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Geez, I don't remember whether I have a sourceforge account... uh-oh.  My username is taken, which means that I probably did register an account in the past.  But I don't know what email address it was registered with and I probably don't have that email address any more.  Gaaah... well, I'll try the non-automated recovery process.  Maybe I'll have a sourceforge account in a day or two.

Well, it's up to TheHood.  If you (prissi) are willing to give me SVN access, and TheHood is willing to authorize me to commit to pak128.britain without review, that will work.

I've gotten kind of spoiled by how cheap branching is with git, of course.  Subversion is perfectly nice, but branching is expensive, and file renames don't merge easily.  And in this case, I will have to go through and mark every single .dat file with the line ending property.  Although I can automate that, I have to rerun the script every time a new file is added.

EDIT: Anyway, I've made it through the 'boats' translations.  That's three directories...

EDIT 2: Aha.  I figured out how TheHood can avoid committing new files with DOS line endings in SVN.  (I still have to set properties on every single one of the old ones individually.)  Anyway, TheHood, look at this page:

http://www.mediawiki.org/wiki/Subversion/auto-props

In the auto-props section, in addition to the adding the ones listed there, you will want a line:
Code: [Select]
*.dat = svn:eol-style=...I think we don't want "native" we want "all lf all the time" but I have to run, I'll figure out the exact code later...
« Last Edit: May 15, 2012, 10:23:53 PM by neroden »

Offline VS

  • Senior Plumber (Devotee)
  • Devotee
  • *
  • Posts: 4855
  • Vladimír Slávik
    • VS's Simutrans site
  • Languages: CS,EN
I have write access, too, and can mark the files for you, if you make a list.

I am not sure what files you want to have as all-LF, but pak128 uses native everywhere, except translations, and had no problems, ever. And translations need LFs only because translator exports them as LF, so this way you avoid conversion, just export and overwrite the files.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 9454
  • Languages: De,EN,JP
I am not sure what they are currently with pak64; but simutrans (and translator) can handle both line endings.

In principle there are hooks for this; our sourcecode repo ensures all this (and a lot of other stuff with spaces) but in sourceforge there is no easy way with hooks.

But with a script it is dead easy, just do "svn propset svn:eol-styl native *.dat" in every directly and you are set.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Native everywhere is fine (that means they're LF in the repo ).   The problem is having a mix of files, some LF, some CR-LF...

So this means TheHood should follow the instructions above regarding "auto props" and add
Code: [Select]
*.dat = svn:eol-style=native
Anyway.  I have finally recovered my sourceforge account (THAT took a long time).  The username is "neroden".  I can go through and start doing the fixes in svn if I get write access.

So, what has to be done to get write access?  This is already slower than using git ;-)

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
I have write access, too, and can mark the files for you, if you make a list.

*.dat.  Every dat file should be 'native' (which means LF in the repo and on Linux).

Offline Dwachs

  • DevTeam, Coder/patcher
  • Administrator
  • *
  • Posts: 4564
  • Languages: EN, DE, AT
So, what has to be done to get write access?
Send an email to one of the administrators? You should have svn access as of now ;)

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Native everywhere is fine (that means they're LF in the repo ).   The problem is having a mix of files, some LF, some CR-LF...

So this means TheHood should follow the instructions above regarding "auto props" and add
Code: [Select]
*.dat = svn:eol-style=native
Anyway.  I have finally recovered my sourceforge account (THAT took a long time).  The username is "neroden".  I can go through and start doing the fixes in svn if I get write access.

So, what has to be done to get write access?  This is already slower than using git ;)

OK I've changed this. Can I just confirm this should now mean all future commits are in the right format, i.e. no DOS line-endings?
I'm now going to have a go at applying the various patches.

EDIT: for some reason absolutely nothing happens when I try to apply the patch using either TortoiseMerge or GnuWin patch on the command line. I'd be grateful if someone who knows what they are doing could apply these patches.
« Last Edit: June 07, 2012, 12:48:06 PM by The Hood »

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
I'll apply them.  I've been seriously distracted by changing computers among other things, but I can apply my own patches.   ;)

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Excellent. Can you also check when you are doing it that the latest things I have uploaded (e.g. boeing 747) are in the correct line ending format?
Perhaps, to avoid repeating the need for such patches in the future, could you produce a checklist of things we need ensure in a dat file?

Offline Milko

  • Devotee
  • *
  • Posts: 828
Hello Neroden

If you haven't got git installed on your home computer, here's a guide for the preferred configuration with github:
http://help.github.com/win-set-up-git/

This configuration automatically fixes line endings on check-in.  The business with SSH keys is basically about secure communication with github; you must have done a bunch of this in order to upload the blend files, already.

I set up git locally on my PC. I use git gui.

I'm modifing some pak128brit stuff and I'm modifing existing files using microsoft wordpad.

If I use git gui for uploading the changes I will insert more errors at line endings?

Thank's

Giuseppe