The International Simutrans Forum

 

Author Topic: Request for TheHood: please set svn:eol-style=native for text files  (Read 18590 times)

0 Members and 1 Guest are viewing this topic.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
I'm dealing with spurious CRLF / LF conflicts.

The solution is well known, but requires someone with commit powers (i.e., you) to make a change in the SVN repository.  Do you think you could set the Subversion property "svn:eol-style" to "native" for the text files in the pak128.Britain repository?

http://svnbook.red-bean.com/en/1.1/ch07s02.html

I'm not sure which client you're using, but you might also be able to set "auto props" so that all *.dat files will get svn:eol-style=native.

http://svnbook.red-bean.com/en/1.1/ch07.html#svn-ch-7-sect-1.3.2

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #1 on: April 24, 2010, 05:57:13 PM »
OK, can you repeat in English please?  ;)

And note that the SVN is shared with pak64, will this cause any problems for there or will whatever you suggest be confined to the pak128.Britain folder?

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #2 on: April 25, 2010, 09:37:22 PM »
OK, can you repeat in English please?  ;)
Uh, I'm not sure these concepts translate into English.  :-)  Let me try.

People on DOS and most people on Windows are using text files with a format which ends each line with the "carriage return" character ("\r" in C, ASCII decimal 13)  followed by the "line feed" character ("\n" in C, ASCII decimal 10).

People on Linux, UNIX, Mac, some people on Windows, and most of the rest of the world are using text files with a format which ends each line with the "line feed" character only.

This causes spurious differences every time a file gets edited on a different platform, because the editors automatically change the line endings to the "native" format for that platform.  This means that checking for changes becomes a mess because there are all kinds of meaningless changes.

Subversion has a fix for this.  Each file in a Subversion repository has "properties".  One of those properties is "svn:eol-style".  If it is set to "native", Subversion will automatically translate the file to use the correct style of line endings for the person who is checking out the repository.  Thus eliminating the spurious differences.

If you're using the original Subversion client, this can be set with the command line:
svn propset svn:eol-style "native" pak128.Britain/bus/wagons.dat
(And repeat, one by one, for each .dat file and every other text file.)

If you're using TortoiseSVN, the documentation for setting properties appears to be here:
http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-dug-propertypage.html#tsvn-dug-propertypage-props

Note that it should be set for *each individual text file*, not on folders/directories.  And only for the *text files*.  It would do bad things if the svn:eol-style property were enabled on, say, a .PNG file.

Quote
And note that the SVN is shared with pak64, will this cause any problems for there or will whatever you suggest be confined to the pak128.Britain folder?

Confined to the pak128.Britain folder.  At least, I only care about the pak128.Britain folder.

It should actually be completely harmless, and an improvement, for every text file, but I'd ask Prissi before committing changes to the pak64 folder!

Both simutrans and makeobj can handle both types of line endings (tabfile_t::read_line in dataobj/tabfile.cc handles the difference).  It only makes a difference because the version control tools -- both svn and git -- don't know that the line ending difference is irrelevant unless we *tell them*.  (After all, the difference would matter for some file which wasn't a text file.)

Um, does that help?  Hopefully?

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #3 on: April 25, 2010, 09:40:58 PM »
Yes that helps.  Is there a quick way to change all the dat files as you suggest from the command "svn" in linux?  Can't say I'm desperately keen to go through each file (and I don't really have the time to do that right now...)

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #4 on: April 25, 2010, 09:44:40 PM »
Yes that helps.  Is there a quick way to change all the dat files as you suggest from the command "svn" in linux?  Can't say I'm desperately keen to go through each file (and I don't really have the time to do that right now...)

Uh, yes there is, using the magic of the shell.  Do you have access to a Linux machine with an SVN checkout?  I'll give you a magic incantation later this evening, as I have to run to dinner now.  :)

EDIT:

The first line simply puts you in the pak128.Britain directory, so adjust that to whereever it is.
The second line does the work for the .dat files.  Be very careful to use the correct quotation marks, all three are different.  :)
The third line does the same work for the .tab files.
The fourth line does the same work for the .txt files.
Perhaps unsurprisingly, you have to run 'svn commit' afterwards.
Code: [Select]
cd pak128.Britain
for x in `find . -name '*.dat'`; do svn propset svn:eol-style "native" $x; done
for x in `find . -name '*.tab'`; do svn propset svn:eol-style "native" $x; done
for x in `find . -name '*.txt'`; do svn propset svn:eol-style "native" $x; done
svn commit

Any new text files added later still have to be done individually, since I haven't looked up the correct magic for "auto-props".
« Last Edit: April 25, 2010, 11:45:13 PM by neroden »

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #5 on: April 26, 2010, 07:29:19 AM »
Yep, can do that.  I run linux at home and do pretty much all simutrans stuff from the command line.

Offline VS

  • Senior Plumber (Devotee)
  • Devotee
  • *
  • Posts: 4855
  • Vladimír Slávik
    • VS's Simutrans site
  • Languages: CS,EN
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #6 on: April 26, 2010, 07:40:50 AM »
By the way - if you have translation files there as well, set their eol-style to LF. Saves some trouble, too ;)

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #7 on: April 26, 2010, 04:52:49 PM »
By the way - if you have translation files there as well, set their eol-style to LF. Saves some trouble, too ;)

That would mean they would be checked out with LF endings even on Windows machines, correct?  What's the reason that's useful? :)

In any case, the translation files are all in the 'text' subdirectory, and nothing else is in the 'text' subdirectory.  So after issuing the previously mentioned incantations, the way to do that is:
Code: [Select]
cd text
for x in `find . -name '*.txt'`; do svn propset svn:eol-style "LF" $x; done
for x in `find . -name '*.tab'`; do svn propset svn:eol-style "LF" $x; done
svn commit

Since I'm 100% Linux ;) I don't really care whether the style is set to "native" or "LF"; they come out exactly the same for me.  Windows users will have to decide which is more desirable.

Offline VS

  • Senior Plumber (Devotee)
  • Devotee
  • *
  • Posts: 4855
  • Vladimír Slávik
    • VS's Simutrans site
  • Languages: CS,EN
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #8 on: April 26, 2010, 05:56:42 PM »
Translations always need LF endings (or used to for a very long time). That is the reason why around here for editing these files was never suggested notepad, but wordpad...

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #9 on: April 29, 2010, 02:11:00 AM »
FYI, I'm thinking I can get a reasonable git mirror of the svn set up if this is done.

I suppose I should ask how the simutrans git mirror is updated automatically, since at the moment I only know how to manually update a git mirror of an svn repo.  (Maybe someone just has a cron job running, but unfortunately I can't really do that.)  Anyone know who aburch is and how he or she does this?  :)

Offline ansgar

  • *
  • Posts: 80
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #10 on: April 29, 2010, 11:09:29 AM »
I suppose I should ask how the simutrans git mirror is updated automatically, since at the moment I only know how to manually update a git mirror of an svn repo.  (Maybe someone just has a cron job running, but unfortunately I can't really do that.)  Anyone know who aburch is and how he or she does this?  :)

That is a cron job.  If there is interest, I can setup another one for pak128.Britain as well.

Ansgar

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #11 on: April 29, 2010, 11:21:49 AM »
That is a cron job.  If there is interest, I can setup another one for pak128.Britain as well.

Ansgar

That would be helpful indeed! Would you be able to include the Experimental version, too...?

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #12 on: April 30, 2010, 12:33:54 AM »
That would be helpful indeed! Would you be able to include the Experimental version, too...?

Well, what I figured was, if
(a) Ansgar sets up a cron job mirroring pak128.Britain on git, the way simutrans is mirrored
(b) the line-ending problem is fixed in svn
(c) We all remember to set the correct line-ending option in git

Then we will have no trouble merging from pak128.Britain to pak128.Britain-experimental, which is already in git.  (Though to be honest we will probably have to rebranch pak128.Britain-experimental so that git can "recognize" that they have common ancestry, and leave the old git pak128.Britain history as a "dead end"; I know how to do this though, and can do it as soon as the git mirror of "standard" is ready.)

Ansgar, it would be much appreciated if you set up a git-svn mirror of pak128.Britain like the one for simutrans.  Then as soon as TheHood (or someone else with svn commit access) changes the line endings, we can get the pak128.Britain-experimental git repositories working "properly".

Offline ansgar

  • *
  • Posts: 80
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #13 on: April 30, 2010, 01:44:07 AM »
Ansgar, it would be much appreciated if you set up a git-svn mirror of pak128.Britain like the one for simutrans.

http://github.com/aburch/simutrans-pak128.britain/tree/
Please tell me if there are any problems with this.  (Mail preferred, might miss things otherwise.)

Ansgar

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #14 on: May 01, 2010, 04:51:46 AM »
http://github.com/aburch/simutrans-pak128.britain/tree/
Please tell me if there are any problems with this.  (Mail preferred, might miss things otherwise.)

Ansgar

It looks excellent, thank you.  I can't completely test it until the line endings are fixed in SVN (nag nag :) )-- then I'll tell you if there's trouble getting the line ending fix to propagate to git.

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #15 on: May 03, 2010, 04:18:01 PM »
OK, OK, OK...,

Just a quick question - presumably any new dats that I write will default to current settings?  Not sure why then that is causing you problems on Linux as I am writing them all on Linux anyway...  Does this mean I need to change each dat before every SVN commit?

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #16 on: May 03, 2010, 04:34:05 PM »
What text editor do you use for the .dats?  I use MSVC++ to edit the .dat files, as I find the IDE environment very helpful. Just using a different text editor might get your line endings correct automatically.

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #17 on: May 03, 2010, 04:52:04 PM »
I'm using Kwrite which is bundled with KDE, which I use.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #18 on: May 06, 2010, 05:07:48 PM »
OK, OK, OK...,

Just a quick question - presumably any new dats that I write will default to current settings?
EDIT: yes, this is correct.  You have to set the option on each *new* file.  It's a sticky property in the repository, you only have to set it once per file.

Quote
 Not sure why then that is causing you problems on Linux as I am writing them all on Linux anyway...
EDIT: Actually, probably any new dats made by *you* won't cause problems, but if someone else checks them out under Windows and then checks them back in, then it will cause problems, so it is desirable to set the "native" option on all .dat files.

If you can guarantee that no Windows user will ever have commit access and that you will never commit a file downloaded from a Windows user :-) then you might be OK with just converting all the files once.  But it's still better to set the eol-style option.

Quote
Does this mean I need to change each dat before every SVN commit?
I bet you're using a "smart" editor which saves files with the same line endings it "found" in them.  :-P  I've never checked whether Kwrite does that.  I tend to use nano.  EDIT: Yep, Kwrite "cleverly" maintains whatever line endings the file had before you edited it.

In fact, the "native" line style setting should simply eliminate all problems: no matter what your editor does, the line endings should be normalized upon commit.  So no, you won't have to change each .dat before every commit.  You just have to make sure you add the "native" setting when you add a new file.

(Small point, certain very stupid editors can leave files with confused mixed line settings, and 'native' wouldn't fix that, but I don't think Kwrite does that.)

EDIT: Please let me know if/when you have set the "native" setting for the text files.  I've got a whole scheme for properly rebranching the pak-britain.experimental git repo, but I simply can't do it until this is done.
« Last Edit: May 06, 2010, 08:42:40 PM by neroden »

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #19 on: May 09, 2010, 04:32:54 PM »
*impatiently tapping fingers*  ;)

At the moment, for pak128.britain, a diff between the 'experimental' git repo and the lovely (thanks again, Ansgar) git mirror of the svn repo gives many spurious differences due to line ending changes.  This makes it hard to merge changes.  I am waiting for line ending normalization of the svn repo, waiting for that to propagate into the git mirror, and then I will normalize the experimental repo and everything should be hunky-dory.  Wait wait wait.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #20 on: May 09, 2010, 04:53:10 PM »
This would be extremely useful for me, too... :-)

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #21 on: May 09, 2010, 09:45:04 PM »
OK, this should all be done in r280.  Hope that is what you were after, so if you could check it and let me know that would be great.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #22 on: May 10, 2010, 05:33:17 PM »
OK, this should all be done in r280.  Hope that is what you were after, so if you could check it and let me know that would be great.

It's looking beautiful.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #23 on: May 12, 2010, 03:37:53 AM »
OK, Mr. Hood, one more request: we missed one file (parameter.mos), which is causing mild annoyance.
Could you please do this:
Code: [Select]
svn propset svn:eol-style "native" parameter.mos
svn commit

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #24 on: May 13, 2010, 09:37:48 PM »
Done

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #25 on: May 14, 2010, 03:32:32 AM »
Done

Thanks, that should be it for that project!

Offline sdog

  • Devotee
  • *
  • Posts: 2039
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #26 on: May 15, 2010, 11:14:12 PM »
@TheHood, jamespetts, neroden, slightly off topic:
How positive are your experiences so far with developing a pakset with git/github?
I ask this mainly because of this thread:
http://forum.simutrans.com/index.php?topic=5124.msg50123

(which i started out of the observation of friends/people who use either svn or git, in an afterthought i maybe went a bit to far with it.)

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #27 on: May 15, 2010, 11:49:40 PM »
It's a little hard to get into Git at first, but, once it's set up, it's a very good tool, and far easier to manage collaboration than with SVN.

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #28 on: May 16, 2010, 04:35:49 PM »
@TheHood, jamespetts, neroden, slightly off topic:
How positive are your experiences so far with developing a pakset with git/github?
I ask this mainly because of this thread:
http://forum.simutrans.com/index.php?topic=5124.msg50123
Extremely positive for me (and I hadn't used it until, what, a few weeks ago?) I'm on Linux, of course.  James Petts should speak to how easy it is to use it under Windows.

Git is very good with things which aren't text files, pictures etc.  Really the best system I've seen for that.

Github is really nice and adds that really useful "network" graph.  But only if your project is all open source, because otherwise it costs money, and I'm a cheapskate.  :)

I will say that we had one nasty problem relating to the SVN/git interaction (and I think James P. still hasn't bitten the bullet and switched repos...).  If the "authoritative maintainer" decides to use SVN and other people decide use git, set up the git mirror of the SVN properly on day one (Ansgar was very helpful) -- if you don't set up the connection correctly on day one you'll have trouble later.  If everyone is using git, of course there's no problem.

Another thing to note: if the "authoritative maintainer" decides to use git and other people really, really want to use SVN, they *CAN*.  Look at this: http://github.com/blog/626-announcing-svn-support , and also this: http://github.com/blog/644-subversion-write-support

FYI, to go off topic, regarding 'open source', when images are made from 3d models, it's a good idea to put the Blender source files in the repository too, not just the .png files, to make it easier for people to edit stuff.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #29 on: May 16, 2010, 05:18:41 PM »
Doesn't adding the .blends make the source repository too big? Or is that not a problem with Github?

Offline neroden

  • Devotees (Inactive)
  • *
  • Posts: 831
  • Nathanael Nerode
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #30 on: May 16, 2010, 06:04:33 PM »
Doesn't adding the .blends make the source repository too big? Or is that not a problem with Github?

How big is "too big"?  Github does not appear to have a size limit (& git also compresses well apparently).  If we are worried about developers' personal computer disk space, for people with HDs less than 100G, then this may be an issue.  It would probably be more of an issue if people edited the .blends often, but I was thinking they'd be more of a "commit once, leave untouched for a few years" thing.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #31 on: May 16, 2010, 06:25:53 PM »
I know that The Hood hasn't put .blends on the SVN for that reason, but it would be good to upload them if at all possible. The Hood - any chance of doing this?

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #32 on: May 16, 2010, 09:33:30 PM »
The main reason I've not put blends on is that the SVN is shared with pak64 and there are only the files up for actually building the pakset with makeobj.  Prissi may have views on whether adding blends to SVN is acceptable or not. 

I have no objection to releasing the blend files (other than people finding out quite how shoddy many of the models are!), but it might be best to release them somewhere else than SVN.  Suggestions?

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 18745
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #33 on: May 16, 2010, 10:42:28 PM »
Github? ;-)

Offline The Hood

  • Moderator
  • *
  • Posts: 2889
  • pak128.Britain developer
Re: Request for TheHood: please set svn:eol-style=native for text files
« Reply #34 on: May 17, 2010, 07:39:31 AM »
Are you suggesting you give me an account on the pak128-Britain-ex git repository and I dump all the blends in a new subfolder?  That might work.  Of course the graphics themselves aren't experimental only, but seeing as you "run" the git repository and seem to want this, that would probably work.