Author Topic: Should Sourceforge Still Be Used?  (Read 6362 times)

0 Members and 1 Guest are viewing this topic.

Offline def

Should Sourceforge Still Be Used?
« on: December 14, 2015, 10:29:16 PM »
Hi all,

Simutrans's Windows download is hosted on Sourceforge. Multiple open source communities have been moving away from Sourceforge; they cite various concerns, and I'd rather direct you towards Sourceforge's wikipedia article than reiterate them here. Personally, I don't trust Sourceforge as a good source of software. Indeed uBlock origin, an ad-blocker, blocks the site completely out of concern for security.

I had a brief search around these forums and couldn't see any discussion about Sourceforge as a hosting provider, and I'm wondering whether a) this discussion has been had in recent years and b) whether this would be the correct place for such a discussion.

My own experience with downloading Simutrans for Windows via Sourceforge isn't great. Shortly after downloading from one of Sourceforge's mirrors, I received a notification from Windows Defender that it had detected a Trojan called Spallowz.A!plock. I have a very clean installation of Windows 10 - I installed on a new build less than two months ago and I've only installed a few, very trustworthy applications - Steam, a few steam games, Java, Minecraft, Firefox, Chrome & LibreOffice. I've not even used Windows heavily in that time; I mainly use the Linux partition. For me, this provides a strong case that either the download was tainted or Sourceforge's mirror's webpage (I can't recall the name) was trapped. The installer itself seemed to be genuine as it looked to be installing Simutrans - a process I aborted after the detection of the trojan. Unfortunately I do not have the installer file to share with you for analysis - whether it was in temporary files or Windows Defender deleted it I'm not sure.

Is there an alternative host from which one can download the game for Windows? I might use this, but I am thinking of installing via Ubuntu instead.
« Last Edit: December 14, 2015, 10:42:36 PM by def »

Offline Spacethingy

Re: Should Sourceforge Still Be Used?
« Reply #1 on: December 14, 2015, 10:50:11 PM »
Couple of questions here:

What file does Defender report was infected / the infection? I do appreciate you say that you've been careful, but you probably want to be sure it was the file off SF.

What file did you download? I think all the Windows Simutrans stuff has been known to trigger false positives (because of the pak downloaders they contain, I think).
http://forum.simutrans.com/index.php?topic=7152.msg69122#msg69122

Here's a VirusTotal report for the installer from Sourceforge:
https://www.virustotal.com/en/file/f34fdce6b71cf6c357ab67b052fa9e9814bcd84d0d81541104a3d78cf30aa315/analysis/

...and here's the same for the .zip:
https://www.virustotal.com/en/file/d581aaab9fa3c8a4b5cbc2edf1c4ca0de93c6d690ed3d7bec951951bd2031f20/analysis/

All looks fine from here, 1/55 unsafe, with the only hit being a heuristic one anyway.

(incidentally, I like VirusTotal's features with exploring the contents of files and any potential dodgy things in them. Never noticed that before!)
Life is like a new Simutrans transformer:

You only get one of them, and you can't have it on a slope.

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 14939
  • Total likes: 334
  • Helpful: 146
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Should Sourceforge Still Be Used?
« Reply #2 on: December 14, 2015, 10:55:26 PM »
I have had uBlock origin giving concern about Sourceforge, too. I am no expert on these issues, but perhaps they should be looked into carefully?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline isidoro

Re: Should Sourceforge Still Be Used?
« Reply #3 on: December 14, 2015, 10:55:52 PM »
A third option is a false positive, that is, the antivirus software detects genuine ST exe as a virus.

Think that part of the program is low-level and can do things that make AV software suspect of a virus while it isn't.  Maybe any of the dev team can shed some light here.

Meanwhile, adding hashes to the files exposed in Sourceforge could be a good measure to ensure that the files haven't been tampered with.

Offline def

Re: Should Sourceforge Still Be Used?
« Reply #4 on: December 14, 2015, 11:09:39 PM »
What file does Defender report was infected / the infection? I do appreciate you say that you've been careful, but you probably want to be sure it was the file off SF.

I'm not sure how I can find this out. I'll get back to you.

What file did you download? I think all the Windows Simutrans stuff has been known to trigger false positives (because of the pak downloaders they contain, I think).
http://forum.simutrans.com/index.php?topic=7152.msg69122#msg69122

I've checked my DL history. I think this is it:
http://skylink.dl.sourceforge.net/project/simutrans/simutrans/simutrans-online-install.exe

Here's a VirusTotal report for the installer from Sourceforge:
https://www.virustotal.com/en/file/f34fdce6b71cf6c357ab67b052fa9e9814bcd84d0d81541104a3d78cf30aa315/analysis/

"Skylink" isn't in there.

...and here's the same for the .zip:
https://www.virustotal.com/en/file/d581aaab9fa3c8a4b5cbc2edf1c4ca0de93c6d690ed3d7bec951951bd2031f20/analysis/

All looks fine from here, 1/55 unsafe, with the only hit being a heuristic one anyway...
Why is one flagged for problems and the rest not? They should be identical files, right?

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 8654
  • Total likes: 283
  • Helpful: 226
  • Languages: De,EN,JP
Re: Should Sourceforge Still Be Used?
« Reply #5 on: December 15, 2015, 12:12:08 AM »
Recently I switched from Avira to Avast as antivirus. I will switch back, I think, because every second time I compile Simutrans or the online-installer I get a warning that I will start a possible Trojan. Well, I hit the submit as safe button frequently, and it seems that Simutrans.exe at least has been taken out of the heuristics of Avast at least on this computer. Nevertheless, the installer as well as simutrans and the paksetdownloader all use techniques that Trojans are using, i.e. compressed code, http downloads, server on high port addresses. Nothing we can really do about.

Any fresh download (with the first three days) will get this Trojan warning, no matter if downloaded from sourfeforge or any other place.

Offline Spacethingy

Re: Should Sourceforge Still Be Used?
« Reply #6 on: December 15, 2015, 12:21:40 AM »
I'm not sure how I can find this out. I'll get back to you.
Have a check at Windows Defender > History > ..., it may appear there.

Quote
I've checked my DL history. I think this is it:
http://skylink.dl.sourceforge.net/project/simutrans/simutrans/simutrans-online-install.exe

"Skylink" isn't in there.

I've checked the hashes, it's identical to the download that comes from the other SF server.

Quote
Why is one flagged for problems and the rest not? They should be identical files, right?
VirusTotal scans one file with many different anti-virus scanners. As I say, judging by the rather generic name that that particular anti-virus scanner gave the .exe, it guessed (heuristically) that something was wrong. That could be either because there is a virus, and VBA somehow is the only engine capable of spotting it (very unlikely, given VBA isn't a major antivirus), or (much more likely) VBA's heuristic scanning is rather too sensitive. As Prissi said, that's just how some anti-viruses work. No solution can ever be 100% right, you sometimes need to use other sources like VirusTotal to make a decision. Incidentally, my Windows Defender doesn't pick anything up with the latest installers from SF.

If you don't want to use the file (and if you feel unhappy, that's completely up to you, security is a personal thing in my view), you could always use the Windows nightlies from nightly.simutrans.com. The "stable" files on SF are pretty ancient and don't have the latest nice features that the official nightly server's files do. You'd still have to get the pak files from SF, but they don't have .exe's in them so they should be fine.
Life is like a new Simutrans transformer:

You only get one of them, and you can't have it on a slope.

Offline Lmallet

  • Devotee
  • *
  • Posts: 725
  • Total likes: 9
  • Helpful: 12
  • 128 on the brain...
  • Languages: FR, EN
Re: Should Sourceforge Still Be Used?
« Reply #7 on: December 15, 2015, 04:45:39 PM »
Trend Micro is also not happy with the installer coming from http://skylineservers.dl.sourceforge.net/project/simutrans/simutrans/simutrans-online-install.exe.  You can get more info by entering the URL in here; http://global.sitesafety.trendmicro.com/.

I've had issues at work with AVs that use reputation lists to determine if attachements are good or not (Norton and Trend Micro come to mind).  If their scanners find a single questionable file on the server (in this case SF), you end up with the whole domain being blocked.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4502
  • Total likes: 151
  • Helpful: 105
  • Languages: EN, NO
Re: Should Sourceforge Still Be Used?
« Reply #8 on: December 15, 2015, 07:13:09 PM »
Can whoever made the installer check if the file served by Sourceforge is exactly the one he made and uploaded? That should either validate or lay to rest the accusations against Sourceforge.

Of course, without Sourceforge, we might lose publicity (Simutrans has been the featured project often lately) and download capacity/speed (they have lots of mirrors).

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 8654
  • Total likes: 283
  • Helpful: 226
  • Languages: De,EN,JP
Re: Should Sourceforge Still Be Used?
« Reply #9 on: December 15, 2015, 09:15:28 PM »
You can check yourself: SF provides a hash to check downloads.

Offline Isaac.Eiland-Hall

  • Benevolent Dictator
  • Administrator
  • *
  • Posts: 3386
  • Total likes: 254
  • Helpful: 90
  • PanamaCityPC.com/support/
    • Facebook Profile
  • Languages: EN
Re: Should Sourceforge Still Be Used?
« Reply #10 on: December 15, 2015, 09:24:48 PM »
In regard to download capacity: My server has a 100Mbit connection (although CPU might be the first limit) - I handled all the downloads easily on older, slower servers; this server is much faster. :)

But that's only one point; although of course I'd be more than happy for downloads to come from the server again, it's more complicated than that, and my understanding is that Sourceforge is much easier to work with by the developers. :)

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4502
  • Total likes: 151
  • Helpful: 105
  • Languages: EN, NO
Re: Should Sourceforge Still Be Used?
« Reply #11 on: December 15, 2015, 09:59:46 PM »
You can check yourself: SF provides a hash to check downloads.

But their hashes would naturally be made after any modifications they supposedly make. The accusation is not that someone is tampering with the download after it leaves Sourceforge.

Offline sdog

Re: Should Sourceforge Still Be Used?
« Reply #12 on: December 15, 2015, 11:20:57 PM »
Can whoever made the installer check if the file served by Sourceforge is exactly the one he made and uploaded? That should either validate or lay to rest the accusations against Sourceforge.
You cannot prove the absence of something by sampling. Such a check tells you only that for simutrans, at that moment, the files are not tampered with.

Two questions remain, can SF still be trusted. They might have learned something after the GIMP fiasko. With the hype and witchhunt prone nature of the net's public at the moment, it might even have been extremely exagerated. But that leads to the second question do users and the security software they have installed trust SourceForge (SF) sufficiently?

Don't forget we're in the times where people are more used to app-stores and repositories. I'd certainly not install some binary from a website on my machine. However, I'm not representative as Linux user, and can rely on a system of trust for the packages I get through my Linux distro (and Steam).

Quote
Of course, without Sourceforge, we might lose publicity (Simutrans has been the featured project often lately) and download capacity/speed (they have lots of mirrors).

That really depends on SF still having enough influence that their publicity matters. While Simutrans being project of the month at first certainly was a reason to be proud: Who reads that? Did you pay attention to other projects of the month before?


There is a third question related. SF is obviously in trouble. Taking over abandoned projects to monetize can only be an act of desperation. We ought to face that it is not unlikely that they go down in the near future. Make sure all the SVNs there are well backed up. Do not link users to anything on SF but, always link to the relevant DL page on simutrans.com, and have that link to SF

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 14939
  • Total likes: 334
  • Helpful: 146
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Should Sourceforge Still Be Used?
« Reply #13 on: December 15, 2015, 11:30:15 PM »
Happily, the SVN is already backed up on Github.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline sdog

Re: Should Sourceforge Still Be Used?
« Reply #14 on: December 16, 2015, 12:29:39 AM »
Happily, the SVN is already backed up on Github.

That is indeed good. However, as far as I know, that pertains to the Simutrans source code only. The pak-sets are an entirely different matter.* I expect however, that the devs would not like to migrate from SVN to git. That means that the actual SVN might need some form of redundancy/backup.

...

Wait, the actual simutrans source code isn't on SF anyway, but on a private SVN server, isn't it? SF is only used as filehoster for the (stable) binaries. Nightlies are hosted on their own server. That leaves then only the pak-set on SF SVN?



*I've ran a pak128 and pak96 comic mirror for a while, the pak128 github backup is outdated, the pak96 git mirror broke in parts due to svn-git inconsistencies with regard to spurious changes and in parts due to my incompetence with git/svn.



I ought to see that as another incentive to get the git mirror project running again. A course of action could be, getting a working mirror for pak128 first. Running updates from my desktop. Going to the other paksets. Finally trying the hard nut pak96 again.  Then thinking about getting this on a server (or a raspberry pi). When we have regularly updated git repos on github, it is also fair to use it to host our binaries (as another mirror, redundant to SF). If you don't see results posted by me within a fortnight, would someone please be so kind to give me a (proverbial) good kick in the arse?

ps.: Isaac, your profanity filters are not American only, you take the international seriously.
« Last Edit: December 16, 2015, 12:42:53 AM by sdog »

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4502
  • Total likes: 151
  • Helpful: 105
  • Languages: EN, NO
Re: Should Sourceforge Still Be Used?
« Reply #15 on: December 16, 2015, 06:53:36 AM »
You cannot prove the absence of something by sampling. Such a check tells you only that for simutrans, at that moment, the files are not tampered with.

Two questions remain, can SF still be trusted. They might have learned something after the GIMP fiasko. With the hype and witchhunt prone nature of the net's public at the moment, it might even have been extremely exagerated.

I don't expect them to be tampering with the files dynamically at random, although it is technically a possibility. However, if the original installer, untampered with anything, still triggers anti-malware tools the same way, then there is no reason to point the finger on Sourceforge. There will be no proof that they have done anything wrong against us. That was my point. Although some of what I can find in a hurry about what Sourceforge did uses adware and malware interchangeably, it seems adware is the accurate word. The reports we are getting now are more in the true malware (excluding annoying, but harmless adware) category, which seems an unlikely choice. We need more proof.

Happily, the SVN is already backed up on Github.
That is indeed good. However, as far as I know, that pertains to the Simutrans source code only.
[...]
Wait, the actual simutrans source code isn't on SF anyway, but on a private SVN server, isn't it?
Yes. Sourceforge SVN only contains some of the pak sets, of which pak64 is one.

There is a third question related. SF is obviously in trouble. Taking over abandoned projects to monetize can only be an act of desperation. We ought to face that it is not unlikely that they go down in the near future. Make sure all the SVNs there are well backed up. Do not link users to anything on SF but, always link to the relevant DL page on simutrans.com, and have that link to SF
Isn't that last thing the one thing the OP is saying that we absolutely should not do? That we can use Sourceforge for anything but downloads (as long as we have backups). On the other hand, if we use it for other things, we might not be able to prevent there being malicious downloads there as well. There is also the fact that it was abandoned projects, or projects claimed to be abandoned, that was taken over by Sourceforge for adware. Maintaining our presence on Sourceforge might be the safest choice, or we might get an evil "twin" competing with us in the top of google's search results page.

Offline sdog

Re: Should Sourceforge Still Be Used?
« Reply #16 on: December 16, 2015, 05:39:45 PM »
Isn't that last thing the one thing the OP is saying that we absolutely should not do? That we can use Sourceforge for anything but downloads (as long as we have backups). On the other hand, if we use it for other things, we might not be able to prevent there being malicious downloads there as well. There is also the fact that it was abandoned projects, or projects claimed to be abandoned, that was taken over by Sourceforge for adware. Maintaining our presence on Sourceforge might be the safest choice, or we might get an evil "twin" competing with us in the top of google's search results page.
This is meant in case simutrans keeps its downloads on SF. Linking to one of our own domains and redirecting reduces the number of dead links, in case SF goes down or gets worse. I don't see a consensus on leaving SF after all, and there might be no reason. The last news articles of SF sneaking in adware were published in June. It seemed transient. Noteworthy, SF was shortly thereafter sold.

Adware is malware, unless the user installs it on purpose to get adds. That it only nests itself there in order to spam you instead of enslaving your machine in a bot net is of course less severe. In the same sense that burglary is less severe than armed robbery, yet both are criminal offences.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4502
  • Total likes: 151
  • Helpful: 105
  • Languages: EN, NO
Re: Should Sourceforge Still Be Used?
« Reply #17 on: December 16, 2015, 07:15:54 PM »
Adware is malware

It seems so, but adware was still the most accurate word, just as computer game is a more accurate description for Simutrans than computer program. I also lacked a word describing programs that damage or steal data or the computer as a whole (viruses, worms, trojan backdoors/rootkits). Adware may do some spying, but not necessarily, and so can any web page. Though, viruses can actually be harmless as well.

Offline isidoro

Re: Should Sourceforge Still Be Used?
« Reply #18 on: December 16, 2015, 10:03:01 PM »
But their hashes would naturally be made after any modifications they supposedly make. The accusation is not that someone is tampering with the download after it leaves Sourceforge.

But we can compare their hashes with ones taken from our files, can't we?

Offline jamespetts

  • Simitrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 14939
  • Total likes: 334
  • Helpful: 146
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Should Sourceforge Still Be Used?
« Reply #19 on: December 17, 2015, 01:10:29 AM »
Pak128.Britain (both Standard and Experimental) are backed up on Github. Perhaps the same should be done for Pak64, Pak128 and the others?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4502
  • Total likes: 151
  • Helpful: 105
  • Languages: EN, NO
Re: Should Sourceforge Still Be Used?
« Reply #20 on: December 17, 2015, 06:05:30 AM »
But we can compare their hashes with ones taken from our files, can't we?

Yes, but we still need our file from whoever has/had it. And once/if we have that, simply comparing the files directly might be easier than calculating our hash and comparing the hashes.

Pak128.Britain (both Standard and Experimental) are backed up on Github. Perhaps the same should be done for Pak64, Pak128 and the others?

I'd rather have a SVN backup, so we can keep the revisioning.

Offline def

Re: Should Sourceforge Still Be Used?
« Reply #21 on: December 17, 2015, 08:11:31 PM »
For me, this provides a strong case that either the download was tainted or Sourceforge's mirror's webpage (I can't recall the name) was trapped.

I would like to point out that it hadn't occurred to me that windows defender might give a false positive, as spacethingy pointed out. I wouldn't have used the word "strong" in my original post if it had. Something I did omit from the original post was that I experienced some internet connectivity issues shortly after attempting to use the installer. I don't have the skill to isolate the cause for this; other devices' connectivity was unaffected but at the same time they connect via wifi instead of over Ethernet/AC. The issue seemed to resolve itself somehow.

---

Would a good solution to the hosting problem be to run two hosting providers concurrently?

Offline An_dz

  • Web Admin
  • Administrator
  • *
  • Posts: 2504
  • Total likes: 261
  • Helpful: 88
  • D'oh
    • by An_dz
  • Languages: PT, EN, (it, de)
Re: Should Sourceforge Still Be Used?
« Reply #22 on: December 17, 2015, 08:28:37 PM »
One mirror of the files can be GitHub, we already have an official account and we could upload there. GitHub allows uploading any file type on their release feature.

Offline isidoro

Re: Should Sourceforge Still Be Used?
« Reply #23 on: December 17, 2015, 10:02:23 PM »
Yes, but we still need our file from whoever has/had it. And once/if we have that, simply comparing the files directly might be easier than calculating our hash and comparing the hashes.
[...]

It depends of what you consider easy.  For me it's easier to calculate the hash locally and compare it with the published hash.  And some bandwidth is saved too.

Offline sdog

Re: Should Sourceforge Still Be Used?
« Reply #24 on: December 17, 2015, 10:05:49 PM »
I'd rather have a SVN backup, so we can keep the revisioning.
I don't think it would be wise to have only a git mirror when the actual workflow is in SVN. That doesn't mean however, that we mustn't have a git mirrror. From my experimenting with git-svn, this can range in difficulty from trivial to extremely hard. In the former case, nothing speaks against it. (the predicament is, that the one thin where a git mirrror is specifically requested is the difficult case ;-)

Pracitcal part of the reply:
Conversion from SVN to Git with git-svn keeps the SVN history, each SVN revision becomes a git commit. There are now two questions associated with this (a) is all information retained? (b) do you need some specific datum of information that is lost, not easy enough to find, incomplete?

With (b) I'm asking for things that SVN users only really know, based on their workflow, perhaps you really need the revision number? Or the fact that each repository has a sequential numbers with gaps, since the revision counter is incremented for each submission to SVN, might cause problems. The question is so vague as I can't really anticipate what is really needed.

Could you perhaps have a look at the git repository of pak128
https://github.com/simutrans/pak128
in comparison to the SVN and tell me if and what is wrong, if something is needed or desired?

Edit: better have a look at the source mirror:
https://github.com/aburch/simutrans

One mirror of the files can be GitHub, we already have an official account and we could upload there. GitHub allows uploading any file type on their release feature.
Having more than one mirror doesn't hurt, with one caveat. All mirrors have to be kept up-to date. Now, that is perhaps something we ought not load on Prissi's shoulders who publishes the releases. However, simply copying the file from SF and loading it to other mirrors would defeat the point.

However, simutrans git does at the moment not have the actual simutrans source code mirror. That is done by Ansgar (aburch). It is not only impolite, but also against github's rules to use it as a fileserver only. We can distribute the binaries, of the sourcecode we have there. The simples way could be to just clone and pull from aburch regularily. Can we run into trouble in case he stops his service? Perhaps I can ask him if he could add a repo under the simutrans account as a second git remote location, that is next to no effort.

This also fits to something that is not directly related to SF, but to the release process. It was very dissapointing how the last release was overlooked. I simply lost the habit to check the relase thread regularly and haven't any rss reminder system set up. But I've not been the only one who overlooked it.

Can we think of a way that does (a) not increase Prissis workload, (b) makes sure all who disseminate information are informed, (c) provides those who maintain official mirrors (to come) with the original file, as compiled by Prissi?

Offline Isaac.Eiland-Hall

  • Benevolent Dictator
  • Administrator
  • *
  • Posts: 3386
  • Total likes: 254
  • Helpful: 90
  • PanamaCityPC.com/support/
    • Facebook Profile
  • Languages: EN
Re: Should Sourceforge Still Be Used?
« Reply #25 on: December 18, 2015, 12:58:45 AM »
Not pushing anything, but hopefully either helpful or merely useless info - because I don't really fully understand git/svn. I mean, some basics, but not enough to really use...

The effort to get svn on the server failed; but git (at least the command) is built-in and working. In case that's helpful. Apologies if not.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4502
  • Total likes: 151
  • Helpful: 105
  • Languages: EN, NO
Re: Should Sourceforge Still Be Used?
« Reply #26 on: December 18, 2015, 06:49:19 AM »
It depends of what you consider easy.  For me it's easier to calculate the hash locally and compare it with the published hash.  And some bandwidth is saved too.

Well, I was assuming that the one had downloaded the installer from Sourceforge as well in order to check if they get the warnings as well. For everyone but the developers, I would expect that they had downloaded the installer to install Simutrans already.

Conversion from SVN to Git with git-svn keeps the SVN history, each SVN revision becomes a git commit. There are now two questions associated with this (a) is all information retained? (b) do you need some specific datum of information that is lost, not easy enough to find, incomplete?

With (b) I'm asking for things that SVN users only really know, based on their workflow, perhaps you really need the revision number? Or the fact that each repository has a sequential numbers with gaps, since the revision counter is incremented for each submission to SVN, might cause problems. The question is so vague as I can't really anticipate what is really needed.

Git does not track file history. Therefore, Git does not support merging just a subset of files from one branch to another. I think SVN does. Such merges in SVN can not be converted properly to Git. However, Simutrans has to my knowledge never used branches, so that should not be a problem for Simutrans. Simutrans does however rely on SVN revision numbers for versioning between releases (nightlies and other local builds), for both of the game itself and at least pak64. Git has no such thing, using random (to the human mind) hashes instead, which is useless for comparative versioning. It is very useful when someone reports a bug in a particular revision, to easily see if the revision number I am using is higher or lower. Another thing that even the Git fanboys at work have found problematic, is that Git stores the date a code change was "written" (committed locally actually), but not the date when that code changes was included in the public repository (pushed), which may be days, even months or more, later. (The notion of a universal public repository is actually unknown to Git.)

Offline sdog

Re: Should Sourceforge Still Be Used?
« Reply #27 on: December 18, 2015, 05:37:38 PM »
Git does not track file history. Therefore, Git does not support merging just a subset of files from one branch to another. I think SVN does. Such merges in SVN can not be converted properly to Git. However, Simutrans has to my knowledge never used branches, so that should not be a problem for Simutrans.
Thank you very much for the reply Ters.
I'm not certain I understand this first aspect. With git you can track for each file which commit changed it. This also works line by line. Git clients and github have a 'blame' mode where for each line the commit where it was touched last is shown. But I think this is still different from what you meant?

Quote
Simutrans does however rely on SVN revision numbers for versioning between releases (nightlies and other local builds), for both of the game itself and at least pak64. Git has no such thing, using random (to the human mind) hashes instead, which is useless for comparative versioning. It is very useful when someone reports a bug in a particular revision, to easily see if the revision number I am using is higher or lower.
Indeed in a git workflow one would not have the continuous revision number. Simply importing from SVN it is there. git-svn writes it to the commit message, here is an example:
Quote
[Pak128] Finally added for testing new overall station.
See: http://forum.simutrans.com/index.php?topic=12899.msg129020#msg129020
It includes some roof bugfixing as pointed out in the forum.

git-svn-id: https://simutrans.svn.sourceforge.net/svnroot/simutrans/pak128@1391 e96eaf88-093e-0410-925f-862d53288611
If the revision number is too inconspicious, I can see if the default message generated by git-svn can be changed in that regard.

Quote
Another thing that even the Git fanboys at work have found problematic, is that Git stores the date a code change was "written" (committed locally actually), but not the date when that code changes was included in the public repository (pushed), which may be days, even months or more, later. (The notion of a universal public repository is actually unknown to Git.)

I don't see why it would matter when the checkout from SVN was done and the file pushed to the git repository? Isn't the date it was commited to SVN the only one that matters? If you need the date when it was checked out from SVN and pushed to github we can simply log this to a file that is included in the repository, and pushed together with the actual commit.*



* Perhaps it does help those in your git-friendly mates too.
Let's call the file .pushup.log
a short script that is used instead of push
Code: [Select]
git fetch
git checkout -m <branch> .pushup.log
date >> .pushup.log
git add .pushup.log
git commit --amend
git push <origin> <branch>

This checks out the .pushup.log from the branch you want to push to,
merges it into your local code, adds it to your commit, and pushes it.
(there have to be a few checks, e.g. if one has a commit that has not been pushed yet.)

edit: an even simpler idea...
I've been locked into linear svn like thinking, we don't need that.
  • grep from git stat the number of local commits that were not pushed yet n
  • get from git log the IDs  of the last n commits
  • for all i in n : let s= .push-dates/ID(i); date > s; git add > s
  • continue as before
« Last Edit: December 18, 2015, 05:54:34 PM by sdog »

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4502
  • Total likes: 151
  • Helpful: 105
  • Languages: EN, NO
Re: Should Sourceforge Still Be Used?
« Reply #28 on: December 18, 2015, 06:56:07 PM »
I'm not certain I understand this first aspect. With git you can track for each file which commit changed it. This also works line by line. Git clients and github have a 'blame' mode where for each line the commit where it was touched last is shown. But I think this is still different from what you meant?

Git tracks commits. Commits contains information about how files looked at that time (sort of). A commit has zero (rarely), one or two parents. In other VCSes, the individual files have parents, not so in Git. If a commit has file called Readme, and the previous commit has file called Readme, the tools traversing the Git commit log will assume it is the same file. If the previous commit does not contain a file called Readme, but has a another file with roughly the same contents, the tools will assume the file was moved/renamed or copied. In other VCSer, you must therefore explicitly tell them that the file was moved, renamed or copied from someplace else, or there will be a break in the history. With Git, you have the opposite problem. Unrelated files may have their histories joined, although it is probably rare. You still have to inform Git to add the file with the new name to the commit, though.

The lack of tracking of files means that Git can't merge individual files from one branch to another. Although you can merge the branches and dismiss the changes from the other files, the branches will have been marked as merged, so that any later merge will only consider changes since the first merge. (This might have been the case in SVN earlier as well, but I think they've changed it.)

Indeed in a git workflow one would not have the continuous revision number. Simply importing from SVN it is there. git-svn writes it to the commit message

Yes, but we would no longer be importing from SVN if our SVN dies, so what Git does when importing from SVN is irrelevant. In fact, when we are using SVN, what the Git we are not using does at all is irrelevant. The issue is whether or not Git on its own can give us the SVN features we really depend on, or at least like very much.

I don't see why it would matter when the checkout from SVN was done and the file pushed to the git repository? Isn't the date it was commited to SVN the only one that matters?

Once again, I was writing about a pure Git scenario.

Offline sdog

Re: Should Sourceforge Still Be Used?
« Reply #29 on: December 18, 2015, 07:33:18 PM »
Quote
Git tracks commits. Commits contains information about how files looked at that time ...
Ters, thank you for your clarification. I misunderstood you before. This could happen to be a severe obstacle when migrating from SVN to git. Fortunately it is not the case here. Within a pure git workflow it were surprising if it caused trouble, unless someone assumes SVN behaviour.

Quote
Yes, but we would no longer be importing from SVN if our SVN dies, so what Git does when importing from SVN is irrelevant. In fact, when we are using SVN, what the Git we are not using does at all is irrelevant. The issue is whether or not Git on its own can give us the SVN features we really depend on, or at least like very much.

Unless the devs would suddenly prefer git, which I don't see, and for which is no need, there is no reason whatsoever to abandon subversion as versioning system.

I am talking about having a git mirror of the SVN, to serve in part as a redundant backup and in part for its own purposes. That there ought to be  SVN backups and mirrors of the SF repositories remains as a problem to be solved unconcerned by any git things. If the SVN server died, we would simply import to git from the SVN which were to replace it.

I've noticed this in a previous discussion, you tend to defend SVN as if I were to suggest you oughtn't use it. Because I like arguments I get drawn into replying. Yet I certainly don't want to involve myself in any way with your versioning it workflow. When it works it is perfect, when it doesn't, it is the devs problem and prerogative to improve.

Nonetheless, the git mirror must not lack any relevant information the SVN provides. In that regard, your feedback is invaluable.

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4502
  • Total likes: 151
  • Helpful: 105
  • Languages: EN, NO
Re: Should Sourceforge Still Be Used?
« Reply #30 on: December 18, 2015, 09:53:39 PM »
I am talking about having a git mirror of the SVN, to serve in part as a redundant backup and in part for its own purposes. That there ought to be  SVN backups and mirrors of the SF repositories remains as a problem to be solved unconcerned by any git things. If the SVN server died, we would simply import to git from the SVN which were to replace it.
How well does that work? Has anyone ever bothered perfecting a Git to SVN conversion? If we want to backups of or Sourceforge SVN repos, we should make proper backups. We should also have similar backups of our other (main) SVN repo.

Nonetheless, the git mirror must not lack any relevant information the SVN provides. In that regard, your feedback is invaluable.
Since I doubt the feasibility of going from Git to SVN, the only purpose of SVN information in Git is so that those using it can correlate its commit log to the official revision log in SVN.

Offline sdog

Re: Should Sourceforge Still Be Used?
« Reply #31 on: December 18, 2015, 10:22:34 PM »
How well does that work? Has anyone ever bothered perfecting a Git to SVN conversion? If we want to backups of or Sourceforge SVN repos, we should make proper backups. We should also have similar backups of our other (main) SVN repo.
I've never done a SVN backup, I suppose it is rather straightforward, following the instructions you posted.

If the backup is installed on a new SVN, and the structure of the SVN is not changed, it might be as simple as chaning the URL of the svn in .git/configuration, to checkout the new location of the SVN instead of the old one. Keeping the git mirror running. That the SVN is backed up as discussed above is understood.

Was 'git to SVN' a typo, otherwise I cannot follow you why that ought to be ever done?
Quote
Since I doubt the feasibility of going from Git to SVN, the only purpose of SVN information in Git is so that those using it can correlate its commit log to the official revision log in SVN.
Why would anyone go from SVN to Git to SVN? That seems to be a certain path to failure.

Of course, at the point we do the git mirror, we cannot know what information someone might need and what not. Therfore it is best to aim not to omit any information. That is why I asked you.

Offline HarrierST

Re: Should Sourceforge Still Be Used?
« Reply #32 on: December 19, 2015, 03:20:11 AM »
An old world war saying -"loose  words cost lives".

I can not see the point of discussing the  Con's of particular AV software on a site like this. Your just helping  a potential hacker do some damage.  :police:

Offline Ters

  • Coder/patcher
  • Devotee
  • *
  • Posts: 4502
  • Total likes: 151
  • Helpful: 105
  • Languages: EN, NO
Re: Should Sourceforge Still Be Used?
« Reply #33 on: December 19, 2015, 07:35:51 AM »
Was 'git to SVN' a typo, otherwise I cannot follow you why that ought to be ever done?Why would anyone go from SVN to Git to SVN? That seems to be a certain path to failure.

Because, as you yourself seem to agree to, the developers seem unwilling to use Git (for various reasons). If we lose our current SVN repo, then we need to establish a new SVN repo. If our backup is a Git repo, it follows that out new SVN repo must be established from the Git backup.

Offline isidoro

Re: Should Sourceforge Still Be Used?
« Reply #34 on: December 20, 2015, 12:19:25 AM »
I think that a proper SVN backup should be done regularly of the main SVN server and kept elsewhere, for simple security policy.  The forum machine can be a good place to keep the replicas.