News:

Simutrans Wiki Manual
The official on-line manual for Simutrans. Read and contribute.

SVN is down

Started by An_dz, March 06, 2014, 06:06:10 PM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

kierongreen

Last time I seem to remember Tron didn't realise there was a problem for a while?

prissi

I send an email, not sure what else to do. But I did it from my english address, maybe it got stuck in his filter. I resent it from more old german account today.

Isaac Eiland-Hall

I'm not pushing to put on the server, just letting y'all know that it's now set up. Haven't copied things over yet, but they're clustered together so I can put new things on the new server already. :)

Ters

Quote from: kierongreen on March 14, 2014, 07:48:24 PM
Last time I seem to remember Tron didn't realise there was a problem for a while?
While not seeing that one's own DNS doesn't work is understandable, not checking e-mail at least once a week is getting rare, especially for someone running a server. He could be climbing Mount Everest, skiing to the South Pole or hiking through the Amazon rainforest, but there are worse reasons for being incommunicado that seem more likely.

Quote from: prissi on March 14, 2014, 09:34:33 PM
I send an email, not sure what else to do. But I did it from my english address, maybe it got stuck in his filter. I resent it from more old german account today.
The problem with e-mail is that it's very private. If one stops checking e-mail, then that's it. No one will really know. If you don't answer the phone, the one calling will still get some kind of feedback, whether the phone is ringing but simply not aswered, whether the phone is turned off or out of reception range, whether the subscription is cancelled, or maybe someone else answers who knows what's going on. With snail mail, either the letter is returned by the postal service, or someone else will answer it, or the recipient is actively ignoring you. This is a more personal type of contact, which in the worst case scenario can be unpleasant and awkward.

(I have a tendency to think about worst case scenarios, whether I find them likely or not.)

Dwachs

If moving to a different svn server, it would be good to also restore the github mirror. Otherwise everything github-based would have a hard time.
Parsley, sage, rosemary, and maggikraut.

Ters

I'm not sure how/if Git-mirrors will handle a switch to a new Subversion repo (as opposed to just relocating the one we used to have). I guess it depends on whether the main mirror can keep "pulling" into the same branch.

prissi

You can have something like SVN checking out and pushing this to github. That works with any revision system and very little magic of scripting.

It seems Christoph does not get my emails. I have to assume he is on holiday.

Ters

Quote from: prissi on March 16, 2014, 09:17:57 PM
I have to assume he is on holiday.

We can only hope, but for how long?

prissi

Otherwise we can check in the git archieve into sourceforge (for isntance) ... It will mess up revision numbers, but that is proabably only a minor issue, since nobdoy will built old versions for networkgaming.

Markohs

#44
Anyone remembers is tron's server was like this:

svn://tron.homelinux.blablabla/simutrans/simutrans/{trunk,branches,tags}

*or*

svn://tron.homelinux.blablabla/simutrans/{trunk,branches,tags}

Had 2 levels of "simutrans or just one?

EDIT: Nothing, it was option 1

Ters

Why would that matter? A new repo would be a completely new URL anyway.

Markohs

Was a problem I had importing the git svn copy Dwachs sent me.

I've rebuilt the svn repo, but it's a bit of a mess, author is just appended to the commit comment, the timestamps are gone too. It's taking hours to complete. Allowing commit details to be changed (this can be done), someone could make a script to update all commits with the correct value, after.

I think it's the  best think we can get without a "svn export" of the original repo.

If we are moving to github, there are changes to be made to github too, because the current code relies on svn versions to work.



Ters

Quote from: Markohs on March 17, 2014, 05:18:14 PM
I've rebuilt the svn repo, but it's a bit of a mess, author is just appended to the commit comment, the timestamps are gone too.
That won't do.

Quote from: Markohs on March 17, 2014, 05:18:14 PM
If we are moving to github, there are changes to be made to github too, because the current code relies on svn versions to work.
That particular code becomes nearly pointless with Git anyway.

Markohs

So, what's what we are doing to do then? Just use github and that's it? Who has the permissions on https://github.com/aburch/simutrans ? Are we using this one as master or we use a new one? Who's modifying the code to not rely on the revision.h serial number?

Ters

The first question is how long we should keep hoping that the maintainer is on vacation and when to start assuming that something worse has happened.

And Dwachs' Git copy that contains everything seems like the best starting point if we decide to abandon the previous SVN repo.

Markohs

 Yep, that makes sense.

If anyone is interested (even it's not very useful, since the repo lacks too much details), and for reference, the sequence to import Dwachs's repo copy into a fresh SVN repo is:

(Dwachs git copy is in "~/T2", and you are in another directory. )
(setup new svn repo so it has write access to anon, temporally, till you finish dumping)
(git will use around 1.5 Gb of memory while doing this, have it in mind, process will take around 5 hours)


svn mkdir --parents svn://NEWSERVER/simutrans/simutrans/{trunk,branches,tags}
git svn clone svn://NEWSERVER/simutrans
cd simutrans/
git remote add origin -f ~/T2/
git fetch origin master
git checkout -b old_master origin/master
git rebase --onto master --root
git svn dcommit -v --add-author-from --use-log-author

Dwachs

not sure, how to proceed. I sent another email. The owner of the repository seems to alive ;) No email answer still.

I would prefer to wait for him. Setting up another server will break many things ...
Parsley, sage, rosemary, and maggikraut.

Markohs

Just for the record, this is the result of the import, not really good:


Ters

Quote from: Dwachs on March 17, 2014, 07:49:19 PM
not sure, how to proceed. I sent another email. The owner of the repository seems to alive ;) No email answer still.

I would prefer to wait for him. Setting up another server will break many things ...

Well that's good news!

Setting up another server won't break things. Setting up another repo will. Moving the repo seems like a good idea. Isaac's server seems more stable, and isn't on dial-up with dynamic IP. I think also it's very important that two persons have the key, if not to the whole server, so at least to the file storage areas.

Setting up some automated back-up might also be a good idea. If the dump file isn't too large, I could download it from time to time if it is made available somehow.

Markohs

I'll set up a svnsync backup server as soon as we recover our main svn server, just to be sure in the future.

Markohs

Okay, before seting this up, I'll point to what I think should be the installation, so we can agree on the structure, before doing anything, now that the data is safe and tron admin is alive and answering.

I'd just install a main subversion server in:

svn://svn.simutrans.com/simutrans

Other projects in simutrans could get also their own svn repo, that's no problem.

I'd send the root passwords to prissi and dwachs, so there is more than one person capable of fixing things up if we fall into disaster again.

I'd use "svnserve" for this, the server shipped with subversion. I'd copy the pre-commit hooks tron has, who'd need to send them to me, or tron can be allowed to log into the server and setup the server himself.

As for backup, I'd just propose any member that has a server on the internet that wants to set up a mirror, setup read-only mirrors in their own machines, and post the info in some descriptive page. Having two RW mirrors (both in isaac's machines, or not) it's more complicated, and can give lots of problems in the future if they lose sync.

It's also very important we have mirrors outside isaac's servers, because you never know what can happen to his ISP, and we have to have a plan for those situations, even if they are extremely unlikely to happen.

About github, I'd designate one master git-svn repo, prissi's or dwachs one, that's able to push patches into the "official" svn repo, given previous approval of them. git pushes should go to them, they'd the be official git repo. This way we can get useful modifications from github pushed into subversion, from github users.

I'd keep the revision numbers for now, but someone should study where are revision numbers used atm and design a new system to stop using them.

What do you think?

As soon as we get to some agreement, we can start the tests.

Dwachs

Github: we could ask aburch to move the aburch/simutrans repository to the simutrans organization at github:

https://github.com/simutrans

which would be the natural choice of 'master' git server. Then commits to the svn repository can be pushed to the github mirror in a post-commit hook. 

In order to incorporate contribution from others in github, the development needs to take place in github as well. Git-svn is  constantly rebasing against the svn repo, which does not work well with github, I think.

Imho we should stick to using svn as the main development repository. We are used to this system. It provides meaningful revision numbers for nightly compilation and bug reporting.

Locally one still can use git-svn to develop. (I use smartgit http://www.syntevo.com/smartgithg/, which runs on Win/Linux/Mac).
Parsley, sage, rosemary, and maggikraut.

Markohs

I'm okay with developing only in svn, but, in fact it's my prefered option for the develompent environment. But isn't possible to have a git-svn repo, someone keeps up to date with svn, as https://github.com/aburch/simutrans does now, and make it *also* able to accept pull requests from other people, and *also* capable to push those changes *into* svn?

That whould widen our ways of contributing code, and can maybe get valuable new coders.

But ofc, we'd keep out current list of developers, that commit straight to subversion, as we do now.

It whoudn't be a active development git, just a git where some of our developers whould volunteer to review pull requests, and accept them if applicable, making them end accepted in svn automatically with a git svn dcommit? (they can appear in svn as done by the user "git", with the original commiter in the comments, like in the SS I posted.



No idea, I think it's possible, even I don't really know git much.

The github repo being named https://github.com/simutrans makes a lot of sense to me.

Ters

I think it's somewhat icky doing the integration from git to svn on github. What little I know of Git is that rebasing should be done before pushing. Messing with the history in a public repo will just cause chaos if someone manages to pull before the rebasing is done.

Markohs

So, shall I install the master repo then? Dwachs, can you ask tron to send us his hooks?

Specially the pre-revprop-change, and pre-commit one.

Or, if you prefer, I can setup svn.simutrans.com as a read-only mirror of tron's one.

About the git issue... As you wish, we can allways give it a try, having backups and we'll have we can allways sort it up is we get a mess anytime.

I think that if the person administering the git, does periodic svn fetches (we can do this in a post-commit hook in subversion), and rebases his changes before dcomitting back carefully, all should be okay.

Dwachs

Quote from: Markohs on March 18, 2014, 04:16:24 PM
I'm okay with developing only in svn, but, in fact it's my prefered option for the develompent environment. But isn't possible to have a git-svn repo, someone keeps up to date with svn, as https://github.com/aburch/simutrans does now, and make it *also* able to accept pull requests from other people, and *also* capable to push those changes *into* svn?
That would be ideal. However, as fas as I understand, git-svn cannot push merges to the svn. Any pull request from somebody at github would need to be rebased to the svn-head before dcommitting. Which kills the advantage of this workflow. I do not know whether this would be possible without direct access to the local repository containing the git-svn.

Quote
The github repo being named https://github.com/simutrans makes a lot of sense to me.
It would be https://github.com/simutrans/simutrans then.
Parsley, sage, rosemary, and maggikraut.

Ters

Quote from: Markohs on March 18, 2014, 05:05:30 PM
About the git issue... As you wish, we can allways give it a try, having backups and we'll have we can allways sort it up is we get a mess anytime.

Going back to a backup would only mess things up more. In a sense, it's not the master that gets messed up if you rebase in a public repo, it's all the clones, because they are not rebased as far as I understand it. From what I've heard/read, rebasing is done locally on your own computer before anyone else sees what you're doing.

Markohs

Quote from: Ters on March 18, 2014, 06:39:00 PM
Going back to a backup would only mess things up more. In a sense, it's not the master that gets messed up if you rebase in a public repo, it's all the clones, because they are not rebased as far as I understand it. From what I've heard/read, rebasing is done locally on your own computer before anyone else sees what you're doing.

Yep it's local, but if after the commit to svn, he see he has messed up, he can allways go to svn, rever that update, and commit straight in svn again, reverting the changes.

Markohs

Quote from: Dwachs on March 18, 2014, 06:37:13 PM
That would be ideal. However, as fas as I understand, git-svn cannot push merges to the svn. Any pull request from somebody at github would need to be rebased to the svn-head before dcommitting. Which kills the advantage of this workflow. I do not know whether this would be possible without direct access to the local repository containing the git-svn.

You know better than me git!

I just read http://git-scm.com/book/en/Git-and-Other-Systems-Git-and-Subversion and looked like doable, not very hard, but I guess we can just use subversion and leave github as a RO copy, as it's now.

Ters

Quote from: Markohs on March 18, 2014, 06:52:32 PM
Yep it's local, but if after the commit to svn, he see he has messed up, he can allways go to svn, rever that update, and commit straight in svn again, reverting the changes.

Then I misunderstood you. Since you wrote "administering the git", I thought you meant the main git repo, that pulls from svn and which everybody clones (and further makes pull request against). That repo can "never" go back. What each person does to his personal git clone of svn is none of our business. They can't commit back to svn anyway, except our core developers, but they don't seem to like git anyway.

Markohs

No, I was refering exactly to what you expressed. Why can't that git repository, which everybody cloned and can receive pull requests, get svn updates, and push them back to svn when the admin desires so?

And, if he really messed things up, and the mess ended in svn too, *given he also has direct access to svn*, can't he fix the mess in svn, commit, and receive the "fix" in his git again?

prissi

I think the stuff in simubase is obsolete and what has not been allowed to be used due to copyright is not missed any more.

When building the SVN it was made from importing the ZIP files from 84.22.1 to 99.00 or so. Until that any release was one revision, and the author was wrong anyway (it was tron, since he ran the script). As such, I am not overly worried about loosing revision numbers (or if importing to svn on sf rather offsetting them with the current on).

As this years seems to get very busy my contributions to simutrans will be less. I could live if we switch to github. I will fight with git too ... (Probably I will be back to submitting patches, but why not).

Git also has a revision number. git describe on github simutrans gives you:
"v111.3.1-1238-g320599e"

v111.3.1 is the last tag, 1238 is the number of revisions since then and 320599e is a unite nubmer which can be used for network identification. Hence user would report revision xyz of v111.33.1 nightly, which is also fairly unique. One coudl even put hte whole strong in the title.

Isaac Eiland-Hall

Quote from: Markohs on March 18, 2014, 03:13:37 PM
It's also very important we have mirrors outside isaac's servers, because you never know what can happen to his ISP, and we have to have a plan for those situations, even if they are extremely unlikely to happen.

I'm all for external mirrors/backups, but just to note here that my server is in a datacenter; and actually I still have regret for that period of time I pushed hosting Simutrans away; I managed to keep the server during that mess, I just thought I wouldn't. :/

But iWeb.ca is a strong and stable company, and I've had nearly zero downtime during the months-shy of ten years I've been with them.

But again, backups and mirrors are always a good thing :)

Ters

Quote from: Markohs on March 18, 2014, 09:26:36 PM
No, I was refering exactly to what you expressed. Why can't that git repository, which everybody cloned and can receive pull requests, get svn updates, and push them back to svn when the admin desires so?

Because, as I understand it, if you rebase something that others have cloned, then all clones get out of sync. They have to start all over with a new clone, if they have pulled the rebased changelists before they were rebased. http://stackoverflow.com/questions/2715085/rebasing-and-what-does-one-mean-by-rebasing-pushed-commits seems to support my understanding of things.

Since committing back to SVN seems to require rebasing for everything but the most trivial cases, we can't have a public repo do syncing back to SVN. Furthermore, I don't think we can have changes go from a public git repo and back into SVN in a tidy fashion, except by making patches in Git and apply them to SVN, but then the development branches in Git would just lead to nowhere, with the changes just "suddenly" appearing on master.

Tron

Contrary to some rumors I'm not dead.
The DNS name for the server went away some time ago.
I was only informed about this by prissi on Friday (14.3.).
I had a busy weekend, so I read this email along with Dwachs' email, which he sent on Monday (17.3).
If somebody had told me earlier, I would have corrected the problem earlier.

The server is perfectly fine, there is just no DNS name for it currently.
Hopefully I get a new name for it today.